Help Me With These Horrible Crawl Errors

[DannyHoodless]

I am currently using Opencart to run a fairly popular ecommerce website. I have done quite well in terms of rankings so far. However, the Opencart platform only goes so far.

There are endless amounts of crawl errors, from duplicate content down to missing meta tags. There is no easy way to add canonical tags. The pages that are being listed in the crawl errors seem to be mainly search URL's that i have no idea how Moz found them.

An example of the page is:

.....com/index.php?route=product/search&tag=home+office+flooring

Which has ten duplicate pages all missing meta tags.

I have used Google webmaster tools and htaccess to deny access to these pages with the slug "route=product/search" but it doesn't seem to work.

So my question to you guys is as follows:

Is it worth trying to fix these errors and do they have an effect on SEO?

If so, how do I prevent these errors that seem to grow at every crawl?

Cheers,

Danny

[Andy-Halliday]

Hi

Yes its worth fixing.

Could you not add Disallow: ?route=product/* to your robots file?

Can you edit the robots file.

Thanks

Andy

[DannyHoodless]

Andy,

My robots file currently looks like this

User-agent: *

Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/

Is this correctly set up?

[Andy-Halliday]

Nope you need to add the wildcard onto the end of the product line so it looks like this

User-agent: *

Disallow: /index.php?route=checkout/cart
Disallow: /index.php?route=account/return/insert
Disallow: /index.php?route=product/*

Otherwise it will just exclude that specific url.

Let me know if this works.

Thanks

Andy

[DannyHoodless]

Great, thanks Andy, Just doing a crawl now.

[Andy-Halliday]

You don't need to do another crawl - go into WMT and go to Robots.txt tester.

Get the live robots file (if it doesn't pull it in, copy and paste it).

Find a URL you wanted blocking and see if Google crawls it and you should get a warning message

[DannyHoodless]

I have tested with the robots tool too. Does seem to be blocked but running a crawl to see what moz brings up.

[MattRoney]

Hi Danny! How'd that crawl end up? If Andy answered your question, mind marking his response as a "Good Answer?" It'll get him some bonus MozPoints, and it helps us keep track of things in the forum.