Magento Dynamic Pages Being Indexed

[Shop-Sq]

Hi there, I have about 50k Moz medium priority errors in my Crawl Diagnostic report.

The bulk of them are classified as "Temporary Redirect" problems. Then if you drill into those further, I can see that the problem urls all kinda are center around:

mysite.com/catalogsearch/result..

mysite.com/wishlist..

mysite.com/catalog..

Is this something I should disallow in my Robstxt file? And if so how specific do I get with it..

Disallow /catalogsearch/result/?q=

Will listing the /catalogsearch be enough to cover anything after it?

thanks

[customerparadigm.com]

Yes, Magento does have a way of handling search results that are not always SEO-friendly.

To restrict Google from indexing anything in the /catalogsearch/ directory, I'd add this to your robots.txt file:

User-agent: *
Disallow: /catalogsearch/

Here's a Moz blog about the robots.txt file:
http://moz.com/blog/interactive-guide-to-robots-txt

And also, just in case the "killer" robots ever take over, Google has made sure that their two founders are not able to be indexed by both the T-1000 and T-800 models, with a special killer-robots.txt file:
http://www.google.com/killer-robots.txt

Hope this helps!
-- Jeff

[Shop-Sq]

Thank you very much Jeff!

[Essential-Pest]

Jeff is completely right on this one!