How best to fix 301 redirect problems

[daedriccarl]

Hi all

Wondering if anyone could help out with this one.   Roger Bot crawler has just performed it's weekly error crawl on my site and I appear to have 18,613 temp redirect problems!! Rather, the same 1 problem 18,613 times.

My site is a magento store and the errors it is giving me is due to the wishlist feature on the site. For example, it is trying to crawl links such as index.php/wishlist/index/add/product/29416/form_key/DBDSNAJOfP2YGgfW (which would normally add the item to one's wishlist). However, because Roger isn't logged into the website it means that all these requests are being sent to the login url with the page title of Please Enable Cookies.

Would the best way to fix this be to enable wishlists for guests? I would rather not do that but cannot think of another way of fixing it.

Any other Magento people come across this issue?

Thanks, Carl

[danwebman]

Use Rel="no index, no follow" on the link and roger will ignore the link. Job done

You wouldn't get any advantage of seeing the added items in someones wishlist as it would be duplicate content I'm presuming.

[daedriccarl]

Thanks will try and work out how to do that. Magento is a pain to make minor changes to.

Hopefully the crawl stats will look a little healthier next week !!

[adamxj2]

Hey Carl

I have this problem a lot on my Magento stores since theres a ton of ways to access the same product and it can be an SEO nightmare. If you have FTP access I would recommend adding this to the robots.txt file:

Disallow: /wishlist/

This will stop the bots from crawling everything passed /wishlist/. Here's a good resource if you need further assistance:  http://www.magentocommerce.com/knowledge-base/entry/setting-up-a-robotstxt-file

Hopefully that helps!

Good luck

[daedriccarl]

thanks. I completely forgot about the robots.txt method of fixing it. Will give it a go. I will add comparison in there too just to be safe.