Crawl Errors from URL Parameter
-
Hello,
I am having this issue within SEOmoz's Crawl Diagnosis report. There are a lot of crawl errors happening with pages associated with /login.
I will see site.com/login?r=http://.... and have several duplicate content issues associated with those urls.
Seeing this, I checked WMT to see if the Google crawler was showing this error as well. It wasn't.
So what I ended doing was going to the robots.txt and disallowing rogerbot.
It looks like this:
User-agent: rogerbot
Disallow:/login
However, SEOmoz has crawled again and it still picking up on those URLs. Any ideas on how to fix?
Thanks!
-
Hi Tony,
I need more information from you in order to check this out. I'm going to send you a support ticket to reply to.
Thanks,
Joel.