20 x '400' errors in site but URLs work fine in browser...

[Switch_Digital]

Hi, I have a new client set-up in SEOmoz and the crawl completed this morning... I am picking up 20 x '400' errors, but the pages listed in the crawl report load fine... any ideas?

example -

http://www.morethansport.co.uk/products?sortDirection=descending&sortField=Title&category=women-sports clothing

[TampaSEO]

Just ran the example link that you provided through two independent HTTP response code checkers, and both are giving me a 200 response, i.e. the site is OK.

This question has been asked before on here; you're definitely not the first person to run into the issue.

One way to diagnose what's going on is to dig a little deeper into the crawling report that SEOmoz generated. Download the CSV file and look at the referring link, i.e. on which page Roger found the link. Then go to that page and look if your CMS is doing anything weird with the way it outputs the links that you create. I recall someone back in December having the same issue and eventually resolved it by noticing that his CMS put all sort of weird slashes (i.e. /.../...) into the link.

Good luck!

[Switch_Digital]

Thanks Tampa SEO, good advice.

Interestingly, the URL listed in SEOmoz is as follows:

www.morethansport.co.uk/brand/adidas?sortDirection=ascending&sortField=Price&category=sport and leisure

But when I look at the link in the referring page it is as follows:

/brand/adidas?sortDirection=ascending&sortField=Price&category=sport%20and%20leisure

notice the "%" symbol instead of the spaces.

The actual URL is the one listed in SEOmoz but even if I copy and paste the % version, the browser removed the '%' and the page loads fine.

I still can't get the site to throw-up a 400.

[Cyrus-Shepard]

Hi Justin,

Sometimes when rogerbot crawls a site, the servers and/or the content management system can get overwhelmed if roger is going to fast, and this causes your site to deliver error pages as roger crawls.

If the problem persists, you might consider installing a crawl delay for roger in your robots.txt file. It would look something like this:

User-agent: rogerbot
Crawl-delay: 5

This would cause the SEOmoz crawlers to wait 5 seconds before fetching each page. Then, if the problem still persists, feel free to contact the help team at help@seomoz.org

Hope this helps! Best of luck with your SEO!

[Switch_Digital]

Fantastic, thanks, Cyrus and Tampa, prevented many more hours of scratching head!!!

[Switch_Digital]

Sorry, one last question.

Do I need to add a similar delay for Google Bots, or is this issue specifically a Roger Bot problem?

Thanks

[Cyrus-Shepard]

Most major robots obey crawl delays. You could check your errors in Google Webmaster Tools to see if your site is serving a lot of error pages when Google crawls.

I suspect Google is pretty smart about slowing down its crawl rate when it encounters too many errors, so it's probably safe to not include a crawl delay for Google.