Slowing down SEOmoz Crawl Rate

[corwin]

Is there a way to slow down SEOmoz crawl rate? My site is pretty huge and I'm getting 10k pages crawled every week, which is great. However I sometimes get multiple page requests in one second which slows down my site a bit.

If this feature exists I couldn't find it, if it doesn't, it's a great idea to have, in a similar way to how Googlebot do it.

Thanks.

[MeganSingley]

Hi corwin,

This is Megan from the SEOmoz Help Team.  I'm sorry if roger is being a bit too aggressive.   We built our crawler to obey robots.txt crawl-delay directives.  If you ever need to speed him down (though he's usually well-behaved), just add a crawl delay directive to your robots.txt file like this:

User-agent: rogerbotCrawl-delay: 1

Here's a good article that explains more about this technique: http://tools.seobook.com/robots-txt/

I hope this helps!

Cheers

[corwin]

Thank you for the reply Megan, just what I was looking for.