Adjusting SEOmoz Crawling Speed

[cappuccino89]

How do you adjust the SEOmoz crawling speed?

SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.

[kenneth_martin]

Hello,

You can add a crawl delay to your robots.txt file.

It would look something like this:

User-Agent: rogerbot
Crawl-Delay: 5

(time is in seconds between page requests)

Hope that helps!

Kenny