Someone from Moz got back to me and said the best option is to add a directive to robots.txt to tell Roger to slow down the crawl.
Best posts made by cckapow
-
RE: Moz crawl crashing my site?
Welcome to the Q&A Forum
Browse the forum for helpful insights and fresh discussions about all things SEO.
Someone from Moz got back to me and said the best option is to add a directive to robots.txt to tell Roger to slow down the crawl.