Questions
-
Too Many Moz crawl hits on the server
Hi there Rishi, Thanks so much for reaching out and sorry about the trouble here! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file. Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time. One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this: User-agent: rogerbot Disallow: * Could you please pop a message about this as well to help@moz.com using your Moz email address so that we can notify our engineering team and keep you updated as soon as we are following crawl delay directives?
Other Questions | | samantha.chapman0