Too Many Moz crawl hits on the server

[joigifts]

Hi Moz,

I am one of your tool customer (MOZ Pro). Since two days (25th and 26th November) We are experiencing too many requests to the server and the source of that is MOZ. ITs taking 90% of our server capacity.

I am attaching screenshots for your reference.

Please investigate. Appreciate your reply.

Regards,

Rishi

tTJ7a tTJ7a tTJ7a

[samantha.chapman]

Hi there Rishi,

Thanks so much for reaching out and sorry about the trouble here!
Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.

Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files.  I'm so sorry about that!  Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.

One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:

User-agent: rogerbot
Disallow: *

Could you please pop a message about this as well to help@moz.com using your Moz email address so that we can notify our engineering team and keep you updated as soon as we are following crawl delay directives?