Moz crawl crashing my site?

[cckapow]

Has anyone ever tried to stop rogerbot from crawling their site? My tech team thinks that Rogerbot is crashing our site and causing a 504 error. Does anyone have experience with this or trying to stop the crawl? Thanks!

[cckapow]

Someone from Moz got back to me and said the best option is to add a directive to robots.txt to tell Roger to slow down the crawl.

[AndyMozster]

Not happened to my sites, so pinch of salt and all that. However, imo, root cause of this could be a number of things, but is likely a resource limitation on the hosting. Slowing down the crawl may well fix the 504 issue, but if your SEO is a glorious success and the site gets more traffic, then you may be looking at 504's again.

A 504 is a timeout and the problem is server side not with Moz.

• Made any recent DNS changes? Check your DNS health at something like https://mxtoolbox.com/domain/
• Firewall active? Could be it doesn't like Rogerbot.
• Logs should be telling your tech team what's going on
• WordPress? Occasionally database corruption will cause 504's
• If your team isn't able to provide a little more memory (likely culprit to my mind) then consider implementing caching; this will lighten the server load and also load pages faster to boot; good for users and SEO.