MOZ Crawler

[macpalace]

Hi,

how much time it will take MOZ crawler to take entire site? In 24 hours it crawled only 500 pages isn't it too slow? My website has almost 50k pages.

[effectdigital]

Yeah it will take a while but it's necessarily slow. Moz's crawler has to share bandwidth with your real, human users. If they crawl your site too fast they could lag your site or accidentally DDoS you or something

Since it's a paid for service, I'm guessing that Moz err on the side of caution with Rogerbot and don't crawl insanely fast. I know that Rogerbot supports adding a crawl delay to make the crawling even slower:

https://moz.com/help/moz-procedures/crawlers/rogerbot

But I don't think there are any 'go faster' commands. Whilst I really like Moz's crawler I tend to find that the SEMRush one is faster for on-site stuff, though it does have URL crawling caps

To be honest, just get Screaming Frog and learn how to use that. Now that it has a database storage mode (instead of maniacally trying to store all the crawl data in the RAM) - it probably could crawl all 50K URLs. It could probably do it faster than Rogerbot, though you wouldn't wanna go nutty as you could easily DDoS yourself :')

[samantha.chapman]

Hi there,

Sam from Moz's Help Team here! Could you pop a message over to help@moz.com about this so we can ensure everything is running smoothly for you?

Thank you!