Questions
-
Seeing what time a crawl was run, configuring crawl settings
Hey there! Tawny from Moz's Help Team here. The only way to see what time our crawler is crawling your site would be to check for our user-agent in your server logs: rogerbot. Unfortunately, there's no way to control the time of day at which we crawl your site. We just don't have a setting for that. There's also no way to control the day of the week the crawl takes place, for similar reasons. Your site will be crawled around the same time each week, with no way of adjusting when that takes place. We currently don't have a way to slow the crawl rate, either. Normally you'd be able to use a crawl delay in your robots.txt file, but our new crawler isn't following those crawl delays at the moment. In the future, that will be how you would slow down our crawler, but for now that option isn't up and running. Sorry about that! I'm afraid I wasn't too much help here, Patrick. My apologies I couldn't be more useful to you. If you still have questions, feel free to shoot us a note at help@moz.com and we'll do our best to sort things out with you.
Other Questions | | tawnycase0