Questions
-
How can you tell Rogerbot when you want it to crawl?
Hi there! Rogerbot can't actually be told to crawl at a specific time. However, if you're finding that he is crawling your site too aggressively you could set up a crawl delay in your robots.txt file to set the number of seconds between requests to the same server.
Other Questions | | holly_haymaker1