Rogerbot problems

[X-com]

Hi,

One of our client is having a lot of trouble with Rogerbot. The client has a Magento Webshop www.motorcorner.nl.

Roger has got a major impact on the data usage (BPU) of the webshop. And we have no idea how to fix this at the moment.

The webshop is full of filters and we've already blocked these pages for most of the bots with a robots.txt file. This week we have decided to block rogerbot completely and see what the effect is on the data usage fot the next week.

• wat is the effect on Rogerbot with a robots.txt file?

• is it possible to decrease the crawl limit even more? (current 10.000pages)

• is it possible to decrease the crawl speed?

• Is it possible for rogerbot to crawl the webshop once a month?

[TomVolpe]

Hi there,

If you wanted rogerbot to only crawl once a month then you could put this in your robots.txt:

User-agent: Rogerbot
Disallow: *

Then you’d need to remove the week when you plan on checking out your moz dashboard, but this would make your Moz analytics crawl reports fluctuate between 'Roberbot cannot access your site' and the actual crawl errors. And you don’t know exactly when Rogerbot will crawl, so you may end up blocking the crawls you wanted to allow. I would recommend against this method, but as you ask about that specifically I’ve included it as an option.

I think your best course of action would be to add a crawl delay for Rogerbot:

User-agent: Rogerbot
Crawl-limit:2

Where the limit is a number equating to seconds to wait before crawling next page.

You could also use:

User-agent: *
Crawl-limit:5

To limit all but Googlebot, since it does not respect Crawl-limit and follows the speed set in Webmaster Tools under crawl rate.

I don’t believe you can set lower Crawl limit than that, but you could use wildcards in your Robots.txt to disallow certain pages by quite relaxed rules. To say your store has a size query string, which doesn’t actually change the content of the page, you could use:

Disallow: ?size=

This would stop all those pages being crawled. On a site with 1000 products and 3 sizes for each you’re disallowing 3000 pages from being crawled, so be careful with wildcards and use the Webmaster Tools Robots.txt tester to make sure you’re not about to Disallow access to a lot of important pages.

Hope that helps,

Tom

[ErinMcCaul]

Hi there!

Thanks for reaching out! My name is Erin, and I'm on the Moz Help Team. I appologize that it's taken us so long to get back to you. Things have been a little hectic around here!

Thanks for the great answer Tom! I'll do my best to answer all of your questions here as well.

• what is the effect on Rogerbot with a robots.txt file?
  Rogerbot will 100% follow robotx.txt. If you block him, he won't crawl your site.

• is it possible to decrease the crawl limit even more? (current 10.000pages)
  We actually bumped up our crawl limits recently, so I believe 10,000 pages is the lowest you can now set your crawl to.

• is it possible to decrease the crawl speed?
  As Tom said, you can set up a crawl delay to slow Rogerbot down!

• Is it possible for rogerbot to crawl the webshop once a month?
  Not on our side. Rogerbot will always crawl your site once a week. In theory you could block Rogerbot the weeks you don't want your site crawled, but that could get tricky. Adding a crawl delay would honestly be the better bet.

I hope this helps, and if you have any other questions shoot us an email at help@moz.com!

Cheers,

Erin

[JayShoe]

I'm having the same problem. In fact, my host has decided to BLOCK the rogerbot from crawling my site at all! When I asked them if they could limit rogerbot instead of blocking him, they said....

"They don't respect this, that is the problem, so we have no choice, basically. It is their job to configure their crawler properly."

That sucks! Because I want rogerbot to crawl my site!!!