Welcome to the Q&A Forum

I'm having the same problem. In fact, my host has decided to BLOCK the rogerbot from crawling my site at all! When I asked them if they could limit rogerbot instead of blocking him, they said.... "They don't respect this, that is the problem, so we have no choice, basically. It is their job to configure their crawler properly." That sucks! Because I want rogerbot to crawl my site!!!