RogerBot does not respect some rules??

[jgomes]

Hello;

Every week when I see my stats I notice that RogerBot has crawled 10000 form my website, even pages with a no index or not allowed in the robots.txt.

Is it possible to avoid him from crawling the these pages? They are form pages in my site, with are not indexed by google, they have a noindex and they are not allowed for crawling in the robots.txt.

Thanks everyone for your help!!!

[Martijn_Scheijbeler]

Hi Jorge,

Yes this would be possible, Rogerbot is also the User Agent for the crawler. So within you're robots.txt you are capable of letting Roger know which pages you don't like him to crawl. More information about this could be found on this page about Roger himself.

Hopefully this answers your question.

[KeriMorgret]

If Roger is still not listening to you, send an email to help@seomoz.org and open a ticket with the help desk. They'll try to figure out why he's misbehaving and how to get him to listen to you again.