Why does SEOMoz crawler ignore robots.txt?

[loopyal]

The SEOMoz crawler ignores robots.txt

It also "indexes" pages marked as noindex.

That means it is filling up the reports with things that don't matter.

Is there any way to stop it doing that?

[Keszi]

Hi,

You can block RogerBot from Robots.txt

Check for further instructions on: http://www.seomoz.org/dp/rogerbot

"Please note: Adding this code will prevent our crawl test tool from being able to crawl your website."

Gr.,

Istvan

[loopyal]

Thank you István

I added this:

User-agent: rogerbot
Disallow: /sendtoafriend/
Disallow: /photo/
Disallow: /pix/

• because crawlers shouldn't go down those paths and roger is detecting pages without descriptions.

Is what I added OK?

[Keszi]

Hi Alan,

The code should be ok

Try to "drive-test" it with a custom crawl from http://pro.seomoz.org/tools/crawl-test then you will see if it works well.

I am glad the link was useful.

Gr.,

Istvan