Is the update site crawl feature following robot.txt rules?

[jamestown]

I noticed that most of the errors would not be occurring if Moz's tool followed the rules implemented in sites robots.txt. Has anyone else seen this problem and do you know if Moz will fix this?

[Martijn_Scheijbeler]

Hi James.

Do you also have rules for the crawler of Moz in there? You can learn more about Roger here: https://moz.com/help/guides/moz-procedures/what-is-rogerbot

Martijn.

[jamestown]

No because I thought Moz followed GoogleBot rules. Is this not the case?

[Martijn_Scheijbeler]

I'm not a 100% sure about it, but probably in specific cases you want to have your own statements for Rogerbot.