Moz crawler showing pages blocked by robots.txt

[ahyde]

I've blocked a large number of pages which Moz were showing as duplicate or giving 404's in our robots.txt using /?key and /?p etc.

However Moz crawler is still showing as being an issue. I assumed Roger picked up the robots.txt file, or is that not the case?

[DirkC]

Did you check these pages with the robots.txt tester in Webmastertools to be sure that these pages are really blocked for bots? Did you exclude all bots or only the Googlebot?

[ahyde]

Yep. Checked in the tester at the time and did some more spot checks this morning.

I used Disallow: /*?key which I thought was generally recognised by bots?

[DirkC]

Not really an expert in robots.txt - but you could always try to simulate Rogerbot using a tool like Screaming Frog (the tool is free if you crawl less than 500 pages).

You can enter a custom user agent. Rogerbot seems to be rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com) and try crawling the pages on your site. In default mode, the tool will only crawl pages that are allowed by the robots.txt. If the tool crawls your pages, then something is wrong with the robots.txt - if not, than you should contact Moz staff to check with them.