Huge spike in crawl errors today - mozbot ignoring noindex tag?

[Travis-W]

Hi Mozzers,

Today I received a ton of errors and warnings in my weekly crawl due to the mozbot crawling my noindex'd search results pages, such as this - http://www.consumerbase.com/Mailing-Lists.html?q=Construction&type=bus&channel=all&page=7&order=title&orderBy=DESC See image: http://screencast.com/t/qaZzq78j2Udx

Anyone else seen a similar error this week with their crawl?

Thanks!

[DarinPirkey]

Don't forget that Rogerbot is a crawler and not an Index.  Google will crawl those pages too but will (generally) follow your instructions to noindex those pages.

Here is a little information on Rogerbot  http://moz.com/help/pro/rogerbot-crawler

And here is a little information on the frequently asked questions for th diagnostics report  http://moz.com/help/pro/crawl-diagnostics

Hope this helps.

Darin.

[Travis-W]

After the last crawl, everything returned to normal and all the errors disappeared.  Roger must have just had a hiccup.

[DarinPirkey]

Great to hear!