Rogerbot not crawling our site

[cckapow]

Has anyone else had issues with Roger crawling your site in the last few weeks? It shows only 2 pages crawled. I was able to crawl the site using Screaming Frog with no problem and we are not specifically blocking Roger via robots.txt or any other method. Has anyone encountered this issue? Any suggestions?

[Andy.Drinkwater]

Who is your host? Some are starting to crack down on crawlers across the board with many blocking the likes of SEMrush. Might be worth checking if there are no other technical issues.

-Andy

[KristinaKeyser]

Hi there! Kristina from Moz's Help Team here.

I noticed that you're currently in conversation with my colleague, David, who is helping you out on this. It looks like he might have an idea of what's causing this to occur and he's waiting on the Crawl Test results to confirm if it's fixed for you.

I'll let David continue assisting you on this one, but as always you can reach out to our team with any technical issues such as this one by emailing us at help@moz.com.

Thanks so much,
-Kristina