Moz was unable to crawl - how to fix?

[Calderagroup]

Hi there,

I read a few of the earlier posts about this issue, but i can't get the problem with the crawling issue fixed (see notification below). Especially since i'm not really technical, i have no idea where to start. We have many client websites and three of them get this message. I don't see much difference between te robot.txt of the ones working and the ones not working.

Not working:
http://www.calderagroup.nl/
http://www.ptone.nl/
http://snijdersfeerverwarming.nl/

Anyone knows where to start? How to fix?

**Moz was unable to crawl your site on Oct 5, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.

[tawnycase]

Hey there!

Sorry to hear that a couple of your Campaigns are running into Robots.txt crawl issues! Maybe I can help figure out why those sites can't be crawled.

It looks like the robots.txt file for the first site you mentioned is returning a 403 Forbidden response to our user-agent, Rogerbot: https://www.screencast.com/t/TFFaWtikf Anything other than a 301 redirect or a 200 OK status wills top us from being able to access the robots.txt file, which will result in a crawl failure. Please make sure your robots.txt file is fully accessible to rogerbot.

For the second site, ptone.nl, we're getting a 406 Not Allowed error: https://www.screencast.com/t/np1AEJ3oDiy2 Again, we'll need to be able to access those robots.txt files in order to crawl your site.

And that last site is also returning a 403 Forbidden response when we try to access your robots.txt file for that site: https://www.screencast.com/t/CEpKpLq3z

I'd recommend checking in with your web developer or whoever manages your site for you. They'll probably know how to make the necessary changes to make sure that your robots.txt files are accessible to our crawler.

Hope this helps! If you've still got questions, feel free to drop us a line at help@moz.com and we'll do everything we can to answer them all.