Crawl error robots.txt

[Mandiram]

Hello, when trying to access the site crawl to be able to analyze our page, the following error appears:

**Moz was unable to crawl your site on Nov 15, 2017. **Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. Update these tags to allow your page and the rest of your site to be crawled. If this error is found on any page on your site, it prevents our crawler (and some search engines) from crawling the rest of your site. Typically errors like this should be investigated and fixed by the site webmaster.

Can help us?

Thanks!

[Linda-Vassily]

It looks like the error you are referring to did not come through in your question. Could you try editing it?

[Mandiram]

Sorry, the image don't appear  
Try now

[Linda-Vassily]

That just says that you are blocking the Moz crawler. Take a look at your robots.txt file and see if you have any exclusions in there that might cause that page not to be crawled. (Try going to yoursite.com/robots.txt or you can learn more about this topic here.)

[Mandiram]

Yes, our robots.txt it's very simple:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

[Linda-Vassily]

Hmm. How about on the page itself? Is there a noindex?

[Mandiram]

The page is: http://www.yogaenmandiram.com/ And don't have noindex

[Linda-Vassily]

I just popped that into ScreamingFrog and I don't see a noindex on that page, but I do see it on some other pages. (Though that shouldn't stop other pages from being crawled.)

Maybe it was just a glitch that happened to occur at the time of the crawl. You could try doing another crawl and see if you get the same error.

[tawnycase]

Hey there!

This is a tricky one — the answer to these questions is almost always specific to the site and the Campaign. For this Campaign, it looks like your robots.txt file returned a 403 forbidden response to our crawler: https://www.screencast.com/t/f42TiSKp

Do you use any kind of DDOS protection software? That can give our tools trouble and cause us to be unable to access the robots.txt file for your site.

I'd recommend checking with your web developer to make sure that your robots.txt file is accessible to our user-agent, rogerbot, and returning a 200 OK status for that user-agent. If you're still having trouble, it'll be easier to assist you if you contact us through help@moz.com, where we can take a closer look at your account and Campaign directly.

[Mandiram]

Thanks Lind and Tawny! i 'll check it