Moz was unable to crawl your site - robots.txt

[JRGRDZ]

This post is deleted!

[meghanpahinui]

Hi there!

Thanks so much for reaching out!  I'm so sorry you're having trouble with the site crawl!  I took a look at the robots file you provided and looks like when our crawler tries to access the page, it receives a 520 Origin Error from the server.

We do have a resource in our help hub which may help you dive deeper into the issue: https://moz.com/learn/seo/http-status-codes

I hope this helps get you pointed in the right direction!  If you have any follow up questions, please feel free to reach out to us via help@moz.com

[Cesare.Marchetti]

Hi JRGRDZ,

I just checked your robots.txt link: its for sure accessible from the outside, the http status code is OK too (= 200).

The only suspect thing i see, that shouldn't be a problem actually is:

User-agent: *
Crawl-delay: 600

Why do you do that? It allows a crawler (only the ones that follow that rule; Google does not) to access the site only all 600 s.

Cheers,

Cesare

[JRGRDZ]

Good catch! We'll try removing this line. I'l report back our findings.

This article was helpful: https://moz.com/help/guides/moz-pro-overview/crawl-diagnostics/errors-in-crawl-reports

[JRGRDZ]

Thank you. Article was very helpful.

Jorge