Moz was unable to crawl your site on Jun 22, 2020\. We were unable to access your site due to a page timeout on your robots.txt, which prevented us from crawling the rest of your site.

[KPMG-Search-Social]

Site: www.kpmg.us

Getting robots.txt timeout fail since 02/29/20.  We've checked our server logs and see no errors.  Went through all the steps of the "Troubleshooter".

Updated robots.txt to allow rogerbot full access:

User-agent: rogerbot
Disallow:

Any ideas how to get roger to crawl my site????

[lauren.s]

Hi there, thanks for reaching out!

Is the robots.txt for your site located here: "https://www.kpmg.us/robots.txt"?

If so, the issue may be that the robots.txt downloads as a text file which our crawler, rogerbot will be unable to follow. If our crawler is unable to access to the robots.txt it will cause the crawl to fail.

If you're still having issues, please feel free to reach out to the help@moz.com

[KPMG-Search-Social]

Hmmm,  seems all our robots.txt files download as text files. But the others (ex: advisory.kpmg.us/robots.txt) work with rogerbot.  I've asked our IT folk to see how were serving .txt files.

[KPMG-Search-Social]

We fixed the situation where the robots.txt files download (see: https://www.kpmg.us/robots.txt) but rogerbot still cannot crawl the site due to some "timeout" issue on the robots.txt.

[KPMG-Search-Social]

OK. Got a different error: Your site crawl timed out due to a slow server response.  Passing this along to IT.

[KPMG-Search-Social]

Back to the "We were unable to access your site due to a page timeout on your robots.txt".

Could it be the sitemap.xml page specified in the robots.txt is too slow?

Sitemap: https://www.kpmg.us/sitemap.xml