Moz unable to crawl my Zenfolio website

[bpenn11]

Hey guys,

I am attempting to optimize a website for my wife's business but Moz is unable to crawl it.  Zenfolio is the web hosting service (she is a photographer).

The error message is:

**Moz was unable to crawl your site on Apr 1, 2019. **Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster. Read our troubleshooting guide.

I did read the troubleshooting guide but nothing worked.

My robots.txt file disallows a few bots, but not roger bot.  Anyone have any idea what is going on?  Or do I need to request server logs from Zenfolio?

Thanks

[dave.kudera]

Hey!  Dave here from the Help Team

Can you write into us at help@moz.com with the site that is having crawl issues?  We can take a look and help you out.

thanks!

[NickSamuel]

Hi OP.

Hopefully you got in touch with Dave and the Moz team, and you have got your answer!

I haven't got a direct answer but from experience of Moz and other crawler softwares, my money would be on a temporary issue with your server being unresponsive.

As per the "Moz Can't Crawl your site guide, as it states, if it can't read your robots.txt file, then "Roger" won't crawl your site.

This is my simplest explanation I have

Hopefully it's resolved soon!

P.S

I looked at Zenfolio, if it's a self-hosted CMS system, it might be worth checking they haven't got any special rules in place to block Moz and other popular crawlers

[bpenn11]

It appears that Zenfolio really doesn't want Moz crawling their site for some reason.  They want me to jump through hoops to get it to work and, being an ultra small business, cant justify the effort to get it working.

Guess I could always migrate to another web host.

Thanks for the replies.