We recently switched from HTTP to HTTPS and we are having crawling issues!

[DrInfinity]

We switched our website from HTTP to HTTPS and we started to get an email from Moz about the robots.txt being unable to crawl our website. The website is hosted through wordpress but we haven't had any issues until we switched. We have no idea what to do or even what the problem is! If you have had a similar problem and fixed it, we need your help! Thank you.

[meghanpahinui]

Hi there!

Thanks so much for reaching out! I'm sorry you're having trouble!

I took a look at your crawl data and your site to see if I could figure out the issue.  When I first tried to access your robots.txt file from a browser, it returned an error saying there were too many redirects in place. I checked to see what our crawler was receiving from your server and looks like it keeps being served a 301 redirect which points back to itself. However, when I tried to access the file from a browser a bit later, it loaded without a problem. I'm wondering if you can check your server logs to see what your server is sending back to our crawler Rogerbot?

If you could send any further info over to help@moz.com, that would be great! That way we can do some more digging and see what's going on.

Looking forward to hearing from you!

[DougDeVore]

I know this is an old thread, but we are still having the same problem.  I finally got around to sending a note to flywheel about this problem and it came back that everything is fine.  I am not sure what to do here?  It's on a shared hosted, so I don't have console\audit log access, however Flywheel is one of the best wordpress hosting companies out there (only thing they do).

As far as accessing the robots.txt file, I can go directly to it without any problems?
https://southernil.com/robots.txt