On-Page Grader supporting HTTPS?

[ToteIreland]

Hi there,

I am getting the error "Sorry, but this URL is inaccessible" on the Moz On-Page Grader. I have tested with multiple URL's but any that link to my site give the error. I have noticed my site's URL's redirect to https so my question is..is the on-page grader in-compatible with https?

Cheers!
Ross

[JordanRailsback]

Hi Ross!
On-Page Grader should definitely function for https sites, in general. For example, https://moz.com looks to be working just fine.

If you're getting that response from On-Page Grader, it's usually related to some sort of blockage. I'd recommend checking through your robots.txt to make sure that rogerbot and dotbot aren't being blocked, that your SSL is all looking good, and that the site isn't blocking Amazon Web Services (AWS).

[ToteIreland]

Hi Jordan,

Thanks a million for your response! I am still having the same issue..my robots.txt is as follows:

User-agent: * 
Disallow: /

Would this be blocking those bots? I know this blocks bots from accessing ALL content, but does it block all bots from accessing my site in general? Should I be changing it and if so, what would you recommend I change to allow for optimal crawling?

Cheers and thanks for taking the time to respond!

[JordanRailsback]

Ah, yep, that would do it!

That configuration is currently blocking every bot known to man from accessing every portion of your site. Here's an awesome resource to help with robots.txt setup: http://www.robotstxt.org/robotstxt.html

If you were looking to allow all robots to access all pages on your site, then this command would do the trick:

User-agent: *

Disallow:

(Without the blank line between commands. Q&A formatting be damned. )

[ToteIreland]

Hmm I thought that too! It's strange though because my site is being indexed on Google and Moz is able to crawl it for other features. When I run the robots test in Google Webmaster Tools, the one where you run a test to see if URL's are blocked and you pick a bot e.g Googlebot...it says everything is working ok and it is allowed? Wouldn't that not be the case if bots were blocked from crawling the site?

[ToteIreland]

I've changed it too:

User-agent: rogerbot

Allow: /

Would this let Moz crawl the site?

[JordanRailsback]

I would expect that to take care of the issue

[ToteIreland]

Thanks Jordan, it didn't make a difference but what I found was that my hosting company was blocking Amazon Web Services. They said they need a range of IP's from the cloud service that Moz uses to unblock, is there anyway you can advise me on what these are and is Amazon the cloud service that Moz uses?

Cheers!

[JordanRailsback]

Oh good! I'm glad you were able to find the source!

We actually use a set of dynamic IPs, so there's not a range that I'd be able to provide. I'm afraid the best things I could suggest would be to ask them to whitelist our user agents, dotbot and rogerbot.