Moz Pro, our web crawler, and sites that use SNI (804 HTTPS SSL) error

[jon.white]

Hi Phil - sorry you are seeing those errors.  I do not have a firm date, it will likely be in the next few weeks though.

[John-Sharkey]

Just another vote to see this happen.  I use SNI on some sites and a must have to be able to use Moz.

As a side not, I am not seeing any Google crawl errors in using SNI, at least not in Google Search Console.

[RobinGeall]

And a vote from me too Jon, I would need to this before converting from trial.

Good luck with it.

[AaronHemmelgarn]

Do we have a beta timeframe yet? It's been 3 months since the announcement that SNI crawl ability is coming and still no release. I have a very large client that I can't crawl their site. As a development company I can understand that good code takes time, but 3 solid months and really a over a year is what we're really looking at that Moz has known this was an issue.

Please provide us (as customers) with a solid date that SNI will be supported. I don't think that's too much to ask.

[jon.white]

Hello Aaron!  Unfortunately I am unable to provide a date - mainly because if I did, I would be guessing, and I don't think that is helpful.  We used to commit dates to releases, and we stopped, simply because our software is too complex to estimate accurately.

I will say it is close, I sent an email update earlier this week to all who sign up for the beta, I have included this below, it has a little more color:

Thank you for signing up for the Moz Pro Site Crawl beta.  I wanted to give you all an update since it has taken us a little longer to get the beta ready for testing.  We were hoping to have it out here by MozCon (this week) but we are slightly behind.  In the interest of transparency:

• our current crawler does not support SNI (as you well know) and is simply too old and fragile to make meaningful updates to, so we decided in Q1 of this year to re-write our crawler
• the new re-written crawler will support SNI, be much faster and more reliable, and support many more configuration options inc. crawl speed, includes/excludes, UA (though some of these will not be available for the first beta)
• we will also be giving the UX and UI of the crawl a revamp in combination with this crawler
• whilst the SNI support works right now, we are ironing out some issues with the other parts of the crawler before we feel comfortable releasing it in beta
• we had hoped we would be able to show off the beta at MozCon this week but some recent changes have slowed us a little
• we will be offering a second free-trial for those who are not current customers and wish to try it again when it is ready

I appreciate your patience, we are very close and hope to have something soon.

[TrueluxGroup]

Hi Jon,

I think we might have the same problem.

Our site is https://www.honesthouse.co.uk/ and we're getting the 804 SSL error too

Is there anything we can do?

Cheers

[DavidLee]

Hello!

I can confirm your site does use SNI which can be checked at https://globalsign.ssllabs.com/analyze.html?d=www.honesthouse.co.uk&latest

If anyone else is unsure about your own site, you can use the above link, replace the URL and click on any server results which will confirm SNI status shown here: http://www.screencast.com/t/GLzjD7gjzjR

As for any workaround, other than disabling SNI completely, the quickest option to get crawled would be to sign up for the beta.

Jon will keep everyone posted with any new status and I can assure you this is our top priority feature we are working on.

[AaronHemmelgarn]

Hi @David,

You mention a beta, is there someplace that we signup? I am a Moz Pro account holder and haven't seen anything offered or an e-mail.

Please provide steps for signup, as currently we have clients that we can't crawl and still paying for Moz.

[DavidLee]

Yes the sign up link is in the very first post by Jon

The link is http://goo.gl/forms/LCvL9Ix8JDHfbAvr1

[AaronHemmelgarn]

OK great, thank you.

[TrueluxGroup]

Can you recommend any suitable alternatives for web crawling our site while your new crawler is in development.

We already use screaming frog but it would be handy to have something that flagged up any problems like MOZ did with our other sites.

Cheers

[planetargon]

Do you have a rough ETA on the new crawler? Being without useful crawl data for several months has been painful

[etiennebeneke]

Hi Jon

Same problem this side, not sure if it's SNI. First we had 904 server error and then 902. Our site has not been crawled by RogerBot since 9 Oct when we switched from HTTP to HTTPS.

I am hoping for the best as I cannot use the full functionality of MOZ at the moment as we have one crawl error which completely keeps me in the dark as a SEO.

Many thanks
Etienne

[jon.white]

Hello Etienne - sorry to hear this.

Have you checked in with help@moz.com yet? They might be able to fix your problem.

[DavidLee]

Hi Etienne

I checked the campaign you have with the 902 response and the issue is with your host blocking AWS IPs. I could not find any SSL or SNI errors for the domain.

You will want to reach out to your hosting provider to make sure they are not blocking any AWS IP ranges. Our IPs are dynamic so we will not be able to provide a range for you.

There are also other issues with the SSL configuration which will need to be looked at which could result in different errors once the IPs are unblocked.

http://www.screencast.com/t/b8qEdNah

You can run a test here for any SSL domain: https://globalsign.ssllabs.com/analyze.html

Hope this helps!

[etiennebeneke]

Brilliant David,

I appreciate the lengths you have gone to get more info on my issue. I will get the host to address this. Thanks very much!

Etienne

[DavidLee]

Anytime!

[fivefifty]

Hello:

Have a clarifying question: using this site:  https://globalsign.ssllabs.com/analyze  one of our sites got the following status:

HTTP Strict Transport Security (HSTS) with long duration deployed on this server.

and not this one:

This site works only in browsers with SNI support.

However, we recently received this error through Moz:

Error Code 804: HTTPS (SSL) Error Encountered
Your page requires an SSL security certificate to load (using HTTPS), but the Moz Crawler encountered an error when trying to load the certificate.

Please advise if the HTTP Strict Transport Security (HSTS) also causes the Error Code 804: HTTPS (SSL) Error Encountered in Moz?

Thank You

[DavidLee]

Hi Ryan

I don't believe HSTS would prevent issues or this would apply to all https pages on every crawl report. For 804 response for any single page you want to monitor for the same URL appearing in the crawl report every week. Outside of SNI 804s will commonly be temporary issues.

If the same URL appears week to week with an 804 you would have to involve your web dev/host to further investigate.

[etiennebeneke]

Hi again David

I forwarded your response below to the server admin folks and they wanted me to check with you re this:

1. Perhaps MOZ can do a simple telnet test to www.ooba.co.za on port 443 to see if they can connect at the most basic level.

2. The SSL issues they are referring to are not very clear to me, I know we support SSL3, which is insecure but shouldn’t prevent anyone browsing the website. We also don’t support TLS 1.1 or 1.2, only TLS 1.0, do they have any specific requirements regarding TLS support?

I have to admit that most of this is rather Greek to me.

Thanks David

Etienne