Moz Pro, our web crawler, and sites that use SNI (804 HTTPS SSL) error

[jon.white]

TL;DR

• Many sites and CDN’s across the internet use SNI to help keep pages and resources secure

• Right now, Moz Pro is unable to crawl sites that use SNI, affecting these customers (you will see an 804 HTTPS SSL error if you are affected)

• This is being fixed, and we are almost ready to provide beta access to SNI customers

• If you have an SNI site, are a Moz Pro customer, and wish to be a part of this beta - sign up here http://goo.gl/forms/LCvL9Ix8JDHfbAvr1

Detail

SNI (Server Name Identification) is an extension to the TLS security protocol that allows for multiple HTTPS websites to be served from the same IP address without requiring them all to use the same certificate.  This was always possible using wildcard certificates, but these are often impractical, since they are restrictive, and require a webmaster to know all of the hostnames to be served at the time of requesting the certificate.  With SNI, a webmaster can issue a single certificate and support multiple (and changing) hostnames.

SNI has become increasingly popular over the last couple of years thanks to:

• Increase in mainstream browser support

• Its flexibility for usage in CDNs, like Cloudflare

Unfortunately the Moz Pro web crawler has not kept up to date with this trend, and is at present unable to crawl sites that use SNI (the symptom is an 804 HTTPS SSL error).  This affects customers with SNI hosted sites, who miss out on the value of Moz Pro’s Site Crawl tools.  Whilst it might seem a simple fix to support SNI, our crawler software uses libraries that are unable to support the technology without a significant re-write.

However, I am happy to say that this re-write is in process, and almost ready for testing.  Behind the scenes we have been working on a major overhaul of our crawl technology which you are going to start to see across many of our products. In fact, Moz Content adopted the new crawler this week, resulting in much faster crawls and support for SNI.

Moz Pro is not ready for full implementation yet. That said, we are pretty close to being able to beta test with customers.  For that reason we would like to invite any customers with SNI sites into our closed Beta that will likely start some time in August.

Some notes about the beta:

• Initially, it will be a little behind where the current crawl tools are in terms of features and configuration. That said, we will be iterating fast and beta customers will get automatic access to all new features

• As with any beta there will be bugs and potential data outages - I want to be transparent about this up-front

• We will be looking for feedback throughout the beta, and we would love to have our customers help shape the future of our software

• You must be a paid Moz Pro subscriber (or in a free trial)

If you are interested please sign up here - http://goo.gl/forms/LCvL9Ix8JDHfbAvr1. Questions and comments welcome below!

[matthewnmartin]

Thank you for the update. We have several sites that cannot be crawled and after a lot of investigation have concluded we would need to cancel Moz. I'm glad this is being released and submitted the form to be contacted regarding the Beta Testing.

I am curious though, what is in it for us? As of now we are paying for a service that does not work. I seriously doubt we will be interested in continuing to pay for a service just to be a test dummy (which one of the requirements is a paid account, unless I am misunderstanding).

Either way, we are excited about the solution and interested in learning more.

Thanks.

[jon.white]

Thanks for your response.

Firstly, I empathize with your decision to cancel, and I hope we can win you back in the future.

In terms of why a customer might participate. Many of our customers who have SNI problems are still Moz customers, and still use our Rankings, Links and Keyword Research tools.  In this case, they would also be able to get some crawl data (albeit in a beta form) as a temporary fix whilst we finish the crawler.  We had already planned to launch a beta with select customers, we think this might also help us help some customers who are having struggles with our product.  But I realize this route is not for everyone.

Please let me know if there is any other feedback you guys have about Moz.  jon@moz.com

[matthewnmartin]

Great, we would like to participate as some results are better than nothing. We filled out the form so i suppose we wait.

[jon.white]

Thank you. I look forward to sharing something.

[Globalgraphics]

Hi Jon, Do you have an update as to when the new crawler will be available. We have just started getting 804 HTTPS SSL error's for one of our sites.

[jon.white]

Hi Phil - sorry you are seeing those errors.  I do not have a firm date, it will likely be in the next few weeks though.

[John-Sharkey]

Just another vote to see this happen.  I use SNI on some sites and a must have to be able to use Moz.

As a side not, I am not seeing any Google crawl errors in using SNI, at least not in Google Search Console.

[RobinGeall]

And a vote from me too Jon, I would need to this before converting from trial.

Good luck with it.

[AaronHemmelgarn]

Do we have a beta timeframe yet? It's been 3 months since the announcement that SNI crawl ability is coming and still no release. I have a very large client that I can't crawl their site. As a development company I can understand that good code takes time, but 3 solid months and really a over a year is what we're really looking at that Moz has known this was an issue.

Please provide us (as customers) with a solid date that SNI will be supported. I don't think that's too much to ask.

[jon.white]

Hello Aaron!  Unfortunately I am unable to provide a date - mainly because if I did, I would be guessing, and I don't think that is helpful.  We used to commit dates to releases, and we stopped, simply because our software is too complex to estimate accurately.

I will say it is close, I sent an email update earlier this week to all who sign up for the beta, I have included this below, it has a little more color:

Thank you for signing up for the Moz Pro Site Crawl beta.  I wanted to give you all an update since it has taken us a little longer to get the beta ready for testing.  We were hoping to have it out here by MozCon (this week) but we are slightly behind.  In the interest of transparency:

• our current crawler does not support SNI (as you well know) and is simply too old and fragile to make meaningful updates to, so we decided in Q1 of this year to re-write our crawler
• the new re-written crawler will support SNI, be much faster and more reliable, and support many more configuration options inc. crawl speed, includes/excludes, UA (though some of these will not be available for the first beta)
• we will also be giving the UX and UI of the crawl a revamp in combination with this crawler
• whilst the SNI support works right now, we are ironing out some issues with the other parts of the crawler before we feel comfortable releasing it in beta
• we had hoped we would be able to show off the beta at MozCon this week but some recent changes have slowed us a little
• we will be offering a second free-trial for those who are not current customers and wish to try it again when it is ready

I appreciate your patience, we are very close and hope to have something soon.

[TrueluxGroup]

Hi Jon,

I think we might have the same problem.

Our site is https://www.honesthouse.co.uk/ and we're getting the 804 SSL error too

Is there anything we can do?

Cheers

[DavidLee]

Hello!

I can confirm your site does use SNI which can be checked at https://globalsign.ssllabs.com/analyze.html?d=www.honesthouse.co.uk&latest

If anyone else is unsure about your own site, you can use the above link, replace the URL and click on any server results which will confirm SNI status shown here: http://www.screencast.com/t/GLzjD7gjzjR

As for any workaround, other than disabling SNI completely, the quickest option to get crawled would be to sign up for the beta.

Jon will keep everyone posted with any new status and I can assure you this is our top priority feature we are working on.

[AaronHemmelgarn]

Hi @David,

You mention a beta, is there someplace that we signup? I am a Moz Pro account holder and haven't seen anything offered or an e-mail.

Please provide steps for signup, as currently we have clients that we can't crawl and still paying for Moz.

[DavidLee]

Yes the sign up link is in the very first post by Jon

The link is http://goo.gl/forms/LCvL9Ix8JDHfbAvr1

[AaronHemmelgarn]

OK great, thank you.

[TrueluxGroup]

Can you recommend any suitable alternatives for web crawling our site while your new crawler is in development.

We already use screaming frog but it would be handy to have something that flagged up any problems like MOZ did with our other sites.

Cheers

[planetargon]

Do you have a rough ETA on the new crawler? Being without useful crawl data for several months has been painful

[etiennebeneke]

Hi Jon

Same problem this side, not sure if it's SNI. First we had 904 server error and then 902. Our site has not been crawled by RogerBot since 9 Oct when we switched from HTTP to HTTPS.

I am hoping for the best as I cannot use the full functionality of MOZ at the moment as we have one crawl error which completely keeps me in the dark as a SEO.

Many thanks
Etienne

[jon.white]

Hello Etienne - sorry to hear this.

Have you checked in with help@moz.com yet? They might be able to fix your problem.