Moz Pro, our web crawler, and sites that use SNI (804 HTTPS SSL) error

[TrueluxGroup]

Can you recommend any suitable alternatives for web crawling our site while your new crawler is in development.

We already use screaming frog but it would be handy to have something that flagged up any problems like MOZ did with our other sites.

Cheers

[planetargon]

Do you have a rough ETA on the new crawler? Being without useful crawl data for several months has been painful

[etiennebeneke]

Hi Jon

Same problem this side, not sure if it's SNI. First we had 904 server error and then 902. Our site has not been crawled by RogerBot since 9 Oct when we switched from HTTP to HTTPS.

I am hoping for the best as I cannot use the full functionality of MOZ at the moment as we have one crawl error which completely keeps me in the dark as a SEO.

Many thanks
Etienne

[jon.white]

Hello Etienne - sorry to hear this.

Have you checked in with help@moz.com yet? They might be able to fix your problem.

[DavidLee]

Hi Etienne

I checked the campaign you have with the 902 response and the issue is with your host blocking AWS IPs. I could not find any SSL or SNI errors for the domain.

You will want to reach out to your hosting provider to make sure they are not blocking any AWS IP ranges. Our IPs are dynamic so we will not be able to provide a range for you.

There are also other issues with the SSL configuration which will need to be looked at which could result in different errors once the IPs are unblocked.

http://www.screencast.com/t/b8qEdNah

You can run a test here for any SSL domain: https://globalsign.ssllabs.com/analyze.html

Hope this helps!

[etiennebeneke]

Brilliant David,

I appreciate the lengths you have gone to get more info on my issue. I will get the host to address this. Thanks very much!

Etienne

[DavidLee]

Anytime!

[fivefifty]

Hello:

Have a clarifying question: using this site:  https://globalsign.ssllabs.com/analyze  one of our sites got the following status:

HTTP Strict Transport Security (HSTS) with long duration deployed on this server.

and not this one:

This site works only in browsers with SNI support.

However, we recently received this error through Moz:

Error Code 804: HTTPS (SSL) Error Encountered
Your page requires an SSL security certificate to load (using HTTPS), but the Moz Crawler encountered an error when trying to load the certificate.

Please advise if the HTTP Strict Transport Security (HSTS) also causes the Error Code 804: HTTPS (SSL) Error Encountered in Moz?

Thank You

[DavidLee]

Hi Ryan

I don't believe HSTS would prevent issues or this would apply to all https pages on every crawl report. For 804 response for any single page you want to monitor for the same URL appearing in the crawl report every week. Outside of SNI 804s will commonly be temporary issues.

If the same URL appears week to week with an 804 you would have to involve your web dev/host to further investigate.

[etiennebeneke]

Hi again David

I forwarded your response below to the server admin folks and they wanted me to check with you re this:

1. Perhaps MOZ can do a simple telnet test to www.ooba.co.za on port 443 to see if they can connect at the most basic level.

2. The SSL issues they are referring to are not very clear to me, I know we support SSL3, which is insecure but shouldn’t prevent anyone browsing the website. We also don’t support TLS 1.1 or 1.2, only TLS 1.0, do they have any specific requirements regarding TLS support?

I have to admit that most of this is rather Greek to me.

Thanks David

Etienne

[fivefifty]

Thanks for the clarification.

[DavidLee]

Hi Etienne

There is a public 3rd party tool which also fails to connect which is hosted on AWS:

https://www.hurl.it/

If you run a GET request to this URL you will see a time out. Your admins can use this tool to quickly isolate any issues as this timeout is the same we are running into.

The SSL issues reported may not block normal browser connections but possible non-browser user-agents which would be another bot or server. We do not have any requirements or TLS, 1.0 and up is supported.

Hope this helps!

[etiennebeneke]

Many thanks David for your effort. I will feed this information through to the server guys.
Etienne

[John-Sharkey]

Hello Jon,

Any update on the new SNI fix, or the beta crawler?  Maybe a Xmas present??

Kind regards,
John

[jon.white]

We are testing in-house beginning next week, I don't have a sense how long this testing cycle is going to take, but this is the final stage before we launch it.  Thanks for your patience.

[John-Sharkey]

Absolutely... thanks so much for your work on this!

Cheers,
John

[spadedesign]

We have yet to see a fix for this issue or receive any word of trying the new software.

We are going on 5 or 6 months.

[IWaldron]

Any update to this estimate? Anxiously waiting...

[Buchs-A-S]

This post is deleted!

[CroyalDK]

Any news on the SNI fix?