804 HTTPS (SSL) error

[lanehouk]

I am getting these 804 crawl errors on multiple campaigns/domains

We are using Cloudflare Flexible SSL on all these domains. Can anyone advise on this?

[jameskais]

Hi Lane - Our current crawl libraries do not support SNI which is typically what is enabled by default by Cloud Flare. Our engineering team is still investigating expanding our libraries but an ETA is not available quite yet.

[SEMbyotic]

I too am using CloudFlare and have the same error. I'd like to be notified when you have an update.

Thanks,
Adam

[dsmolinski]

My understanding is that Moz can't crawl sites with SSL/Https.  I like Moz but if it doesn't work then I can't continue to pay for it then.

This was a response I received from support:

if you tried the suggestions in those posts and were unable to find a solution, then unfortunately it looks like we will not be able to crawl your page at this time. Our crawler has trouble with SSL pages and is often not able to crawl them at all. We are working towards fixing this, but it's a large-scale project. I'm sorry to be the messenger on this. If there is anything else I can help with, please let me know.

[jameskais]

Hey there - sorry if there was a miscommunication but we do in fact support SSL, however at this time we do not support SNI (Server Name Identification) which is a technology used to host multiple SSL certificates from a single IP address. If you are currently leveraging CloudFlare or CloudFront we will not be able to crawl your site. If you are using a standard SSL from it's own IP we can most likely crawl that page.

[dsmolinski]

This post is deleted!

[jameskais]

At this time they will need their own as we do not support any SNI at this time. With that being said we are aware of it and have investigated solutions but an ETA is not quite yet available. If these are the only sites you planned on using Moz with we won't be of much service at this time =(.

[FabricEleven]

Is there a way around this? Maybe creating a cname in cloudflare like moz.domain.com and set it to not use their https would that work? Or will all the results be skewed since domain name has changed?

[CandymanKen]

Hi-

Have there been any updates to this? We have been using  cloudlfare for probably a year and all of a sudden I am getting the 804 errors on our SSL.

Any help would be appreciated.

Ken

[dswz]

Please fix this. I am using SNI certificates from Cloudflare on different domains. IPs are getting rare.

[Arizuka]

Same problem for us. We are glad to use moz but we have migrated from http to https and nothing works. Do you have any clue if it will be or not in your roadmap developments?

[micvitale]

same problem for me.

I am working with https and cloudflare.

what if I disable cloudflare just for the crawl test? would it work? how could I disable it temporarily?

thank you

[ClockSoftware]

Same here.

We use CloudFront and receive the 804 error response. We consider to cancel our subscription  as we get no data and there is no information if and when a solution can be expected.

[bluedotproductions]

We too use SNI for our certificates on my co-located server with a top host.

It sounds like you were planning on implementing this since July 2015? It's Christmas Day - 2015, that's a long time Moz.

[lrafagnin]

Hi there,

We upgraded our site to CloudFlare Pro which supports all browsers that have TLS 1.0 and above - it gives the domain an static IP which means no SNI.
OpenSSL also connects to the domain without any issues but no luck in getting the crawler to work.
I also disabled Browser agent check security on CloudFlare which could be causing issues but that also didn't work.

Can anyone point me in the right direction?
Is there any way to get MOZ to crawl a site hosted on CloudFlare or is this a deadend?

If someone could provide a bit of code which I can use to test and debug the crawling request without having to use the crawling test tool that would be very helpful.

Cheers,
Lucas

[SEMbyotic]

I wish I had a solution for you, but I'm still using SNI via standard CloudFlare so I'm not able to leverage the Moz crawl at all. If you find a solution, it'd be great if you can post it here for myself and the others.

[lrafagnin]

So good news on my side, I was able to get the crawler test to work last night
My current setup is

• CloudFlare on Pro plan $20/month - you need this because it sets an static IP to your website
• On CF settings, disable BrowserAgent security checks - I still need to review this further but it seems CF blocks Moz's Rogerbot crawler because they cannot verify the user agent

I'll keep you posted on any new developments I have

[DavidLee]

Hi Lucas

Glad to hear you were able to find a workaround! One concerning bit is the disabling of the BrowserAgent security check. Would you be able to check with CloudFlare to see if this is recommended? Your findings may benefit many users that run into this issue and we can recommend this setup to other CF users.

Cheers!

[lrafagnin]

Hi David,

I did some more testing and it seems that the Crawler test was able to scan with the browser agent check on which is good news.
Moz will be indexing our site over the weekend and I can let you know then and confirm everything works as expected.

Cheers,
Lucas

[DavidLee]

Sounds great Lucas!

Will give these settings to another user to see if it works.

Thanks again!