804 HTTPS (SSL) error

[Arizuka]

Same problem for us. We are glad to use moz but we have migrated from http to https and nothing works. Do you have any clue if it will be or not in your roadmap developments?

[micvitale]

same problem for me.

I am working with https and cloudflare.

what if I disable cloudflare just for the crawl test? would it work? how could I disable it temporarily?

thank you

[ClockSoftware]

Same here.

We use CloudFront and receive the 804 error response. We consider to cancel our subscription  as we get no data and there is no information if and when a solution can be expected.

[bluedotproductions]

We too use SNI for our certificates on my co-located server with a top host.

It sounds like you were planning on implementing this since July 2015? It's Christmas Day - 2015, that's a long time Moz.

[lrafagnin]

Hi there,

We upgraded our site to CloudFlare Pro which supports all browsers that have TLS 1.0 and above - it gives the domain an static IP which means no SNI.
OpenSSL also connects to the domain without any issues but no luck in getting the crawler to work.
I also disabled Browser agent check security on CloudFlare which could be causing issues but that also didn't work.

Can anyone point me in the right direction?
Is there any way to get MOZ to crawl a site hosted on CloudFlare or is this a deadend?

If someone could provide a bit of code which I can use to test and debug the crawling request without having to use the crawling test tool that would be very helpful.

Cheers,
Lucas

[SEMbyotic]

I wish I had a solution for you, but I'm still using SNI via standard CloudFlare so I'm not able to leverage the Moz crawl at all. If you find a solution, it'd be great if you can post it here for myself and the others.

[lrafagnin]

So good news on my side, I was able to get the crawler test to work last night
My current setup is

• CloudFlare on Pro plan $20/month - you need this because it sets an static IP to your website
• On CF settings, disable BrowserAgent security checks - I still need to review this further but it seems CF blocks Moz's Rogerbot crawler because they cannot verify the user agent

I'll keep you posted on any new developments I have

[DavidLee]

Hi Lucas

Glad to hear you were able to find a workaround! One concerning bit is the disabling of the BrowserAgent security check. Would you be able to check with CloudFlare to see if this is recommended? Your findings may benefit many users that run into this issue and we can recommend this setup to other CF users.

Cheers!

[lrafagnin]

Hi David,

I did some more testing and it seems that the Crawler test was able to scan with the browser agent check on which is good news.
Moz will be indexing our site over the weekend and I can let you know then and confirm everything works as expected.

Cheers,
Lucas

[DavidLee]

Sounds great Lucas!

Will give these settings to another user to see if it works.

Thanks again!

[Pablostevens71]

I am also having this issue.  I have a GoDaddy SSL on their Managed Wordpress Hosting and just started getting the 804 error in December 2015.  Are there any other common things to check other than on GoDaddy, there's not really much to configure there and everything else looks OK.

[DavidLee]

If SNI is not enabled with GoDaddy, the other issue would be related to not having intermediate SSL certificates setup.

[KyleMamaBird]

Is there any updates on the issues with Cloudflare?

We like Moz but really would like this resolved.

[crowdsite.com]

Hi Moz Users,

Shouldn't it be fixed by now? We got the same problem at https://www.crowdsite.com

We also use SNI on our servers, is there a ATE to fix?

[DavidLee]

The fix we are working on involves a complete re-write of our crawler which will take time to complete. I'm afraid I do not have an ETA but will keep this thread posted with any updates.

[lrafagnin]

Guys,

I don't work for Moz and I I know it sucks that they don't support SNI, even if every major browser since IE6 does it, but it isn't a hard fix to add a dedicated IP address for SSL.
If you use AWS you can get CloudFront's dedicated IP for $600/month or you can add CloudFlare Pro for $20/month.
Every hosting company I know provides dedicated IP for SSL so the only issue here is the additional cost.
You guys are paying $100/month for Moz so might as well add another $20 (per domain) and forget about it
Happy to give some guidance with the setup if anyone isn't technical, you should also be able to switch the DNS with zero downtime.

Cheers,
Lucas

[crowdsite.com]

Hi Lucas,

It is possible with most hosting providers, we use Google Cloud Platform and i believe it's possible there. But it involves changing all the certificates from all sites (Our site runs on 12 domains), it's a lot unnecessary work.

Cheers, Roel

[lrafagnin]

Hi Roel,
I totally agree with you, but Moz seems to be really slow to move on this (it's been over 4 years since python released SNI support).
If you really want to use Moz I would strongly suggest just making the changes on your CDN instead of waiting for them to make a move, and you can always revert the setup once they release SNI support.
Google Cloud supports dedicated IP (they call it a VIP - virtual IP) and it costs $39/month.
You can find some info on how to setup here, I have never done it but seems pretty straight forward
https://cloud.google.com/appengine/docs/python/console/using-custom-domains-and-ssl

[empleostampico]

Hi Lucas,

I am using cloudflare and can't get moz crawler to run even though I have disabled browser integrity and flushed cache. Is there anything else you had to do to have this work?

Many thanks

[lrafagnin]

Hi Rodrigo,

You need to be on CloudFlare Pro which is $20/month to get a dedicated IP address for SSL which will solve your problem.
You don't need to disable browser integrity but you might need to wait a few minutes until CloudFlare adds the IP to your domain. You can check it with a tool like https://www.whatsmydns.net/