804 HTTPS (SSL) error

[lrafagnin]

Hi there,

We upgraded our site to CloudFlare Pro which supports all browsers that have TLS 1.0 and above - it gives the domain an static IP which means no SNI.
OpenSSL also connects to the domain without any issues but no luck in getting the crawler to work.
I also disabled Browser agent check security on CloudFlare which could be causing issues but that also didn't work.

Can anyone point me in the right direction?
Is there any way to get MOZ to crawl a site hosted on CloudFlare or is this a deadend?

If someone could provide a bit of code which I can use to test and debug the crawling request without having to use the crawling test tool that would be very helpful.

Cheers,
Lucas

[SEMbyotic]

I wish I had a solution for you, but I'm still using SNI via standard CloudFlare so I'm not able to leverage the Moz crawl at all. If you find a solution, it'd be great if you can post it here for myself and the others.

[lrafagnin]

So good news on my side, I was able to get the crawler test to work last night
My current setup is

• CloudFlare on Pro plan $20/month - you need this because it sets an static IP to your website
• On CF settings, disable BrowserAgent security checks - I still need to review this further but it seems CF blocks Moz's Rogerbot crawler because they cannot verify the user agent

I'll keep you posted on any new developments I have

[DavidLee]

Hi Lucas

Glad to hear you were able to find a workaround! One concerning bit is the disabling of the BrowserAgent security check. Would you be able to check with CloudFlare to see if this is recommended? Your findings may benefit many users that run into this issue and we can recommend this setup to other CF users.

Cheers!

[lrafagnin]

Hi David,

I did some more testing and it seems that the Crawler test was able to scan with the browser agent check on which is good news.
Moz will be indexing our site over the weekend and I can let you know then and confirm everything works as expected.

Cheers,
Lucas

[DavidLee]

Sounds great Lucas!

Will give these settings to another user to see if it works.

Thanks again!

[Pablostevens71]

I am also having this issue.  I have a GoDaddy SSL on their Managed Wordpress Hosting and just started getting the 804 error in December 2015.  Are there any other common things to check other than on GoDaddy, there's not really much to configure there and everything else looks OK.

[DavidLee]

If SNI is not enabled with GoDaddy, the other issue would be related to not having intermediate SSL certificates setup.

[KyleMamaBird]

Is there any updates on the issues with Cloudflare?

We like Moz but really would like this resolved.

[crowdsite.com]

Hi Moz Users,

Shouldn't it be fixed by now? We got the same problem at https://www.crowdsite.com

We also use SNI on our servers, is there a ATE to fix?

[DavidLee]

The fix we are working on involves a complete re-write of our crawler which will take time to complete. I'm afraid I do not have an ETA but will keep this thread posted with any updates.

[lrafagnin]

Guys,

I don't work for Moz and I I know it sucks that they don't support SNI, even if every major browser since IE6 does it, but it isn't a hard fix to add a dedicated IP address for SSL.
If you use AWS you can get CloudFront's dedicated IP for $600/month or you can add CloudFlare Pro for $20/month.
Every hosting company I know provides dedicated IP for SSL so the only issue here is the additional cost.
You guys are paying $100/month for Moz so might as well add another $20 (per domain) and forget about it
Happy to give some guidance with the setup if anyone isn't technical, you should also be able to switch the DNS with zero downtime.

Cheers,
Lucas

[crowdsite.com]

Hi Lucas,

It is possible with most hosting providers, we use Google Cloud Platform and i believe it's possible there. But it involves changing all the certificates from all sites (Our site runs on 12 domains), it's a lot unnecessary work.

Cheers, Roel

[lrafagnin]

Hi Roel,
I totally agree with you, but Moz seems to be really slow to move on this (it's been over 4 years since python released SNI support).
If you really want to use Moz I would strongly suggest just making the changes on your CDN instead of waiting for them to make a move, and you can always revert the setup once they release SNI support.
Google Cloud supports dedicated IP (they call it a VIP - virtual IP) and it costs $39/month.
You can find some info on how to setup here, I have never done it but seems pretty straight forward
https://cloud.google.com/appengine/docs/python/console/using-custom-domains-and-ssl

[empleostampico]

Hi Lucas,

I am using cloudflare and can't get moz crawler to run even though I have disabled browser integrity and flushed cache. Is there anything else you had to do to have this work?

Many thanks

[lrafagnin]

Hi Rodrigo,

You need to be on CloudFlare Pro which is $20/month to get a dedicated IP address for SSL which will solve your problem.
You don't need to disable browser integrity but you might need to wait a few minutes until CloudFlare adds the IP to your domain. You can check it with a tool like https://www.whatsmydns.net/

[empleostampico]

Hi Lucas,

Thank you for the quick response. I am using cloudflare plus at moment, and I do have a dedicated IP and SSL with my hosting provider. Would that not be the same as above? I used the dns tool and I do get a constant IP address.

Again, thank you very much for helping on this! - I understand it is not your duty to do so.

[lrafagnin]

Sorry Jorge, not sure why I called you Rodrigo...
No worries, happy to help, it drove me crazy too

So, I reviewed my CloudFlare settings, I got firewall security level set to low, no mobile redirect but I do have a redirect on HTTP to HTTPS on all pages.
I also have IPv6 on although I doubt that would make any difference.

What I can off is that your application might be blocking the requests maybe, not CF.
Do you run custom code or is it an off the shelf solution like Wordpress?
And do you run it on AWS? EC2 instances or though an ELB?

I would suggest as your next step to check if you see any requests from Moz to your server
Just trigger a Crawl test on this link and wait a few hours, you will get the report by email
https://moz.com/researchtools/crawl-test

Let us know how it goes

[empleostampico]

Hi Lucas,

You're definitively not going crazy about my name! My business partner created the Moz account and I just changed the name of moz profile to mine as I'll be managing this..hence the confusion.

I am running a wordpress install hosted with siteground and paid for cloudflare plus via siteground, all the site redirects to https as well, no mobile redirect.

I just activated IPV6 compatibility, HTTP/2 + SPDY, set firewall security level to LOW and gave the moz crawl test a try, I'll post my results tomorrow.

By the way, I just took a look at your site and I'm impressed, it's awesome, my website is a local job portal so there's a few good features you have I'm jealous about! I'm not a designer so we went with a "premium" wp template and started tweaking from there. Out of curiosity is your site a custom design solution or is there a way I can get my hands on that theme?  

Many thanks again,

Jorge

[empleostampico]

UPDATE: Nope, it didn't work, the CSV file from Moz has only one entry which is the 804 https error.
, have you any other suggestions to tweak cloudflare for this?