Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.

[N1ghteyes]

Hi All,

Recently i've noticed that Moz can no longer crawl any of my wbsites that are behind cloudflare. After speaking to CF support, we have come to the conclusion that a recent change by CF to force SNI for all SSL domains is the issue.

As this is likely to affect almost everyone using Cloudflare could anyone please confirm this?

In the mean time, one of the websites currently blocked is on CF enterprise, so we've requested SNI be turned off. Will test crawl again when thats done to confirm SNI is the issue.

[jameskais]

Hello there! At this time we are not able to support SNI but it is on the table. We have to give Rogerbot a few tweaks and some internal upgrades before he is ready to crawl these pages. In the meantime you can disable SNI or forego crawls until we've got this rolled out. No ETA has been decided for this yet.

[N1ghteyes]

The main problem here is that i can't disable SNI unless i pay for a business account ($200 USD per website, per month), except for the enterprise account everything else is Pro which is more than enough for my clients (no way i could justify $200 a month just so moz can crawl).

SNI is a technology thats been supported since IE7 and is now standard in all browsers, so still not supporting it now is just a little insane....

But as i said, this is going to be effecting everyone using SSL under cloudflare (which is probably everyone using cloudflare) so could you guys please make this a priority? its not just an edge case issue.

[jameskais]

Toby - while I am unable to choose the prioritization we have actively investigated this and it is now with our product team for a resolution. This is correct that it is basically affecting all cloudflare users so it is large pain point so it will be prioritized as such.

I do wish we could get this fixed in a day but it is a larger project that needs to be addressed as such.

[N1ghteyes]

Hi James,

Thanks, I know it's not a 1 day fix, it's just somewhat frustrating to find that Moz doesn't yet support technology that is fast becoming standard, and thats been around for years. I look forwards to hearing when Moz does have an ETA for this! The Crawler is the main reason I created a Moz account, so i can't emphasize enough how much of an effect not having a long term solution at least announced soon is going to have on my decision to remain a Moz subscriber in the long term.

If it makes any difference, turns out Bing doesn't support it either, so something else to remember as more people start seeing crawler errors / lack of rankings that they may not understand!

http://webmasters.stackexchange.com/questions/77267/bing-and-lack-of-sni-support

And thank you for confirming the issue :).

[MrBlastit]

Is there an ETA for SNI support as yet?

[vascoas]

We are also running in the same problem. @James Kaiser, Could you please followup the status of this issue? Do you have any release date for this fix? tks.

[dynport]

Same here with AWS/Cloudfront. Please make an update. SNI is wildly supported and seems to have no impact on Google Bot or similar.

[dynport]

in response to Toby New, Bing Webmastertools/Crawler seems to handle the SNI well for us.

[leetracey]

Ditto. We've had to upgrade all our CF sites to pro, which costs us. SNI is so widely supported in the fabric of the Internet that it's really quite foundational to support it.

It's been a year now guys. What's the score with supporting it?

[Aurelien_TRF]

Hey guys,

Any news on this feature? or at least an ETA

[leetracey]

This was the response I got:

From Moz support:

Apologies about this! Unfortunately, the version of python we use does not support SNI and in order for us to upgrade to a version that does, it involves completely re-writing our crawler which can take some time. We have learned of more sites running into this issues, so it is a priority. I can't say exactly when we will have an update. Sorry for the inconvenience!

---

Not ideal, so I queried them about us having to pay $/month to Cloudflare to overcome a technical issue that should be supported (IMO) by Moz.

From Me:

Thanks for this: now I understand why it's not done.  the only things I have to say in reply are:

1. We now have to pay $XXX/month to Cloudflare mode that we need to in order for Moz to crawl the sites. Can you please arrange to compensate us for this in some fashion? Either some more projects available to us, or perhaps a couple of months free? I really don't believe it's fair to ask us to pay more on other services in order for Moz to be able to spider our website, considering what we are doing is totally standard.

2. The non-SLI issue has been going on for months, if not years. So cannot really be a priority for Moz. I'm not the only one stunned on the forums.

---

From Moz support:

I'm afraid we don't offer compensation for these SNI issues, unfortunately. It's a limitation of the technology right now. Rewriting crawlers takes a significant amount of time and effort, and although it is high on our priority list I still don't have any estimated release dates of when you might see a new one. Sorry about this!

[leetracey]

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.

[MaxxDesign]

Hi Lee,

OP here (Work account rather than personal),

Are the ones not being crawled behind SSL? As far as we can make out, Cloudflare should fall over to support non SNI crawlers for all CF Pro and above accounts. It seem however that RogerBot is spoofing a modern UA but not actually supporting modern SSL. The end result is that CF doesnt fall back properly and Moz can't crawl.

As i post this, im waiting on a Moz support response to a non SSL site behind CF thats not being crawled. Will post up if we work out what that is.

[leetracey]

Hi Toby,

All sites are SSL only (or with a redirect that makes it so). Seems that all our CF Pro sites are now being crawled.

Lee

[MaxxDesign]

Hi Lee,

Thats interesting.. Can you use the Crawl test tool on them? or does that fail?

Currently i have no campaignes setup as i just wanted to confirm if i could crawl or not. The crawl test tool fails for me with both SSL and non SSL sites on CF pro accounts...

[leetracey]

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

[MaxxDesign]

Hmm ok, thats something to go on then. Moz Support couldn't crawl when i asked them to check, and gave me nothing conclusive to work with other than 'check your logs' which were completely empty (no hits in access log or against CF firewall logs)

Guessing the crawltest is just broken at the moment then...

I'll setup a campaign and see if it works. Will update when i know one way or another.

Thanks!

[Quotall]

Hi All,

I've just found out that all GoDaddy Wordpress Managed websites (since Q3 2015) now have SNI enabled by default and it cannot be switched off.  Many of our clients use this hosting.

C'mon Moz - the world has moved on and we need this fixing fast.

[SYG]

Been a long time MOZ user and just found out this was an issue after implementing SSL on Hubspot.com

They are hosting our site and offered a no cost option to activate SSL, which I did, and now MOZ cannot crawl any longer.

Now anyone who is hosting their site at Hubspot and activates SSL will not be able to use Moz to crawl their site

Can't believe this is not being fixed asap by MOZ