Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.

[N1ghteyes]

Hi James,

Thanks, I know it's not a 1 day fix, it's just somewhat frustrating to find that Moz doesn't yet support technology that is fast becoming standard, and thats been around for years. I look forwards to hearing when Moz does have an ETA for this! The Crawler is the main reason I created a Moz account, so i can't emphasize enough how much of an effect not having a long term solution at least announced soon is going to have on my decision to remain a Moz subscriber in the long term.

If it makes any difference, turns out Bing doesn't support it either, so something else to remember as more people start seeing crawler errors / lack of rankings that they may not understand!

http://webmasters.stackexchange.com/questions/77267/bing-and-lack-of-sni-support

And thank you for confirming the issue :).

[MrBlastit]

Is there an ETA for SNI support as yet?

[vascoas]

We are also running in the same problem. @James Kaiser, Could you please followup the status of this issue? Do you have any release date for this fix? tks.

[dynport]

Same here with AWS/Cloudfront. Please make an update. SNI is wildly supported and seems to have no impact on Google Bot or similar.

[dynport]

in response to Toby New, Bing Webmastertools/Crawler seems to handle the SNI well for us.

[leetracey]

Ditto. We've had to upgrade all our CF sites to pro, which costs us. SNI is so widely supported in the fabric of the Internet that it's really quite foundational to support it.

It's been a year now guys. What's the score with supporting it?

[Aurelien_TRF]

Hey guys,

Any news on this feature? or at least an ETA

[leetracey]

This was the response I got:

From Moz support:

Apologies about this! Unfortunately, the version of python we use does not support SNI and in order for us to upgrade to a version that does, it involves completely re-writing our crawler which can take some time. We have learned of more sites running into this issues, so it is a priority. I can't say exactly when we will have an update. Sorry for the inconvenience!

---

Not ideal, so I queried them about us having to pay $/month to Cloudflare to overcome a technical issue that should be supported (IMO) by Moz.

From Me:

Thanks for this: now I understand why it's not done.  the only things I have to say in reply are:

1. We now have to pay $XXX/month to Cloudflare mode that we need to in order for Moz to crawl the sites. Can you please arrange to compensate us for this in some fashion? Either some more projects available to us, or perhaps a couple of months free? I really don't believe it's fair to ask us to pay more on other services in order for Moz to be able to spider our website, considering what we are doing is totally standard.

2. The non-SLI issue has been going on for months, if not years. So cannot really be a priority for Moz. I'm not the only one stunned on the forums.

---

From Moz support:

I'm afraid we don't offer compensation for these SNI issues, unfortunately. It's a limitation of the technology right now. Rewriting crawlers takes a significant amount of time and effort, and although it is high on our priority list I still don't have any estimated release dates of when you might see a new one. Sorry about this!

[leetracey]

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.

[MaxxDesign]

Hi Lee,

OP here (Work account rather than personal),

Are the ones not being crawled behind SSL? As far as we can make out, Cloudflare should fall over to support non SNI crawlers for all CF Pro and above accounts. It seem however that RogerBot is spoofing a modern UA but not actually supporting modern SSL. The end result is that CF doesnt fall back properly and Moz can't crawl.

As i post this, im waiting on a Moz support response to a non SSL site behind CF thats not being crawled. Will post up if we work out what that is.

[leetracey]

Hi Toby,

All sites are SSL only (or with a redirect that makes it so). Seems that all our CF Pro sites are now being crawled.

Lee

[MaxxDesign]

Hi Lee,

Thats interesting.. Can you use the Crawl test tool on them? or does that fail?

Currently i have no campaignes setup as i just wanted to confirm if i could crawl or not. The crawl test tool fails for me with both SSL and non SSL sites on CF pro accounts...

[leetracey]

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

[MaxxDesign]

Hmm ok, thats something to go on then. Moz Support couldn't crawl when i asked them to check, and gave me nothing conclusive to work with other than 'check your logs' which were completely empty (no hits in access log or against CF firewall logs)

Guessing the crawltest is just broken at the moment then...

I'll setup a campaign and see if it works. Will update when i know one way or another.

Thanks!

[Quotall]

Hi All,

I've just found out that all GoDaddy Wordpress Managed websites (since Q3 2015) now have SNI enabled by default and it cannot be switched off.  Many of our clients use this hosting.

C'mon Moz - the world has moved on and we need this fixing fast.

[SYG]

Been a long time MOZ user and just found out this was an issue after implementing SSL on Hubspot.com

They are hosting our site and offered a no cost option to activate SSL, which I did, and now MOZ cannot crawl any longer.

Now anyone who is hosting their site at Hubspot and activates SSL will not be able to use Moz to crawl their site

Can't believe this is not being fixed asap by MOZ

[Virtualsolutions]

Is there an update or ETA for this fix? I have a client hosted on GoDaddy and cannot crawl the site due to CF

[MaxxDesign]

Hi, OP here (on a different Work account!)

If they upgrade to the pro level, CF SSL certs should now auto fallback for non SNI requests.

Not ideal, and Moz still need to fix the underlaying issue with the out of date version of Python that Rogerbot is written in... but it should provide a fix.

[Senson]

Just want to point out that all modern browsers support SNI and more and more sites will be using it as Google and Mozilla start to make their browsers warn users of unencrypted sites.

That this issue hasn't been fixed since it was brought to Moz.com's attention is disappointing.

[a_w00kie]

Is there an alternative to Moz that supports SNI ? not being able to crawl SNI enabled websites is a major problem.