Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.

[dynport]

Same here with AWS/Cloudfront. Please make an update. SNI is wildly supported and seems to have no impact on Google Bot or similar.

[dynport]

in response to Toby New, Bing Webmastertools/Crawler seems to handle the SNI well for us.

[leetracey]

Ditto. We've had to upgrade all our CF sites to pro, which costs us. SNI is so widely supported in the fabric of the Internet that it's really quite foundational to support it.

It's been a year now guys. What's the score with supporting it?

[Aurelien_TRF]

Hey guys,

Any news on this feature? or at least an ETA

[leetracey]

This was the response I got:

From Moz support:

Apologies about this! Unfortunately, the version of python we use does not support SNI and in order for us to upgrade to a version that does, it involves completely re-writing our crawler which can take some time. We have learned of more sites running into this issues, so it is a priority. I can't say exactly when we will have an update. Sorry for the inconvenience!

---

Not ideal, so I queried them about us having to pay $/month to Cloudflare to overcome a technical issue that should be supported (IMO) by Moz.

From Me:

Thanks for this: now I understand why it's not done.  the only things I have to say in reply are:

1. We now have to pay $XXX/month to Cloudflare mode that we need to in order for Moz to crawl the sites. Can you please arrange to compensate us for this in some fashion? Either some more projects available to us, or perhaps a couple of months free? I really don't believe it's fair to ask us to pay more on other services in order for Moz to be able to spider our website, considering what we are doing is totally standard.

2. The non-SLI issue has been going on for months, if not years. So cannot really be a priority for Moz. I'm not the only one stunned on the forums.

---

From Moz support:

I'm afraid we don't offer compensation for these SNI issues, unfortunately. It's a limitation of the technology right now. Rewriting crawlers takes a significant amount of time and effort, and although it is high on our priority list I still don't have any estimated release dates of when you might see a new one. Sorry about this!

[leetracey]

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.

[MaxxDesign]

Hi Lee,

OP here (Work account rather than personal),

Are the ones not being crawled behind SSL? As far as we can make out, Cloudflare should fall over to support non SNI crawlers for all CF Pro and above accounts. It seem however that RogerBot is spoofing a modern UA but not actually supporting modern SSL. The end result is that CF doesnt fall back properly and Moz can't crawl.

As i post this, im waiting on a Moz support response to a non SSL site behind CF thats not being crawled. Will post up if we work out what that is.

[leetracey]

Hi Toby,

All sites are SSL only (or with a redirect that makes it so). Seems that all our CF Pro sites are now being crawled.

Lee

[MaxxDesign]

Hi Lee,

Thats interesting.. Can you use the Crawl test tool on them? or does that fail?

Currently i have no campaignes setup as i just wanted to confirm if i could crawl or not. The crawl test tool fails for me with both SSL and non SSL sites on CF pro accounts...

[leetracey]

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

[MaxxDesign]

Hmm ok, thats something to go on then. Moz Support couldn't crawl when i asked them to check, and gave me nothing conclusive to work with other than 'check your logs' which were completely empty (no hits in access log or against CF firewall logs)

Guessing the crawltest is just broken at the moment then...

I'll setup a campaign and see if it works. Will update when i know one way or another.

Thanks!

[Quotall]

Hi All,

I've just found out that all GoDaddy Wordpress Managed websites (since Q3 2015) now have SNI enabled by default and it cannot be switched off.  Many of our clients use this hosting.

C'mon Moz - the world has moved on and we need this fixing fast.

[SYG]

Been a long time MOZ user and just found out this was an issue after implementing SSL on Hubspot.com

They are hosting our site and offered a no cost option to activate SSL, which I did, and now MOZ cannot crawl any longer.

Now anyone who is hosting their site at Hubspot and activates SSL will not be able to use Moz to crawl their site

Can't believe this is not being fixed asap by MOZ

[Virtualsolutions]

Is there an update or ETA for this fix? I have a client hosted on GoDaddy and cannot crawl the site due to CF

[MaxxDesign]

Hi, OP here (on a different Work account!)

If they upgrade to the pro level, CF SSL certs should now auto fallback for non SNI requests.

Not ideal, and Moz still need to fix the underlaying issue with the out of date version of Python that Rogerbot is written in... but it should provide a fix.

[Senson]

Just want to point out that all modern browsers support SNI and more and more sites will be using it as Google and Mozilla start to make their browsers warn users of unencrypted sites.

That this issue hasn't been fixed since it was brought to Moz.com's attention is disappointing.

[a_w00kie]

Is there an alternative to Moz that supports SNI ? not being able to crawl SNI enabled websites is a major problem.

[orlando-rehash]

Any updates to this? I just started the Moz 30-day trial, and if this isn't available (I'm not going to pay for Cloudflare pro), I'm going to cancel Moz when my free 30 days are up.

[Pentangle]

Raised in "Jun 03, 2015" and still it doesn't work?

Seriously need sorting or we'll be off to a competitor.

[kent.dagnall]

+1 We must also use another service if Moz cannot fix their crawler issues. Roger should be one of the best bots out there for what Moz charges. Come to find out it's built on an ancient Python version and needs to be completely rewritten.