Does Rogerbot support SNI? - And if not All sites behind Cloudflare are probably uncrawlable.

[leetracey]

Interestingly, I do have some of our sites on CF Pro ($5/month plan) that are being crawled, but some not (yet at least). Will post if this changes.

[MaxxDesign]

Hi Lee,

OP here (Work account rather than personal),

Are the ones not being crawled behind SSL? As far as we can make out, Cloudflare should fall over to support non SNI crawlers for all CF Pro and above accounts. It seem however that RogerBot is spoofing a modern UA but not actually supporting modern SSL. The end result is that CF doesnt fall back properly and Moz can't crawl.

As i post this, im waiting on a Moz support response to a non SSL site behind CF thats not being crawled. Will post up if we work out what that is.

[leetracey]

Hi Toby,

All sites are SSL only (or with a redirect that makes it so). Seems that all our CF Pro sites are now being crawled.

Lee

[MaxxDesign]

Hi Lee,

Thats interesting.. Can you use the Crawl test tool on them? or does that fail?

Currently i have no campaignes setup as i just wanted to confirm if i could crawl or not. The crawl test tool fails for me with both SSL and non SSL sites on CF pro accounts...

[leetracey]

The crawl test currently fails, it just says error submitting the crawl test. Very odd.

[MaxxDesign]

Hmm ok, thats something to go on then. Moz Support couldn't crawl when i asked them to check, and gave me nothing conclusive to work with other than 'check your logs' which were completely empty (no hits in access log or against CF firewall logs)

Guessing the crawltest is just broken at the moment then...

I'll setup a campaign and see if it works. Will update when i know one way or another.

Thanks!

[Quotall]

Hi All,

I've just found out that all GoDaddy Wordpress Managed websites (since Q3 2015) now have SNI enabled by default and it cannot be switched off.  Many of our clients use this hosting.

C'mon Moz - the world has moved on and we need this fixing fast.

[SYG]

Been a long time MOZ user and just found out this was an issue after implementing SSL on Hubspot.com

They are hosting our site and offered a no cost option to activate SSL, which I did, and now MOZ cannot crawl any longer.

Now anyone who is hosting their site at Hubspot and activates SSL will not be able to use Moz to crawl their site

Can't believe this is not being fixed asap by MOZ

[Virtualsolutions]

Is there an update or ETA for this fix? I have a client hosted on GoDaddy and cannot crawl the site due to CF

[MaxxDesign]

Hi, OP here (on a different Work account!)

If they upgrade to the pro level, CF SSL certs should now auto fallback for non SNI requests.

Not ideal, and Moz still need to fix the underlaying issue with the out of date version of Python that Rogerbot is written in... but it should provide a fix.

[Senson]

Just want to point out that all modern browsers support SNI and more and more sites will be using it as Google and Mozilla start to make their browsers warn users of unencrypted sites.

That this issue hasn't been fixed since it was brought to Moz.com's attention is disappointing.

[a_w00kie]

Is there an alternative to Moz that supports SNI ? not being able to crawl SNI enabled websites is a major problem.

[orlando-rehash]

Any updates to this? I just started the Moz 30-day trial, and if this isn't available (I'm not going to pay for Cloudflare pro), I'm going to cancel Moz when my free 30 days are up.

[Pentangle]

Raised in "Jun 03, 2015" and still it doesn't work?

Seriously need sorting or we'll be off to a competitor.

[kent.dagnall]

+1 We must also use another service if Moz cannot fix their crawler issues. Roger should be one of the best bots out there for what Moz charges. Come to find out it's built on an ancient Python version and needs to be completely rewritten.

[Rachelle_Henderson]

Hi! Has this been taken care of yet? It's February 2017 and we also use cloudflare, and right now I can't pull any rankings since Moz can't crawl our sites. We love Moz and do not want to cancel the service, but this is ridiculous.

[jameskais]

Hi Rachelle, I've added your name to our beta crawl list and hopefully we can get you added shortly. As a note SNI does not impact ranking collections but only site crawls. Please let help@moz.com know if you are seeing issues with rankings collections.

[jameskais]

Hi Kent, I've added your name to our beta crawl list and hopefully we can get you added shortly. We are well underway with that re-write and will get you on a crawl that works for you shortly!

[Rachelle_Henderson]

Hi James, thank you so much! We are seeing issues with rankings for our 2 accounts using Cloudflare. I gave myself a heart attack thinking that the client's site tanked and then dug around to find the issue in the forums. It may be worth it to create a blog to let the public know what's going on so other webmasters don't freak out too since it seems a lot of us are running into this error.

[acreedon]

Hi James, this is also a big problem for us. Can you add me to the beta group - aine@npqmag.org? Otherwise we may have to go elsewhere for our SEO tools.