803 Errors, how to deal with this?

[A_Fotografy]

Quick question. My server guys asking with what IP address MOZ is crawling my site. WP-Engine is saying that maybe they have blocked that IP address.

How do I find this IP address?

A

[mediawyse]

Unfortunately, and I could be wrong here, but I believe that Moz doesn't use anything close to a static IP address for their crawl and instead uses thousands which constantly change when they run a crawl. It's all hosted from the cloud. So there is honestly no way to get a "range" for you to pass along.

Instead, all you can do I believe is make sure you whitelist the useragent which I believe is just "roger."

[KristinaKeyser]

Hi there! Kristina from Moz's Help Team here.

I just wanted to quickly interject and say that Casey is right! Unfortunately, we do not use a static IP address or range of IP addresses, as we have designed our crawler to have a dynamic approach. This means we use thousands of dynamic IP addresses which will change each time we run a crawl. This approach gives us the best dynamic view of the web, but it can make us incompatible with some servers or hosting providers.

The best option I'd be able to suggest would be to identify our crawler by User-agent: rogerbot

I hope this helps! As always, you can connect with our team about any issues you're seeing in the tools by sending a note to help@moz.com.

Have a great day!
-Kristina

[A_Fotografy]

So got the response from Moz about this issue and seems like I can't do anything about and it is common problem with MOZ Roger boot.

My guess this is for both 803 and 804 errors.

" Unfortunately at this time our crawler isn't compatible with Server Name Indication (SNI). In general, SNI is a totally acceptable security configuration, but our crawler simply isn't equipped to be able to handle these settings.

If you are able to disable the SNI configuration on your site, then we should be able to crawl it normally. While it's something that we'd love to be able to accommodate for the future, we don't yet have the resources to be able to make those changes. "

Spoke with WP-Engine guys and they said it would be madness to disable this security option as it would definitely broke websites.

Sad face me and not sure what to do now.

A

[mediawyse]

I think the recourse here is simple. IGNORE IT.

Again, this is a "Moz Crawl" prominence issue -- they just assign a higher value to it than most. I seriously doubt this is hurting you. Go over to www.ssllabs.com and run your SSL configuration through that tool there. If you returns a high score, you're good.

[A_Fotografy]

This is the one MOZ did tests with and they said that Roger can't crawl site because of that SNI support issue. I got A- grade so I guess all should be good.

Thanks. Casey.

[A_Fotografy]

So errors keep creeping up and I have noticed significant drop in my website rankings too. So there is definitely a problem.

Seems like here is many threads on what is the problem, but no solution how exactly to fix it.

Even MOZ guys are shaking heads and not sure what to do about it.

Sad face me.

A

[A_Fotografy]

Is there another tool I could use to crawl my whole website to understand if that is only MOZ problem or could be something bigger.

[A_Fotografy]

Another thing I noticed that when all this 803/804 issue started my crawled page amount almost doubled.

I have a photography website with loads of photos. So it treats all photos as a separate pages and brings all these 8xx errors.

Should I be doing something about this?

Thanks,

A

[mediawyse]

I use Screaming Frog and it's bad-ass. But you have to pay for it.

A free option for you may be Xenu Link Sleuth. Also a quality crawler.

[A_Fotografy]

I have screaming frog too. But it crawls only website pages. All of those are fine.

Pretty much all errors I get are for images on my website. So basically for separate files.

Also wanted to ask.

Could this type of 8xx error problem be caused by one of the plugins in WP? Around that time I installed Optimise plugin (which was suggested by WP Engine to speed up my site)

Thanks Casey.