Rogerbot and Dotbot blocked by WP Engine hosting

[wearehappymedia]

We recently discovered a problem with our MOZ account which led to checking our host's settings. Turns out that WP Engine (who we love to death) automatically BLOCK Rogerbot and Dotbot from crawling sites they host. What? Yeah exactly expect I was cursing a little too. So we went through the steps of getting that sorted one bot at a time but a week on and we're still getting 403 returns via OSE. This is more of a discussion than a question expect what I would like to see is a force crawl button so that we can test that we're no longer serving a 403 error from our host MOZ. I thought WP Engine were huge and I'm surprised this has not come up before but there's no info on it anywhere. Is there a way to force a mozbots crawl??

[John-Henry]

Hi Radi,

Thanks for posting your question, I'm John-Henry Scherck, Web Strategist for WP Engine.

After some testing on our end, we are seeing Rogerbot getting status 200s when it/he crawls sites on WP Engine, I even tested it on my personal site that also runs on WP Engine and was able to crawl it successfully. However, that doesn't mean everything is resolved - we want you to get the most out of Moz and have a great experience using WP Engine

So, the best thing to do is to submit a ticket through our user portal and we can look into the 403 issue for you.

However, I can confirm that Dotbot has been blocked by WP Engine for some time. At least as far back as 2012, probably earlier. Normally bots get blocked for being resource intensive, but we can take an internal look at allowing Dotbot for users who need to use it. If you want to use Dotbot on your site, please fill out a ticket in the user portal and our support and tech ops teams will look into for you.

Thanks for being a customer and please feel free to reach out to me in the future if you have any issues,

John-Henry

[DavidLee]

Hi Radi

Just wanted to add that dotbot only crawls once a month while rogerbot crawls weekly. So you will not see any new metrics on OSE until we update our next Index.

[wearehappymedia]

Thanks guys I'll wait until the month cycles through and see if anything has changed.