Moz can't crawl domain due to IP Geo redirect loop

[SalvationArmy]

Hi,

I'm trying to crawl our domain www.salvationarmy.org.au via my Moz account and it only ever returns results for one page when it should be crawling more than 3,000 pages.

In talking to support, they have said that because of the redirect we have in place it is creating a 302 loop and therefore not delivering results.  Usually in this case I would obtain Moz's IP addresses and add them to the redirect settings as an exception, but Moz have said they use cloud-based services for crawling so the IPs change all the time.

Does anyone have any idea how to solve this issue?  At this point I've paid for a year's subscription to a product I can't use.

Thanks,

Mel

[TomVolpe]

Hi,

If you have manually set up your geo redirect in your htaccess then you could modify your rules to redirect only if not Moz’s crawler (rogerbot) like this:

uk redirect

RewriteCond %{HTTP_USER_AGENT} !=rogerbot

RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^GB$

RewriteRule ^(.*)$ http://uk.abcd.com$1 [L]

Which means both conditions must be satisfied before the redirect happens, the user agent must not be rogerbot, and then it checks the country code. You may have to adjust it a bit depending on your setup but it’s just the same as adding an exception based on IP, so if you could already do that you can set up a user agent condition just as easily.

If you’re using PHP you could use $_SERVER['HTTP_USER_AGENT'] wrap your geoip function with something like:

if($_SERVER['HTTP_USER_AGENT'] != 'rogerbot' ){

You will have to check if it is not empty before you implement it (or work it into your code) as some servers have $_SERVER['HTTP_USER_AGENT'] as not set.

Thanks, hope that gives you a few ideas to try!

Tom