Moz & Xenu Link Sleuth unable to crawl a website (403 error)

[ZaddleMarketing]

It could be that I am missing something really obvious however we are getting the following error when we try to use the Moz tool on a client website. (I have read through a few posts on 403 errors but none that appear to be the same problem as this)

Moz Result

Title 403 : Error

Meta Description 403 Forbidden

Meta Robots_Not present/empty_

Meta Refresh_Not present/empty_

Xenu Link Sleuth Result

Broken links, ordered by link:

error code: 403 (forbidden request), linked from page(s):

Thanks in advance!

[matbennett]

403s are tricky to diagnose because they, by their very nature, don't tell you much.  They're sort of the server equivalent of just shouting "NO!".

You say Moz & Xenu are receiving the 403. I assume that it loads properly from a browser.

I'd start looking at the .htaccess .  Any odd deny statements in there?  It could be that an IP range or user agent is blocked.  Some people like to block common crawlers (Not calling Roger names there).  Check the robots.txt whilst you are there, although that shouldn't return a 403 really.

[ZaddleMarketing]

Hi Mat

Thanks for the reply - robots.txt file is as follows:

## The following are infinitely deep trees
User-agent: *
Disallow: /cgi-bin
Disallow: /cms/events
Disallow: /cms/latest
Disallow: /cms/cookieprivacy
Disallow: /cms/help
Disallow: /site/services/megamenu/
Disallow: /site/mobile/

I can't get access to the .htaccess file at present (we're not the developers)

Anyone else any thoughts? Weirdly I can get Screaming Frog info back on the site :-/

[ChiarynMiranda]

Hey There,

The robots.txt shouldn't really affect 403s; you would actually get a "blocked by robots.txt" error if that was the cause. Your server is basically telling us that we are not authorized to access your site. I agree with Mat that we are most likely being blocked in the htaccess file. It may be that your server is flagging our crawler and Xenu's crawler as troll crawlers or something along those lines. I ran a test on your URL using a non-existent crawler, Rogerbot with a capital R, and got a 200 status code back but when I run the test with our real crawler, rogerbot with a lowercase r, I get the 403 error (http://screencast.com/t/Sv9cozvY2f01). This tells me that the server is specifically blocking our crawler, but not all crawlers in general.

I hope this helps. Let me know if you have any other questions.

Chiaryn
Help Team Ninja

[ZaddleMarketing]

Awesome - thank you.

Kind Regards

Liam

[ZaddleMarketing]

Hi Chiaryn

The sage continues....this is the response my client got back from the developers - please could you let me have the answers to the two questions?

Apparently as part of their ‘SAF’ (?) protocols, if the IT director sees a big spike in 3^rd party products trawling the site he will block them! They did say that they use moz too.  What they’ve asked me to get from moz is:

• Moz IP address/range
• Level of throttling they will use

I would question that if THEY USE MOZ themselves why would they need these answers but if I go back with that I will be going around in circles - any chance of letting me know the answer(s)?

Thanks in advance.

Liam

[ChiarynMiranda]

Hey Liam,

Thanks for following up. Unfortunately, we use thousands of dynamic IPs through Amazon Web Services to run our crawler and the IP would change from crawl to crawl. We don't even have a set range for the IPs we use through AWS.

As for throttling, we don't have a set throttle. We try to space out the server hits enough to not bring down the server, but then hit the server as often as necessary in order to crawl the full site or crawl limit in a reasonable amount of time. We try to find a balance between hitting the site too hard and having extremely long crawl times. If the devs are worried about how often we hit the server, they can add a crawl delay of 10 to the robots.txt to throttle the crawler. We will respect that delay.

If the devs use Moz, as well, they would also be getting a 403 on their crawl because the server is blocking our user agent specifically. The server would give the same status code regardless of who has set up the campaign.

I'm sorry this information isn't more specific. Please let me know if you need any other assistance.

Chiaryn