Pages with 301 redirects showing as 200 when crawled using RogerBot

[Welford-Media]

Hi guys,

I recently did an audit for a client and ran a crawl on the site using RogerBot. We quickly noticed that all but one page was showing as status code 200, but we knew that there were a lot of 301 redirects in place. When our developers checked it, they saw the pages as 301s, as did the Moz toolbar.

If page A redirected to page B, our developers and the Moz toolbar saw page A as 301 and page B as 200. However the crawl showed both page A and page B as 200.

Does anyone have any idea why the crawl may have been showing the status codes as 200? We've checked and the redirect is definitely in place for the user, but our worry is that there could be an issue with duplicate content if a crawler isn't picking up on the 301 redirect.

Thanks!

[DavidLee]

Hi Gareth!

This will be hard to troubleshoot without knowing the full URLs. Could you send us the exact URLs you are working with to help@moz.com so we can take a look?

[matthew.dimmett]

I have this situation on my sites too. This might not be right, but to double check that my 301s aren't being seen as duplicates of the 200s that they are redirecting to I'll filter my crawl test tool report from Moz to only show the records where the HTTP status code doesn't equal 200 and duplicate page content is yes.

I have a couple different crawlers that I use and they each work a little differently. I think some might look at your situation and return page A as a 200 just because when it resolves to page B it is a 200, where other crawlers are more sensitive and see page A as a 301.

I hope that helps!