Crawl Diagnostics Report Lacks Information

[loopyal]

When I look at the crawl diagnostics, SEOMoz tells me there are 404 errors.

This is understandable, because some pages were removed.

What this report doesn't tell me is how those pages were discovered.

This is a very important piece of information, because it would tell me there are links pointing to those pages, either internal or external. I believe the internal links have been removed.

If the report told me how if found the link, I would be able to take immediate action. Without that information, I have to go so a lot of investigation. And when you have a million pages, that isn't easy.

Some possibilities:

• The crawler remembered the page from the previous crawl.
• There was a link from an index page - i.e. it is in the database still
• There was an individual link from another story - so now there are broken links
• Ditto, but it in on a static index page
• The link was from an external source - I need to make a redirect

Am I missing something, or is this a feature the SEO Moz crawler doesn't have yet?

What can I do (other than check all my pages) to discover this?

[Red_Mud_Rookie]

This may be a silly question, but have you got Google Webmaster tools installed? That will show you the source of the errors.

If your site is on IIS then you should also use the awesome IIS SEO toolkit provided by Microsoft for free.

[MichaelCorfman]

If you download the csv file for the crawl you can sort it by http status to get all of the 404 errors together.  Then there is a specific column that contains the referrer that provides the information you are after.

[loopyal]

Thank you Ralph.

Yes, had it for years. So is this a GWT report? I thought it was SEOMoz !

No not IIS, Linux.

[Red_Mud_Rookie]

I think it's the SEOMoz crawler, but what I have found is that the error reports are limited here whereas GWT is much bigger and shows the links leading to the error. My guess is that SEOMoz limit the number of crawl errors they show due to limitations set on their crawler i.e. while their crawl is comprehensive, it's not going to capture what Google does.

[loopyal]

OK thank you, Ralph

I can work on that.