Where does the crawler find the urls?

[Fammy]

The SEO Moz crawler has found a number of 500 error pages, and 404s etc which is very useful

however some of the urls are weird/broken formats we don't recognise and nobody remembers ever using - not weird enough to imply hacking, but something broken in the CMS

Is there anyway to find out where the crawler found these urls? I can patch up and redirect the end result as best I can but I would prefer to fix plug the leak

thanks

[Alex-Harford]

You can pop the links into the free Xenu Link Sleuth* - after you've done a crawl just right-click on the URL you're interested in and click 'URL Properties' - you'll see any inlinks it finds listed there. Depending on the size of your site, it could take a while for the crawl to complete.

You could try the link: property in Google first, though it won't be as thorough as Xenu.

*If you haven't seen it before, don't worry about how the Xenu website looks - the software is kosher - as recommended by many SEOmoz staff. Screaming Frog is a paid alternative (with a limited free version).

[Fammy]

thanks for the tips. It is a little frustrating that the information I need has passed through seomoz's system but I guess they don't have the inclination or resources to show us the info

Xenu reckons it can handle 1m urls, we are in the position of not really knowing how many pages our site has!

[KeriMorgret]

If you export the crawl diagnostics to a CSV, we do have this information in the last column.