Broken Inner Links - Tool Recommendations?

[uderic]

Do you have any recommendations for tools that scan an entire website and report broken inner links?

I run several UGC centered websites and broken inner links, and external, is an issue.

Being that these websites are several hundred thousand pages large, I am not really all that excited about running software on my desktop (xenu link sleuth for example). Any online solutions you could recommend would be great!

[Hondaspeder]

I use Google webmaster tools. Go to diagnostics, then crawl errors.

[StalkerB]

You might be out of luck on a site that size.

I think WebCEO can do this with their online version but to get 100,000 urls crawled I think it'll cost you a bomb (the sort of money that it'd be cheaper to buy a second PC to run Xenu, lol).

Anyway - http://www.webceo.com/ - I think it may also be possible to install the download version to a server and run it that way.

[Hondaspeder]

I might be wrong, but I think Google WMT can accomplish this with ease. I'm looking at 1000 right now. Externally you'll probably have to use xenu =/

[StalkerB]

WMT is alright, apart from the fact you can't force Google to crawl all your pages. I would doubt that even a majority of the pages were crawled and indexed by Google (though I don't know what the site is).

Plus, as you say, it only deals with internal links and 404s coming in.

Do you know what the upper limit is on how many crawl errors WMT will display?

[E-dreamz]

I think Xenu is your best option here. The size of the site nearly cuts out the chance a web tool could handle it.

Just recently on a site review I had to run Xenu on a site with 160,000 pages. It only took 4 hours running at 30 threads to complete. Any modern PC should handle it fine.

[IPINGlobal54]

I have used this in the past http://www.auditmypc.com/free-sitemap-generator.asp - (Click on the image in the top right of the instructions) a free tool for site map generation that will show broken internal links in the process. I don't think it has any limits to it, although I have not tried it on a site as large as you are suggesting. Just ensure you are not logged into your site when you run it. Although Google webmaster tools is ok, you can't verify changes made very quickly.

[kwoolf]

Well you've found the best solution right here at SEOmoz! Instead of wasting time learning new systems to find out if they'll work or not, just solve your problem. Sign with PRO Elite and you can crawl 100,000 pages.

[KeriMorgret]

Ideric, did any of these suggestions answer your questions, or have you been able to otherwise find a tool for this? I know others would find the information useful.

At a previous company, we had a custom-written solution to check external links, and made it check response headers until a 200 OK showed, or it got five levels deep. What we'd often find is that we'd have a 301 for an external link, and it'd go from non-www to www. Wouldn't necessarily worry about fixing that, but then later realized that from there, the www link was a 404, OR went to a 200 OK category landing page that said "we've reorganized our site, search here for that individual resource".

[ericpratum]

If it happens to be a wordpress site, there is a plugin called something like "Broken Link Checker." If I recall correctly, that checks internal and outbound links. Otherwise, not too sure.