Broken Links from Open Site Explorer

[sbaylor]

John, I certainly see your point, but our IT guys would not take too kindly to me running a spider-based app from inside their precious network, which is why I was looking for a less intrusive solution.

I'm planning on a campaign to revive "flummoxed" to the everyday lexicon next.

[jesse-landry]

How else do you expect to trace these broken links without using a "spider?"

Obviously it's the solution. And the programs take up all of like 8 megs... so what's the problem/concern?

I second the screaming frog solution. It will tell you exactly what you need to know and has ZERO risk involved (or whatever it is that's hanging you up). The bazooka comparison is ridiculous, because a bazooka destroys your house. Do you really think a spider crawl will affect your website?

Spiders crawl your site and report findings. This happens often whether you download a simple piece of software or not. What do you think OSE is? Or Google?

I guess what we're saying is if you don't like the answer, then so be it. But that's the answer.

PS - OSE uses a spider to crawl your site...

PPS - Do you suffer from arachnophobia? That movie was friggin awesome now I want to watch old Jeff Daniels films.

PPSS - Do you guys remember John Goodman being in that movie? Wow the late 80s early 90s were really somethin' special.

[jesse-landry]

run the spider based app from outside their "precious network" then. hell, i could run it right now for you from my computer at work if I wanted. Use your laptop or home computer. It's a simple spider you don't have to be within any network to run it. You could run one for CNN.com if you'd like as well...

[sbaylor]

Jesse:

That movie was creepy, but John Goodman was awesome in it.

I started this thread because I was frustrated that OSE restricts my results to 25 links, and I simply wanted to find the rest for that particular URL. I was assuming that there was either:

a. A method for getting the rest of the links that Roger found

b. Another way of pulling these reports from someone who already spiders them (since I can't get any using the link:[URL] in Google and Webmaster Tools isn't showing them).

Thanks to all for your suggestions.

[jesse-landry]

I understand. Thing is, there is a way and the spider doesn't affect anything. Like I said, I have screaming frog installed on my computer and I could run a report for your website right now and you or your IT department would never know it happened.. I just don't understand the part where the software doesn't work for you but to each their own i suppose.

[jesse-landry]

I got tired of the confusion and went ahead and proved it. Not sure if this is the site you wanted results for, but I used the site linked in your profile (www.gallerydirect.com)

took me about 90 seconds and I had a full list... no 404s though

anyway here's a screenshot to prove it:

http://gyazo.com/67b5763e30722a334f3970643798ca62.png

so what's the problem? want me to crawl the fbi site next?

[sbaylor]

Jesse:

I appreciate your feedback, but am surprised that the ScreamingFrog report found no 404s. SEOmoz found 15 in Roger's last crawl, but those aren't the ones that I'm currently trying to solve.

The problem page is actually showing up as duplicate content, which is kinda screwy. When visiting the page, our normal 404 error doesn't appear (which our developers are still trying to figure out), but instead, an error message appears:

http://www.gallerydirect.com/about-us/media-birchwood

If this were a normal 404 page, we'd probably be able to find the links faster.

[LynnPatchett]

Indeed if it is not showing a 404, that makes things a bit difficult!

You could try another way, use OSE!

http://www.opensiteexplorer.org/links?page=1&site=www.gallerydirect.com%2Fabout-us%2Fmedia-birchwood&sort=page_authority&filter=&source=internal&target=page&group=0

Use the exact page, filter for only internal links, boom 127 pages that link to it. There might be more, but this should get you going!

[irvingw]

Use the word FREE with an asterisk because sreaming frog is now limiting the free version to 500 pages. Xenu is better, even brokenlinkcheck.com lets you spider 3000 pages.

500 pages makes the tool practically worthless for any site of decent size.

[jesse-landry]

oops! i did not know this. Thanks Irving.

[jesse-landry]

One more thing to add - GWMT should report all 404 links and their location/referrer.

[sbaylor]

Lynn, that was a BIG help.  I had been running that report, but was restricted to 25 responses. When I saw your suggestion to filter for only internal links, I was able to see all 127.

Big props.  Thanks!

[sbaylor]

Jesse,

That's where I started my search, but GWMT wasn't showing this link. I can only presume that because it isn't coming back a 404 (it is showing that "We're Sorry" message instead) that they're considering that message to be content.

Thanks!

[Cyrus-Shepard]

Whew! Big thread.

Sometimes, when you can't find all the broken links to a page, it's easier simply to 301 redirect the page to a destination of your choice. This helps preserve link equity, even for those broken links you can't find on large sites. (and external links, as well)

Not sure if this would help in your situation, but I hope you're getting things sorted out!