Tricky 301 question

[DonnaDuncan]

It really depends on the nature, link and traffic patterns of your site Andy. If the vast majority of those 2,000+ 404's are coming from pages that should never have been indexed in the first place, you can probably get away with Luis's 2nd suggestion. If they're differentiated, valuable, and show evidence of incoming links and traffic, you've got some work ahead of you.

You might be able to streamline the process by inventorying and grouping like pages, then doing group redirects. But I suggest you do some analysis first to determine whether the effort is warranted.

[Yeeply.com]

Donna,

Andy has been very specific about this: "WITHOUT it being a massive manual job" hehe thanks for supporting my answer.

Luis

[DonnaDuncan]

Yeah. I heard him. I guess I'm saying "probably not".

I like how you're keeping us honest though Luis. I don't like it when people respond with what they want to say rather than with an answer to the specific question.

[AlanMosley]

do you know if they had any external links?

If they don't have external links then I would just let them 404.
some people have some wired thoughts of what 301's do. They simply redirect a request, so a request o A is told to remake the request to to B, so the crawler will follow it that way and award the pagerank to the new page with a small loss on each request.

If no external links what is there to gain? don't complicate your site with unnesasary redirects, there is a small argument that the pages may have been bookmarked at old url, but I think that argument is so weak I would not bother

[AndyMacLean]

Thanks Luis, unfortunately neither 1 or 2 are ideal.

1. I don't think there is much logic in the change of url structure between old and new product urls which makes that idea impossible.

2. Thats going to be a last resort

Andy

[AndyMacLean]

Thanks, the regex is a good idea and might be part of the solution for some urls at least but there seems to be some discrepancies in logic between old and new product urls and some of the new product urls are actually still the same as the old (which of course is fine).

[AndyMacLean]

Thanks Donna & Luis. Luis is right i'm looking for a way for this not to be a mammoth manual task for their developer.

[AndyMacLean]

Thanks Alan, yes they have good external links to many pages. They retail a very niche product and have a lot of forum, review, social type links. It might be though if need be they just have to focus mostly on 301s for the pages with those links. As best practise I am in favour of 301'ing regardless of external links as the link equity gets messed up and causes ranking issues, as in this case, as well as sending a signal to the engines about the amount of wasted resource they will use crawling a site with 1000s of 404s.

[osaka73]

If the majority of URLs have no logic, then it makes things a bit tricky in regards to minimizing the amount of work.

I once had a very active and large website with about 500-1000 single lines of rewrite code (1 for each URL) in my htaccess.  Surprisingly, it did not slow the server down at any noticeable rate, unless you are very sensitive to milliseconds and even then, one trial to the next could easily differ from regular internet congestion.  My point is, nobody ever noticed.

Here's a few ways that I would handle this job to get through it as quickly and effortlessly as possible.

The more aggressive and time consuming approach:
I would output all the URLs that were changed from phpmyadmin or whatever mysql administration tool you might use to a spreadsheet.  From that spreadsheet, I would add the original URL.
Then with the old URL (A1) and new URL (A2)  I would write a formula to output the correct rewrite (A3.)  Then simply copy and paste that formula down all the rows that it applies to.  You might need to break up the URLs to grab the right pieces for your formula.

Of course use, regex where you can, and keep your .htaccess rewrites to a minimum.

If that is still too much work, hire someone to do it through elance.com

The somewhat sloppy pace-yourself-approach:

Another approach you could take is to just monitor google webmaster tools for all the page not found errors.  And once a day or once a week, grab those URLS, create the rewrite, and mark it as fixed in webmaster tools.

The reason I say this is somewhat sloppy is because, you might find that you could have used regex in a lot of instances to better handle all those missing URLs.

But it may be a good way of staying on track with google, and handling the issues only as they arise so it does not feel like such a mammoth task.

[AlanMosley]

I don't know what you mean about link equity, if there is no link pointing to the page then there is nothing lost.
As for search engines finding a lot of 404s, they will remove them from the index after a while, no problem there, you are returning the correct status code, that's what they want. This will allow them to clean up there index and stop crawling the pages.

[AndyMacLean]

Pages dont just get equity from external links of course. If a category page has 10 links to it the product pages linked to on that page benefit. The wholesale drop in rankings isn't because every page had an external link to it.

[AndyMacLean]

Thanks

[AlanMosley]

That's my point, you only need to worry about the pages that had external links
Thanks