Massive URL Migration with thousands of 301

[rpaiva]

Hey Everyone!

I'm currently working on a project that we have A Lot of product pages and we have thousands of URL's that need to be 301'd over.  I know this can be a major issue and could lead to tons of errors.  What is everyone's thought of doing such a huge Migration, Should I do it all in phases? or should I do them all at once so they can all be indexed together?

What would you suggest to be the best way to go about doing such a massive migration?

[WilliamKammer]

Assuming not all of your pages hold incredible value and don't get visited a lot, I don't think you'd need to do it in phases. You can save yourself some time by throwing in some regex to grab large chunks of URLs at a time and redirect them that way.

Your more valuable pages should be one-to-one redirected as to not confuse users and to retain the most juice to the right place, but for less important pages, grab a bunch at a time.

One way to do this is:

redirectmatch 301 ^/sub/directory/(.*)$    http://site.com/newdirectory

This would grab all pages under site.com/new/directory/ANYTHING to site.com/newdirectory/.

Hope that helps.

[rpaiva]

Hey Thanks for the response!

I'm currently dealing with a site that has a large duplication problem and i'm Canon tagging them all to then 301 them all over.  Why shouldn't I do a phase approach though?  Wouldn't it be easier to see where errors happen if I come at this migration with a phased approach?

[WilliamKammer]

Webmaster's Tools has been pretty good to me regarding telling me about 404s I may have missed after a migration, but using the tactic I described really limits missed URLs, assuming you have a good grasp on the subdirectories of your site.

Phasing is an option, but say, in the case of a redesign or domain migration, I like to do it all at once. That way Google isn't trying to index the website on two different URL structures or domains. Doing it in one shot makes it clear to Google what has happened, since everything is now moved, instead of just a fraction at a time. I'm sure this is a point of debate, and not necessarily the definitive way to do it.

[rpaiva]

I have never done a migration of a website with so many URL's it is kind of overwhelming.  I do have a 404 page with the catchall rule in place.  See i'm coming from the business side of SEO i'm not really the developer getting in there and actually doing the migration ( I tip my hat to all you developers without you I am nothing but a voice).  I'm guiding my developer to the safest route.  Doing the migration all at once does make sense to prevent both sites being cached