301ing entire E-Com website - Using TopPages Report as a Guide

[Blenny]

Hi Folks,

My company is in discussions with an ERP provider to migrate our existing website to a new backend system, new url structures..the whole 9 yards.

We have over 15,000 products, and a lot of "odd" url structures created by an outdated system that we were unable to adjust. All and all, if you're talking about a raw url count, it's in the 100s of thousands.

During discussions, we were told that bulk 301 redirects would be a problem. They could be performed manually though (greaaattt).

Due to this, I ran an OpenSite Explorer report, and isolated our top pages. After exporting the CSV, I was able to breakout "all" urls that have links from other domains.

My question is, has anyone used the OpenSite Explorer on a website of similar size, to form the basis of a migration? Do you have enough confidence in the tool to use it in this way, or should we re-negotiate our agreement until a way can be found to mass 301 ALL urls. I'm at least a little concerned that OpenSite Explorer isn't indexing all of the links out there. Gasp..or would there be a better tool to accomplish what I'm trying to do?

Thanks!

[gdavies09031977]

As far as I can tell, Open Site Explorer doesn't show all the links - in fact it shows only a very tiny proportion of the links to your site. Yahoo site Explorer, while it's still available, is a more accurate list of links to your site. For example on our little site, OSE shows 30 in bound links, but Yahoo shows almost 300!! As painful as it may be, I would 301 every page, but do them in SE Rank order, it only has to be done once, and when it's done it's done. That said, I'm surprised your web company can't find a way to automate it?!

[ShaMenz]

Just to clarify what you are doing here, a couple of questions:

1. ... am I correct in assuming that you are wanting to use the same domain, but the site structure will be changed?
2. Also, will there be ANY pages within the new structure that might simply be moved to a different location within the same domain (for example move from root directory to /sub directory), but retain the same file name?

These things will make a difference as to how you can approach the issue.

Sha

[Blenny]

Thanks..great answer..and yeah, that's pretty much how I feel with that last sentence..ugghhh..

[Blenny]

Hi Sha, thanks for your response.

1. Yes, we will use the same domain but structure will be completely changed.

2. Unfortunately, our current CMS is so backward, that it will be a complete file name and, directory change.

Thanks!

[ShaMenz]

OK...that makes it more difficult.

I was hoping that you might have some files whose filenames would be retained and would only need to move from one directory location to another within your new site structure.

Since it is possible to redirect the entire contents of a directory to another directory with a single line of code, this would have made it easier.

Sha

[Dr-Pete]

As much as we love OSE here, it is not a good bet for doing a complete inventory of your entire current site. I'd probably recommend a desktop crawler, like Xenu or Screaming Frog. See my comparison post here:

http://www.seomoz.org/blog/crawler-faceoff-xenu-vs-screaming-frog

I've got to believe that there's some logic that could connect the old and new URLs, though, even if it has to go through a database look-up. Creating 15,000 manual redirect rules isn't only going to be a huge pain, it may cause a huge processing bottleneck. I'd really push back on what "can't" means and why it can't be done. Too often, "can't" is code for "don't want to", in my experience.