Domain restructure, sitemaps and indexing

[magdaknight]

I've got a handcoded site with around 1500 unique articles and a handcoded sitemap. Very old school.

The url structure is a bit of a mess, so to make things easier for a developer who'll be making the site database-driven, I thought I'd recategorise the content. Same content, but with new url structure (I thought I'd juice up the urls for SEO purposes while I was at it)

To this end, I took categories like:

/body/amazing-big-shoes/
/style/red-boots/
/technology/cyber-boots/

And rehoused all the content like so, doing it all manually with ftp:

/boots/amazing-boots/
/boots/red-boots/
/boots/cyber-boots/

I placed 301 redirects in the .htaccess file like so:

redirect 301 /body/amazing-boots/ http://www.site.co.uk/boots/amazing-boots/

(not doing redirects for each article, just for categories which seemed to make the articles redirect nicely.)

Then I went into sitemap.xml and manually overwrote all the entries to reflect the new url structure, but keeping the old dates of the original entries, like so:

<url><loc>http://www.site.co.uk/boots/amazing-boots/index.php</loc>
    <lastmod>2008-07-08</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.5</priority></url>

And resubmitted the sitemap to Google Webmasters.

This was done 4 days ago. Webmaster said that the 1400 of 1500 articles indexed had dropped to 860, and today it's climbed to 939.

Did I adopt correct procedure? Am I going about things the right way? Given a little time, can I expect Google to re-index the new pages nicely?

I appreciate I've made a lot of changes in one fell swoop which could be a bit of a no-no... ?

PS Apologies if this question appears twice on Q&A - hopefully I haven't double-posted

[RyanKent]

Did I adopt correct procedure? Am I going about things the right way? Given a little time, can I expect Google to re-index the new pages nicely?

I would say Yes to all three, but clarify with details below.

When you submit a sitemap to Google with 1500 pages, there is no guarantee they will index all of your pages. It sounds like you have done a lot of intensive, manual work. Fortunately, you have done things the correct way in properly redirecting each page to the new URL. If Google indexed 1400 pages before, they should index around that same number once again. It may take several weeks depending on a few factors such as your site's Domain Authority, navigation and how many links each page has received.

With respect to the sitemap, I would highly recommend using sitemap generation software. It is simply not reasonable to manually update a sitemap with 1500 entries. I would have updated the lastmod date on the sitemap but it may not make any difference.

A popular sitemap tool: http://www.xml-sitemaps.com/. The free version only generates 500 pages, but for $20 you can buy the full version and automate it.

[magdaknight]

Thanks Ryan, that's a weight off my mind. I'll definitely take up your advice on the sitemap generator, too. Thanks for the recommendation - I'd seen a few around, but wasn't sure - it's great to be pointed in the right direction!

[RyanKent]

Just to clarify, I know the sitemap tool I mentioned is very popular. Many small sites use it because it is online, fast and free. I have used it a few times myself. I can't necessarily say I recommend it because I have never personally purchased the software. I would say that if I was looking to obtain a sitemap for your site, I would start with that tool but may take a look at some others.

[magdaknight]

That's fair. I get that you're not recommending it personally - but it does seem popular with consistently good feedback from people, so I'll give it a go

[IamJustinM]

Looks perfect to me too. Did the internal linking structure change at all or is that still the same? If it's all the same you should be right back where you were in no time.  And you should see some benefits from having a more common sense, easy to understand URL structure. Cheers!

[magdaknight]

Sadly I did change the internal linking structure, so that internal links now point to new urls not the old ones. The good news is that even with changing the internal linking structure, Google seems to be keeping abreast of it all. The number of urls indexed has now jumped - in a day - from 939 to 1024, so good old Google is clearly keeping up with the changes. Looks like my fears were ungrounded. Yay

[IamJustinM]

Oh, no what you did is perfect! I guess I meant the site architecture/navigation, but you answered it in your original post when you said "Same content" so disregard that question. Congrats.

[connectiveWeb]

If your developer will be making the website dynamic via a system like WordPress there will be automated ways to keep your sitemap up to date every time you publish a new page to your system and then it will even ping the search engines that the sitemap is updated It will be a "set it and forget it" type of thing with sitemaps if you are moving in that direction

Good luck!