New server update + wrong robots.txt = lost SERP rankings

[9Studios]

Over the weekend, we updated our store to a new server.  Before the switch, we had a robots.txt file on the new server that disallowed its contents from being indexed (we didn't want duplicate pages from both old and new servers).

When we finally made the switch, we somehow forgot to remove that robots.txt file, so the new pages weren't indexed.  We quickly put our good robots.txt in place, and we submitted a request for a re-crawl of the site.

The problem is that many of our search rankings have changed.  We were ranking #2 for some keywords, and now we're not showing up at all.  Is there anything we can do?  Google Webmaster Tools says that the next crawl could take up to weeks!  Any suggestions will be much appreciated.

[Ben-HPB]

That does not sound fun at all.... So you just changed the server, complete copy?

My first question would be other than the server did anything else change? Copy or URL's

My second question would be is the other server still up and live to the internet?

[Malarowski]

If this is all that happened the next crawl should fix it. Just sit tight and they should bounce up again in a week or so.

[inhouseninja]

Remain calm.  You should be just fine.  It just takes time for Google to digest the new robots.txt.   I would be concerned if things didn't change in 3-4 weeks.  Adopt a rule to not freak out on Google until you've given the problem 14 days to resolve.  Sometimes Google moves things around and this is natural.

If you want Google to crawl your site faster, build some links and do some social media.   That will encourage Google to speed it up.

[Dr-Pete]

I hate to say it, but @inhouseninja is right - there's not a lot you can do, and over-reacting could be very dangerous. In other words - don't make a ton of changes just to offset this - Google will re-index.

A few minor cues that are safe:

(1) Re-submit your XML sitemap

(2) Build a few new links (authoritative ones, especially)

(3) Hit social media with your new URLs

All 3 are at least nudges to re-index. They aren't magic bullets, but you need to get Google's attention.

[Ben-HPB]

I agree, let Google re-index first and then re evaluate the situation.

[9Studios]

Thanks everyone for the help!  Fortunately we remedied the problem almost immediately, so it only took about a day to get our rankings back.  I think the sitemap and fixed robots.txt were the most important factors.

[inhouseninja]

Dr. Pete,

I just ran across one of your webinars yesterday and you brought up some great ideas.  Earned a few points in my book

Too often SEOs see changes in the rankings and react to counter-act the change.  Most of the time these bounces are actually a GOOD sign. It means Google saw your changes and is adjusting to them.  If your changes were positive you should see positive results.  I have rarely found an issue where a user made a positive change and got a negative result from Google.  Patience is a virtue.