Site being indexed by Google before it has launched
-
We are currently coming towards the end of a site migration, and are at the final stage of testing redirects etc. However, to our horror we've just discovered Google has started indexing the new site. Any ideas on how this could have happened? I have most recently asked for robots.txt to exclude anything with a certain parameter in URL. Is there a chance this, wrongly implemented, could have caused this?
-
Many ways - Google discovers URLs through a large number of methods, although primarily through links. I have seen some pretty amazing ways of discovery though...
- Links posted in emails where the emails ended up on the web (like a private newsletter with a public archive)
- Links showing up in click stream data services like alexa
- Links showing up from "recently registered" domain lists
The rule of thumb is always ALWAYS start with a robots.txt. It is the first thing you should do when setting up a dev environment.
-
Duplicate question, closing this question so all answers can be given at http://www.seomoz.org/q/site-being-indexed-by-google-before-it-has-launched-2