Development site accidentally crawled - Will this cause problems?

[AndrewY]

We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com).  There has been no SEO and no links created to this site, or so I thought.

Recently, I found out that Google somehow found its way to this development site and has been indexing the pages!  I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain.

I immediately created a robots.txt file to disallow access to the entire development domain.  My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain.

We're slated to launch in 2-3 weeks.  Is there anything else I should do?  Should I even be worried?  I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts.

Thank You!

[AU-SEO]

Setting a Robots.txt file for the Dev Site to be No index was a correct response. You can also add a No index no follow meta tag to the Dev site as well.

Another step you can take is to set up a Google Webmaster Tools account for the Dev site and block there as well.

Some dev sites are placed behind a firewall or require a sign on to access, this process can block google as well.

The risks you have is essentially creating an entire duplicate of your current website. Google will always try and crawl everything it can on the net regardless of Noindex tags. No index simply means please dont place in your index. It is important to remember that there are other Search Engines out there besides Google, Bing/yahoo, Ask, Blekko, etc... and all do not automatically honor the Noindex no follow tag. So any secure pages or documents should be just that - secured.

If those pages are no longer in the index, and are not security or confidential in nature I wouldn't worry too much.

• Phil G

[Francisco_Meza]

I don't even worry about that anymore. I let Google see me build out a site anyway. I used to worry about that, but not anymore.

"I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain."

They probably came to the server and hit every site on it.

[AndrewY]

Great Answer, thanks Phil!  One follow-up question:

In my robots.txt for the development site, I have the following:

User-agent: *

Disallow: /

Is this the correct configuration for the robots.txt file to accomplish what I want, that being removing the entire site from being crawled and from the exiting index?  Or should I be configuring it differently?

Also, good tip on Webmaster Tools.  I'll be request removal there as well.