Development site crawled

[EileenCleary]

We just found out our password protected development site has been crawled.  We are worried about duplicate content - what are the best steps to take to correct this beyond adding to robots.txt?

[KyleJack]

Oops! That sounds unfortunate, Marcy. How did that happen?

Once you have added the correct rules to the robots.txt - I'm guessing you're using "Disallow: /" - you can request, if your development site is registered in Google Webmaster Tools, that Google remove the site from its index.

www.google.com/webmasters/tools/url-removal

Hope that helps,

K

[firstconversion]

Put the site behind a login

[EileenCleary]

Thanks Stephen & Kyle!  We had the site behind a login, so we're not sure how this happened.  Any idea?

[ollan]

Unfortunately, robots.txt won't prevent your site from being crawled and indexed if there is a link from an external site pointing to yours. What you need to do is use

on all your development pages. I don't know how big your site is, so this may or may not be a lot of work. Do this, then after the next Google crawl, your pages will be dropped from the SERPs.