Dev Site Was Indexed By Google

[ntsupply]

Two of our dev sites(subdomains) were indexed by Google. They have since been made private once we found the problem. Should we take another step to remove the subdomain through robots.txt or just let it ride out?

From what I understand, to remove the subdomain from Google we would verify the subdomain on GWT, then give the subdomain it's own robots.txt and disallow everything.

Any advice is welcome, I just wanted to discuss this before making a decision.

[WhiteboardCreations]

Hey Tyler,

We would follow the same protocol if in your shoes. Remove any instance of the indexed dev subdomain(s), then create your new robot.txts files for each subdomain and disavow any indexed content/links as an extra step. Also, double check and even resubmit your root domain's XML sitemap so Google can reindex your main content/links as a precautionary measure.

PS - We develop on a separate server and domain for any new work for our site or any client sites. Doing this allows us to block Google from everything.

Hope this was helpful! - Patrick

[donford]

Hi Tyler,

You definitely don't want to battle yourself for duplicate content. If the current sub-domains have little link juice (in links) to them, I would simply block the domain from being further indexed. If there are a couple pages that are of high value it maybe worth the time to use a 301 redirect to prevent losing any links / juice.

Using robots.txt or noindex / tags may work, but in my personal experience the easiest and most efficient way to block any indexing is simply use .htaccess / .htpasswrd this will prevent anybody without credentials from even viewing your site effectively blocking all spiders / bots and unwanted snoopers.

[KeriMorgret]

I've had this happen before. In the dev subdomain, I added a robots.txt that excluded everything, verified the subdomain as its own site in GWT, then asked for that site (dev subdomain) to be removed.

I then went and used a free code monitoring service that checked for code changes of a URL once a day. I set it up to check the live site robots.txt and the robots.txt of all of the dev sites, so I'd know within 24 hours if the developers had tweaked the robots.txt.

[customerparadigm.com]

We ran into this in the past, and one thing that we (think) happened is that the links to the dev site were sent via email to several gmail accounts.  We think this is how Google then indexed the site, as there were no inbound links posted anywhere.

I think that the main issue is how it's perceived by the client, and if they are freaking out about it.  In that case, using an access control password to prevent anyone from coming to the site will limit anyone from seeing it.

The robot.txt file should flush it out, but yes, it takes a little bit of time.