Development site is live (and has indexed) alongside live site - what's the best course of action?

[McTaggart]

Hello Mozzers,

I am undertaking a site audit and have just noticed that the developer has left the development site up and it has indexed. They 301d from pages on old site to equivalent pages on new site but seem to have allowed the development site to index, and they haven't switched off the development site. So would the best option be to redirect the development site pages to the homepage of the new site (there is no PR on dev site and there are no links incoming to dev site, so nothing much to lose...)? Or should I request equivalent to equivalent page redirection?

Alternatively I can simply ask for the dev site to be switched off and the URLs removed via WMT, I guess...

Thanks in advance for your help!  

[CommT]

Hi Luke,

I'm interested in other responses to this question...

If I was in your position after seriously berating the dev I would make sure you disallow the dev site in your robots.txt and use webmaster tools to remove the URLs from the index. Then I would password protect the dev site so the search engines couldn't get there even if they try.

Like I say, I'm interested in other responses! This is what I would do, but I don't really know if it's definitely the right thing to do. Does anyone else have anything to add?

Best of luck - its crappy when someone else's error cocks up your work: when our site launched for the first time our IT department screwed up on a monumental scale by getting the DNS settings wrong.

Amelia

[Dan-Lawrence]

Hi

I'm in a similarish situation with a clients site.

Their situation is that the dev site is on a subdomain i.e. staging.domain.com and they want to keep the staging area active for demonstrating future development work, so situation may be slightly different from yours.

They have now blocked via robot.txt but that's like shutting the stable door after the horse has already bolted.

I asked Moz Q&A a few months ago and got the below answer from a few very helpful and wize Mozzers

1. Setup a completely different Webmaster Tools account unrelated to the main site, so that there
   is a new W.T account specific to the staging area sub-domain

2. Add a robots.txt on the staging area sub domain site that disallows all pages and all crawlers
   OR use the no-index meta tag on all pages but Google much prefers Robots.txt usage for this

Note: Its very important when you update the main site it does not include or push out these files and
instructions too (since that would result in main site being de-indexed)

3. Request removal of all pages in GWT. Leave the form field for the page to be removed blank,
   since will remove all subdomain pages

4. After about 1 month OR you see that the pages are all out of the Search Engine listings (SERPS),
   and Google has spidered and seen the robots.txt, then put up a password on the entire staging
   site.

Hope that helps

All Best

Dan

[RobertFisher]

Dan,

This is a very good answer. Just to emphasize, probably the most important piece with a "dev" site is the last one Dan mentions: Password protection. Once you clean up the issue, add it then you should not have the issue going forward.

Even with robots.txt on our dev sites and our design studio, we have had pages end up on the SERPS. Because of the DA of our design studio (where clients go to approve a comp, etc.) we recently had a new political client's comp ranking for a search term on page one. (Ahead of their actual site (we were building another to replace it). So, even with robots.txt, there is still no guarantee it will not be crawled.
Adding password protection will assist in that.

Lastly, if you have someone building you a site, and they say they do not want to take down the dev version after your launch, tell them you do not wish to pay them. It will go down. That is unreasonable. I cannot think of a reason to keep the dev version live once the client site launches.

Again, good job Dan.

[Dan-Lawrence]

Thanks Robert and for the extra comments too !

I cant remember which Mozzer helped me with the above in the first place who should be credited but ill track down the original thread and add it to this post since also contains further info and discussion

All Best

Dan

[McTaggart]

Yes a great answer there from Dan - and thanks for your useful input - good point re: not relying on robots.txt alone!

[McTaggart]

Many thanks Dan - much appreciated - that process there makes perfect sense even though in my case too :)))) I will report back on progress in a month or so...

[McTaggart]

Thanks Amelia - yes you're definitely on the right lines - Dan's response below is v helpful too, that's for sure. I do struggle with developers from time to time, so teaching myself coding and so on via codeacademy, etc. - learnt at uni many years ago but v out of date! Will come in useful for SEO too.

[Dan-Lawrence]

Very pleased to have been of assistance

heres links to older threads where i asked similar before, for further verification and credit to those that originally helped me:

http://moz.com/community/q/removing-a-staging-area-dev-area-thats-been-indexed-via-gwt-since-wasnt-hidden-from-the-index

http://moz.com/community/q/staging-development-areas-should-be-not-indexable-i-e-no-followed-no-index-in-meta-robots-etc