Google indexing staging / development site that is redirected...

[mozroadjan]

Hi Moz Fans! - Please help.

We had a acme.stagingdomain.com while a site was in development, when it went live it redirected (302) to acmeprofessionalservices.com (real names redacted!!)

no known external links to staging site

although staging site url has been emailed from Google Apps(!!!)

now found that staging site is in the index even though it redirects to the proper public site.

and some (but not all) of the pages are in the index too. They all redirect to the proper public site when visited.

It is convenient to have a redirect from the staging site to the new one for the team, Chrome etc. remember frequently visited sites. Be a shame to lose that.

Yes, these pages can be removed using webmaster tools.
But how did they get in the index to start with?

And if we're building a new site, and a customer has an existing site is there a danger of duplicate content etc. penalties caused by the staging site?

We had a similar incident recently when a PDF that was not linked anywhere on the site appeared in the index. The link had been emailed through Google Apps, and visited in Chrome, but that was it.

So 3 questions.

Why is the staging site still in the index despite the redirects?

How did they get in the index in the first place?

Will the new staging site affect the rank of the existing site, eg. duplicate content penalties?

[evolvingSEO]

Hi There

1. It could still be in the index because they are 302 redirect and not 301. 302 is temporary, and therefore Google may not de-index those URLs. It also takes time. I've seen Google take months to noindex redirecting URLs. Also, make sure you are not blocking crawling of the dev site, or Google will not see the redirects.

2. I am not sure how they got there to begin with. I pretty much always can find some sort of error - maybe someone tweeted a staging URL, maybe crawling wasn't blocked, maybe there was one link to staging from the live site etc etc. Regardless - somehow Google crawled it To prevent this in the future always block crawling of staging servers well before you ever put anything on them.

3. Usually Google tries to sort this out. They won't give you a penalty for "technical" duplicate content (penalties are more for "malicious" duplicate content ie: stealing people's content). So you won't get penalized, but the more you can help Google out by sorting it out, the more time Google can spend crawling the correct site etc.

What I would do now is, if you do want the staging URLs to redirect (which might not be the best solution if you want to ever go back and work on the staging server again) - but if you do, use 301 redirects and make sure you are allowing crawling of the staging site. Keep it registered in webmaster tools and this way you can monitor the indexation levels.