How to prevent development website subdomain from being indexed?

[SproutDigital]

Hello awesome MOZ Community!

Our development team uses a sub-domain "dev.example.com" for our SEO clients' websites. This allows changes to be made to the dev site (U/X changes, forms testing, etc.) for client approval and testing.

An embarrassing discovery was made. Naturally, when you run a "site:example.com" the "dev.example.com" is being indexed. We don't want our clients websites to get penalized or lose killer SERPs because of duplicate content.

The solution that is being implemented is to edit the robots.txt file and block the dev site from being indexed by search engines.

My questions is, does anyone in the MOZ Community disagree with this solution? Can you recommend another solution? Would you advise against using the sub-domain "dev." for live and ongoing development websites?

Thanks!

[OlegKorneitchouk]

So....

1. If the dev site has not been indexed yet, you can block crawlers via robots.txt
2. If the dev site is already indexed and you want it removed, add meta NOINDEX tag to all pages allow the site to be crawled via robots.txt (reason: you want google to crawl and noticed the noindex tag on the pages so that they remove it from search results. if the site is indexed and you block crawler via robots.txt, google will keep the pages indexed but won't crawl them again). Once deindexed, you can block via robots.txt again

---

As long as its blocked (and you build that into your process), having the dev site on the same domain shouldn't be an issue. We have our own dev domain + server that autoblocks all pages from being indexed.

[SproutDigital]

Thanks for your prompt feedback Oleg!

Your proposed action is exactly what we're implementing.

Randy Holland, Sprout Digital

[Chris_Hickman]

Hey there, in addition to Oleg's comment you can add an htpasswd file to your server to require a username and password to be entered before any users or robots are allowed to access your website.

You can find information on setting this up here.