How is Google finding our preview subdomains?

[ZeeCreative]

I've noticed that Google is able to find, crawl and index preview subdomains we set up for new client sites (e.g. clientpreview.example.com). I know now to use "meta name="robots" and robots.txt) to block the search engines from crawling these subdomains. My question though, is how is Google finding these subdomains? We don't link to these preview domains from anywhere else, so I can't figure out how Google is even getting there.

Does anybody have any insight on this?

[NakulGoyal]

The discovery can be from multiple mediums. Do you or the client have Google Toolbar installed ?

[matbennett]

We usually put them behind an .htaccess login now.  We've had situations where the development site have been outranking the live site. Great demo of the power of on-site optimisation, but still a bit annoying for the client.

People used to always blame google toolbar for this. Likewise using chrome could potentially add something to the "to crawl" list. I wonder what the respective privacy policies say about that. I've also seen staging sites pick up links. When an external link on the staging site has been clicked it has alerted someone else, appeared as a link back/trackback etc.

[NakulGoyal]

I agree with Mat. You never know, but yes Chrome could be another major source. It also depends what you set as your privacy when you setup Chrome (Send anonymous usage data to Google, Yes/No ?) and so on.

[SWKurt]

I would also consider adding a noindex tag if you want the urls removed.

[irvingw]

If the subdomains are added to WMT google will know about it. if you are designing sites for clients and putting them on your site as subdomains it behooves you to make sure 100% that their dev sites are not being seen by Google. It's duplicate content and your subdomain is the original source of this content. Looks unprofessional too

a) verify any subdomain you are creating for a client in WMT

b) block it in robots.txt and noindex nofollow all pages globally

c) for the ones that are already indexed, go into google WMT and go into that subdomain account and request removal of the site in Googles index. This will remove the indexing for that subdomain only don't worry it won't remove your main site from the index.

[EGOL]

I think that this is a good idea.  But you gotta be careful.

Our competitor (who ranked #1 and we ranked at #2) had their site redesigned and the design company included the noindex on every page. They forgot to take it off when the new design went live.  It took them quite a while to figure it out and we enjoyed all of their sales for about a month.

We are #1 now and they are #2.   Must have been a bad design job. 

[ZeeCreative]

I don't specifically use the Google Toolbar, but others in the office may (although I don't think so). It sounds like Chrome could be a potential source as well?

[ZeeCreative]

Thanks for the insight guys.

[ZeeCreative]

Thanks for your response Irving. We put some of our preview sites on subdomains of our main domain, but then remove them after the site goes live, so their shouldn't be any duplicate content issues. The main question is just how Google is finding these subdomains.