How did my dev site end up in the search results?

[EcommerceSite]

We use a subdomain for our dev site. I never thought anything of it because the only way you can reach the dev site is through a vpn. Google has somehow indexed it. Any ideas on how that happened? I am adding the noindex tag, should I used canonical? Or is there anything else you can think of?

[VERBInteractive]

Have you updated the dev sites robots.txt to disallow everything? It is up to the bot to listen, but that combined with removing all of the dev URLs from Google Webmaster tools should do the trick.

[DmitriiK]

Hi there.

If what you say is true, then there are only two answers: you got a leak somewhere or your settings/configuration is messed up.I'd say go talk to your system admin and make sure that everything what's supposed to be closed is closed, IPs, which are supposed to be open for use are open and those IPs only.

[DmitriiK]

Robots.txt won't help anyhow. Bots still can see that there is such directory, they just won't see what's inside of those directories/subdomains.

[EcommerceSite]

We never updated the robots.txt because the site was behind a firewall. If you click on any of the results it will not load the page unless on my VPN.

[EcommerceSite]

They are telling me that there are no holes, and I have tried getting to the pages but can not do it unless I am on my vpn.

[DmitriiK]

Then there is a leak somewhere - Google bots can "see" your subdomain.

Or it's been simply linked from somewhere. Then Google will try to follow the link and that would make it indexed.

[DmitriiK]

Then my first assumption is that it's linked from somewhere - read my comment a little above.

[EcommerceSite]

I have ran screaming frog to see if there are any links to any pages and but couldn't see any. Even if Google did try to follow it the firewall would stop them. It is so strange.

[DmitriiK]

I never used screaming frog, does it check both external and internal links?

[EcommerceSite]

It just uses internal links. Do you think I should try the webmaster tools removal? That seems like it could go wrong.

[DmitriiK]

Don't use tool removal, it can go bad indeed. Now, are you sure that there are no external links coming from anywhere?

For now I'd recommend putting noindex, nofollow on that dev subdomain and do manual recrawl through GWT.

[MattRoney]

Personally, I'd still recommend using robots.txt to disallow all crawlers, even if more steps are taken.