How preproduction website is getting indexed in Google.

[nlogix]

Hi team,

Can anybody please help me to find how my preproduction website and urls are  getting indexed in Google.

[Andy.Drinkwater]

Hi,

Have you noindexed & nofollowed the site and pages? I would also suggest you block all crawlers by disallowing access in the robots.txt file.

Do you know if this has all been done?

-Andy

[Mustansar]

Hi

To prevent most search engine web crawlers from indexing a page on your site, place the following meta tag into the section of your page:

To prevent only Google web crawlers from indexing a page:

You should be aware that some search engine web crawlers might interpret the noindex directive differently. As a result, it is possible that your page might still appear in results from other search engines.

here is complete guide: https://developers.google.com/webmasters/control-crawl-index/docs/robots_meta_tag?csw=1

[Ria_]

In addition to noindexing the pages using the meta tag, if you have WMT / Search Console set up, you can request Google remove those URLs from their index for the time being. I've found that this may take up to a couple of hours from the removal request to the time of actual removal.

As to how they were found, there's a good chance that Google crawled a link to a preproduction webpage and went from there.

[GlobeRunner]

Anoop, when a 'development' or 'preproduction' website or subdomain is getting indexed, that means that you haven't stopped the search engines from crawling it. The search engines, especially Google, are very aggressive at crawling, and they will crawl just about any URL that they find. It seems as though all you have to do is visit that page and it's going to get crawled.

Best way to stop Google from crawling (then indexing) a website is to stop it from getting crawled using the robots.txt file. Keep in mind, though, that even if you tell them to stay out of it using the robots.txt file they will still index those URLs.

The only way to stop Google from crawling would be to password protect the website or make it available only on a private server, or available via VPN only.

[MattRoney]

Hi Anoop! Have everyone's answers helped? Do you still have any questions?

[Chris_Hickman]

As Eric hinted, the best method to prevent any pages being indexed would be to use htaccess password protection dialog on your development site. It's fairly easy to implement. You can find instructions to do so here: http://www.htaccesstools.com/articles/password-protection/