Google Indexing Development Site Despite Robots.txt Block

[CarlWint]

Hi,

A development site that has been set-up has the following Robots.txt file:

User-agent: *

Disallow: /

In an attempt to block Google indexing the site, however this isn't the case and the development site has since been indexed.

Any clues why this is or what I could do to resolve it?

Thanks!

[DeanAndrews]

You would need to block access via htaccess rather than robots file as the robots.txt is only advisory

If you are using wordpress I use this simple plugin JF3 Maintenance Redirect

[CarlWint]

Its my understanding that htaccess is PHP based and as we code in .net we don't have a htaccess file.

Do you know of this this happening before because its not something that I've heard of.

[Travis_Bailey]

It's generally best to noindex/nofollow using the meta robots tag in the header. If it's not too much of a stretch for you, you can also password protect the test site. The over-so-lovely and charming Googles will still display results blocked by robots.txt - though it won't generally cache the content. If you would like, you can hookup the test site with Webmaster Tools and remove the URL(s) from the index.

More on all this here and here.

[CarlWint]

Thanks for your help! Much appreciated

[DeanAndrews]

Hi so I'm assuming your on IIS (I'm no expert on ISS I think you will need to configure the web.config) and I'm just going to step back now and get my coat as I only have experience with Apache