Block an entire subdomain with robots.txt?

[kylesuss]

Hey Ryan,

I wasn't directly involved with the decision to create the subdomain, but I'm told that it is necessary to create in order to bypass certain elements that were affecting the root domain.

Nevertheless, it is a blog and the users now need to login to the subdomain in order to access the Wordpress backend to bypass those elements. Traffic for the site still goes to the root domain.

[john4math]

Placing canonical tags isn't an option?  Detect that the page is being viewed through the subdomain, and if so, write the canonical tag on the page back to the root domain?

Or, just place a canonical tag on every page pointing back to the root domain (so the subdomain and root domain pages would both have them).  Apparently, it's ok to have a canonical tag on a page pointing to itself.  I haven't tried this, but if Matt Cutts says it's ok...

[sprynewmedia]

Sounds like (from other discussions) you may be stuck requiring a dynamic robot.txt file which detects what domain the bot is on and changes the content accordingly.  This means the server has to run all .txt file as (I presume) PHP.

Or, you could conditionally rewrite the /robot.txt URL to a new file according to sub-domain

RewriteEngine on
RewriteCond %{HTTP_HOST} ^subdomain.website.com$
RewriteRule ^robotx.txt$ robots-subdomain.txt

Then add:

User-agent: *
Disallow: /

to the robots-subdomain.txt file

(untested)

[kylesuss]

We have a plugin right now that places canonical tags, but unfortunately, the canonical for the subdomain points to the subdomain. I'll look around to see if I can tweak the settings

[kylesuss]

Thanks for the suggestion. I'll definitely have to do a bit more research into this one to make sure that it doesn't have any negative side effects before implementation

[sprynewmedia]

Option 1 could come with a small performance hit if you have a lot of txt files being used on the server.

There shouldn't be any negative side effects to option 2 if the rewrite is clean (IE not accidently a redirect) and the content of the two files are robots compliant.

Good luck

[kylesuss]

Awesome. We used your second idea and so far it looks like it is working exactly how we want. Thanks for the idea.

Will report back to confirm that the subdomain has been de-indexed.

[sprynewmedia]

You should do a remove request in Google Webmaster Tools.  You have to first verify the sub-domain then request the removal.

See this post on why the robots file alone won't work...

http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts

[kylesuss]

Yeah. As of yet, the site has not been de-indexed. We placed the conditional rule in htaccess and are getting different robots.txt files for the domain and subdomain -- so that works. But I've never done this before so I don't know how long it's supposed to take?

I'll try to verify via Webmaster Tools to speed up the process. Thanks

[sprynewmedia]

Fact is, the robots file alone will never work (the link has a good explanation why - short form: all it does is stop the bots from indexing again).

Best to request removal then wait a few days.

[kylesuss]

Awesome! That did the trick -- thanks for your help. The site is no longer listed