Block subdomain directory in robots.txt

[gamesecure]

Instead of block an entire sub-domain (fr.sitegeek.com) with robots.txt, we like to block one directory (fr.sitegeek.com/blog).
'fr.sitegeek.com/blog' and 'wwww.sitegeek.com/blog' contain the same articles in one language only labels are changed for 'fr' version and we suppose that duplicate content cause problem for SEO. We would like to crawl and index 'www.sitegee.com/blog' articles not 'fr.sitegeek.com/blog'.

so, suggest us how to block single sub-domain directory (fr.sitegeek.com/blog) with robot.txt?

This is only for blog directory  of 'fr' version even all other directories or pages would be crawled and indexed for 'fr' version.

Thanks,
Rajiv

[DirkC]

The easiest way would be to put the robots.txt in the root of your subdomain & block access for search engines

User-agent: Googlebot
Disallow: /

If you subdomain & the main domain are sharing the same root - this option is not possible. In that case, rather than working with robots.txt I would add a canonical on each page pointing to the main domain, or block all pages in the header (if this is technically possible)

You could also check these similar questions: http://moz.com/community/q/block-an-entire-subdomain-with-robots-txt and http://moz.com/community/q/blocking-subdomain-from-google-crawl-and-index - but the answers given are the same as the options above.

Apart from the technical question, qiven the fact that only the labels are translated, these pages make little sense for human users. It would probably make more sense to link to the normal (English) version of the blog (and put (en Anglais) next to the link.

rgds,

Dirk

[N1ghteyes]

Just to add to this, if your subdomain has more than /blog on it, and you only want to block /blog, change Dirk's robots.txt to:

User-agent: Googlebot
Disallow: /blog

or to block more than just google:

User-agent:*
Disallow: /blog

[gamesecure]

Thanks Dirk,

we will fix the issue as you suggested.

Could you explain more on duplicate content if we post articles on both 'FR' and 'EN' versions?

Thanks,

Rajiv

[DirkC]

Hi Rajiv,

If you post the same content on both FR & EN version:

• if both are written in English (or mainly written in English) - best option would be to have a canonical pointing to the EN version
  Example: https://fr.sitegeek.com/category/shared-hosting - most of the content is in English - so in this case I would point a canonical to the EN version

• if the FR version is in French - you can use the HREF lang tag - you can use this tool to generate them, check here for common mistakes and doublecheck the final result here.

Just some remarks:

• partially translated pages offer little value for users - so it's best to fully translate them or only refer to the EN version

• I have a strong impression that the EN version was machine translated to the FR version. (ex. French sites never use 'Maison' to link to the Homepage - they use Acceuil). Be aware that Google is perfectly capable to detect auto-translated pages and they consider it to be bad practice (check this video of Matt Cutts - starts at 1:50). So you might want to invest in proper translation or proofreading by a native French speaker.

rgds

Dirk