Robots.txt questions...

[Horizon]

All,

My site is rather complicated, but I will try to break down my question as simply as possible.

I have a robots.txt document in the root level of my site to disallow robot access to /_system/, my CMS. This looks like this:

# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

**User-agent: ***
Disallow: /_system/

I have another robots.txt file in another level down, which is my holiday database - www.mysite.com/holiday-database/ - this is to disallow access to /holiday-database/ControlPanel/, my database CMS. This looks like this:

**User-agent: ***
Disallow: /ControlPanel/

Am I correct in thinking that this file must also be in the root level, and not in the /holiday-database/ level? If so, should my new robots.txt file look like this:

# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

**User-agent: ***
Disallow: /_system/
Disallow: /holiday-database/ControlPanel/

Or, like this:

# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

**User-agent: ***
Disallow: /_system/
Disallow: /ControlPanel/

Thanks in advance.

Matt

[Marcus_Miller]

Hey Matt

The first of your options looks right and google and other engines look for the robots.txt file in the site root rather than for each directory.

If you had a reason for not wanting that info in the root robots.txt file you can always use  the robots meta tag on the pages in a given directory.

Few useful links:

Robots.txt
http://www.google.com/support/webmasters/bin/answer.py?answer=156449&&hl=en

Robots Meta Tag
http://www.google.com/support/webmasters/bin/answer.py?answer=93710

Hope that helps buddy

Marcus

[YannickVeys]

Like:

# /robots.txt file for http://webcrawler.com/
# mail webmaster@webcrawler.com for constructive criticism

**User-agent: ***
Disallow: /_system/
Disallow: /holiday-database/ControlPanel/

Search engines typically only look in the root of your domain to find robots.txt and sitemap.xml files.

[Horizon]

Cheers gents.

[johnshearer]

Good answer Yannick.

here are some resources:

http://www.free-seo-news.com/all-about-robots-txt.htm

http://www.robotstxt.org/robotstxt.html

Good luck