Robots.txt

[BailHotline]

Hello Everyone,

The problem I'm having is not knowing where to have the robots.txt file on our server.

We have our main domain (company.com) with a robots.txt file in the root of the site, but we also have our blog (company.com/blog) where were trying to disallow certain directories from being crawled for SEO purposes...

Would having the blog in the sub-directory still need its own robots.txt? or can I reference the directories i don't want crawled within the blog using the root robots.txt file?

Thanks for your insight on this matter.

[Naghirniac]

You only need a robot file at your main directory and it is used for the whole website.

If you want to have more info's about robots.txt, there is an very good post from Lindsay: http://www.seomoz.org/blog/robot-access-indexation-restriction-techniques-avoiding-conflicts

With this I think it will be easier for you to go pro in robots files. Good luck!

[BailHotline]

Thanks Naghimiac,

Your link is very resourceful, but on the other hand I was looking for something more specific as to blogs being in a sub-directory. I know by default WordPress has its own .htaccess file in the root of the blog directory and I have a separate .htaccess file in the root of my main domain. This is why I was thinking it needed its own robots.txt file.

Is the robot.txt known for only being in the root level of the main directory even if a blog is in a sub-directory?

[john4math]

The bots won't care about that.  If you have your site on www.company.com, your robots.txt will reside at www.company.com/robots.txt, and its directives will apply to any pages living under www.company.com.  When a bot comes to www.company.com/blog, it'll look for the robots.txt at www.company.com/robots.txt to see if it's allowed to crawl there.  It won't look in a subdirectory. Robots.txt always resides on the root level.

If you had your blog at blog.company.com instead of company.com/blog, then you would have to have a separate robots.txt at blog.company.com/robots.txt.  As you have your blog in a subdirectory rather than a subdomain, one robots.txt is all you need.

[BailHotline]

Thanks John & Naghimiac,

Both your responses helped me understand the robots.txt file and the proper ways of implementing it.

Thanks again for all your help!