How do I use the Robots.txt "disallow" command properly for folders I don't want indexed?

[SpringMountain]

Today's sitemap webinar made me think about the disallow feature, seems opposite of sitemaps, but it also seems both are kind of ignored in varying ways by the engines.

I don't need help semantically, I got that part. I just can't seem to find a contemporary answer about what should be blocked using the robots.txt file.

For example, I have folders containing site comps for clients that I really don't want showing up in the SERPS. Is it better to not have these folders on the domain at all?

There are also security issues I've heard of that make sense, simply look at a site's robots file to see what they are hiding. It makes it easier to hunt for files when they know the directory the files are contained in. Do I concern myself with this?

Another example is a folder I have for my xml sitemap generator. I imagine google isn't going to try to index this or count it as content, so do I need to add folders like this to the disallow list?

[saibose]

You can type the following syntax:

after User-agent: *

Disallow: /foldername/subfoldername

also, you can name your sitemaps in the robots.txt file.

They can be defined as

Sitemap: http://www.yourdomain.com/sitemap.xml

If you have multiple sitemaps, you can have multiple sitemaps listed.

[KrisRoadruck]

You may also want to think about slapping a robots noindex on the individual pages as well.

[KeriMorgret]

Hi Jay,

There's actually a recent similar discussion at http://www.seomoz.org/q/what-reasons-exist-to-use-noindex-robots-txt regarding deciding what to block via robots.

For site comps for clients, you could also password-protect those to help hide them, or do a different domain that you have entirely excluded in robots. I've also seen services like Basecamp used for posting comps. It all depends on how much you want to hide the comps.

You do want your sitemap itself to be crawled, but I'm presuming this is in the root directory so that shouldn't be a problem. Folders like your sitemap generator and other purely-framework folders can certainly be disallowed. Blocking the files that list the version of your website (if you're using a CMS) can help prevent people from searching for opportunities to hack that version and finding your site.

Also, just do a site:domain.com search on your domain, see what's indexed, see what content from there you don't want indexed, and use that as a starting point.

Are you running on a content management system, or a custom site? For a CMS, here are example robots.txt files for several popular CMSs. http://www.stayonsearch.com/robots-txt-guide

[portalseo]

Hi,

Usin;

User-agent: *
Disallow: /folder/subfolder

is fine, however if you have information stored in your website that you certainly want crawled make sure it is in your site map and use ...

User-agent: *
allow: /folder/subfolder

adding a no follow attribute to all of your pages wont be practical, if a spam crawler ignores the robots.txt it will ignore your no follow attribute. If anything new occurs with robots.txt check large website's robots.txt as they always update to new trends i.e

www.google.com/robots.txt

Hope this helps:)