What is the sense of robots.txt?

[jallenyang]

Using robots.txt to prevent search engine from indexing the page is not a good idea. so what is the sense of robots.txt? just for attracting robots to crawl sitemap?

[KeriMorgret]

There are often times pages you don't want indexed, and that's what robots.txt is there for. These are just some things you may not want indexed:

• Premium content for subscription-only members
• Your admin directory
• Printable versions of pages
• Development servers

You keep things you don't want out of the index, and you also don't waste the crawl budgets of the search engines on stuff that's not what you want in the engines in the first place.

[RyanKent]

While your robots.txt file is not the best means to control search engines, it does have a purpose. To respond to your questions:

• the file does not "attract" any robots, but robots who do visit can learn a bit about your site and understand what content you don't wish to be crawled

• you can block parts of your site that you feel have no value for indexing such as Keri mentioned your "print" version of pages, or overlays pages, or login pages, etc.

The idea is that you own the website, and you can have a measure of control over it. You can disallow specific crawlers, etc. although it's up to each crawler whether they actually respect your wishes.

More details can be read at: http://www.robotstxt.org/