Blocking robots.txt

[sesertin]

Do you know any methods to block robots.txt file from external users?

[SimonCullum]

Hi Zsolt

I don't believe there is a way to, not in a Windows environment anyway.

Robots.txt has to be hosted at website root level, as in  example.com/robots.txt which is where search engine bots look for it.  If a bot can find it (which they usually need to if one exists) then a person can too.

If there are pages or folders that you don't want anybody to know about, exclude them from robots.txt and edit the Meta Robots tag on each page concerned to e.g. NoIndex, Follow or whatever is appropriate.

Regards

Simon

[AlanMosley]

There is no setting or feature to do this, but you can do it with code
You could detect the useragnet and create a robot.txt dynamically depending on the useragent.
But i cant see why you would want to do this.

[de4e]

You can block on server side all IP except Google bot for any file, but it may lead to ban, because of cloaking.