Robot.txt File Not Appearing, but seems to be working?

[J-Banz]

Hi Mozzers,

I am conducting a site audit for a client, and I am confused with what they are doing with their robot.txt file. It shows in GWT that there is a file and it is blocking about 12K URLs (image attached). It also shows in GWT that the file was downloaded 10 hours ago successfully. However, when I go to the robot.txt file link, the page is blank.

Would they be doing something advanced to be blocking URLs to hide it it from users? It appears to correctly be blocking log-ins, but I would like to know for sure that it is working correctly. Any advice on this would be most appreciated. Thanks!

Jared

ihgNxN7

[anthonydnelson]

Be sure you are visiting /robots.txt  In all of your copy above, you are referencing robot.txt

Also, check to see if it possibly is only showing up on the www. version or the site or the non-www version of the site.

To be sure if it's working, you can test URLs of your website within Google Webmaster Tools. Go to Crawl->Blocked URLs and scroll down to the bottom.

[J-Banz]

I verified that I was checking /robots.txt. I had trouble verifying if it was under the non-www because everything redirects to the www. I also checked to see if it was being blocked, and it is not.

I went to Archive.org (Wayback Machine), and I can see the robot.txt file in previous versions of the site. I cannot, however, view it online, even though Google says they are downloading it successfully, and the robots.txt file is successfully blocking URLs from the search index.

[Vizergy]

There is an old webmaster world thread that explains how to hide the robots.txt file from browsers.... not sure why one would do this however....

http://www.webmasterworld.com/forum93/74.htm

Perhaps they are doing something like this?