Robots.txt Showing in SERP Results

[vectormedia]

Currently doing a technical audit for a website and when I search "Site:website.com -www" the only result is website.com/robots.txt

I was wondering if anyone else has come across this before -- or what this may mean from a technical audit standpoint.

Thank you!

[MargaritaS]

Did you also search for site:www.webiste.com? Are they blocking the site?What's in the actual robots file?

Also, does this happen when you search for the site in Bing?

[Copstead]

The robots.txt file should not show up.  Sounds like there is something seriously wrong.

[Cyrus-Shepard]

It's quite common for Google to index robots.txt files. (and also, rather odd) But check out all of these robots.txt files:

https://www.google.com/search?q=inurl%3Arobots.txt&pws=0&gl=us

So it's nothing to be alarmed by. With your particular query. "Site:website.com -www" it only shows pages indexed without the "www" so this just says that all the indexed pages most likely begin with www. The exception, of course, is the robots.txt file.

The bigger question for me is, why does Google cache robots.txt files? Oh well.

[bobjones]

nonsense. Search for https://www.google.com.au/search?q=inurl%3Arobots.txt&pws=0

Some of the first results with visible robots.txt I see are:

• google.com
• facebook.com
• wikipedia.com
• seobook.com

I refuse to believe that "something is seriously wrong" with any of these sites.