What can I do if Google Webmaster Tools doesn't recognize the robots.txt file?

[DotCar]

I'm working on a recently hacked site for a client and and in trying to identify how exactly the hack is running I need to use the fetch as Google bot feature in GWT.

I'd love to use this but it thinks the robots.txt is blocking it's acces but the only thing in the robots.txt file is a link to the sitemap.

Unde the Blocked URLs section of the GWT it shows that the robots.txt was last downloaded yesterday but it's incorrect information.  Is there a way to force Google to look again?

[wrttnwrd]

Looking in my log files, Google hits robots.txt just about every time it crawls our site.

What are you trying to accomplish using fetch as Googlebot? Any chance CURL could do the job for you, or another tool that ignores robots.txt?

[loopyal]

If you have shell access on Linux you can use wget or GET or run lynx.

If google is getting the wrong robots file then your web server must be sending out something other than what you think is the robots file.

What happens if you do this in your browser:

http://yourdomain.com/robots.txt

[cbielich]

File permission maybe? Change it to 777 and try it again

[wrttnwrd]

Eesh I don't recommend 777. 644 or, if you're going to change it right back, 755 at most.

[cbielich]

lol it's a robots text file. what are they going to do. Steal it? I should have clarified do a 777 to make sure that is not your problem, then yes change the permission to be tighter

[wrttnwrd]

No, but they might write to it, modify it, or do all sorts of other nasty stuff I've seen hackers do when they get a hold of any writeable file on a system.