Is our robots.txt file correct?

[BMPIRE]

Could you please review our robots.txt file and let me know if this is correct.

www.faithology.com/robots.txt

Thank you!

[StreamlineMetrics]

There are some errors, but since I'm not sure what you are trying to accomplish, I recommend checking it with a tool first. Here is a great tool to check your robots.txt file and give you information on errors - http://tool.motoricerca.info/robots-checker.phtml

If you still need assistance after running it through the tool, please reply and we can help you further.

[BMPIRE]

Thank you for the reply. We want to allow all crawling, except for rogerbot in the community folder.

I have updated the robots.txt to the following, does this look right?:

User-agent: *
Disallow:

User-agent: rogerbot
Disallow: /community/

User-agent: Mediapartners-Google
Disallow:

Sitemap: http://www.faithology.com/sitemap.xml

view the robots here: http://www.faithology.com/robots.txt

[mememax]

Hi, it seems correct to me however try to use the robots.txt checker tool in GWTools. You may try to include a couple of your urls and see if google can crawl them.

I find only redundant the follwing rule:

User-agent: Mediapartners-Google.

If you have already set up a disallow: rule for all bot excluding rogerbot which can't access the community folder why create a new rule stating the same for mediapartners?

Again, why are you saying to all bots they can access the entire site, being that the default rule? Avoid those lines, include just the rogerbot and sitemaps rule and you're done.

[Igal_Zeifman]

What's the end goal here?
Are you actively trying  to block all bots?

If so, I would still suggest "Disallow:/".
The other syn-text may also work, but if Google suggests using a backslash, you should probably use it.