The SEOmoz crawler is being blocked by robots.txt need help

[CGR-Creative]

SEO moz is showing me that the robot.txt is blocking content on my site

[PrivateNameAM]

I would look at your htaccess file.

[CGR-Creative]

what needs to be done in the htaccess file. ? can anyone give me a step by step process

[PrivateNameAM]

Open your htaccess file by adding .txt to it and see if it blocks certain robots from crawling your pages. If it does then remove these. Put the file back on your server. Remove the .txt

[PrivateNameAM]

Or simply - another way - another idea: Go to your robots.txt and see what is going on directly.

You can use Google Webmaster tools to help you make a proper robots.txt file.

Best of luck

[Tom-Anthony]

Hi Jason,

What's in your robots.txt file? It will be a text file in the root directory of your website. If you could share the contents we can help.

[KeriMorgret]

Jason, if you can post the contents of your robots.txt file, or give us a link to the site in question, we can help you diagnose what is happening.

A second question is -- what type of content is being blocked? If it's a directory like /admin that is being blocked, the robots.txt is likely working as intended.

You can also verify your site in Google Webmaster Tools and look in there at the crawling section, as it will tell you what pages Googlebot hasn't been able to crawl. Google offers some help at http://googlewebmastercentral.blogspot.com/2008/03/speaking-language-of-robots.html.