Robots.txt - blocking JavaScript and CSS, best practice for Magento

[LeapOfBelief]

Hi Mozzers,

I'm looking for some feedback regarding best practices for setting up Robots.txt file in Magento.

I'm concerned we are blocking bots from crawling essential information for page rank.

My main concern comes with blocking JavaScript and CSS, are you supposed to block JavaScript and CSS or not?

You can view our robots.txt file here

Thanks,

Blake

[J-S]

Best practice is not to block access to JS / CSS anymore,  to allow google to properly understand the website and give determine mobile-friendliness.

[Andy.Drinkwater]

As Joost said, you should not block access to files with help in the reading / rendering of the page.

Looking at your Robots file, I would look at the following two exclusions. Do they block anything else that runs on a live page that Google should be seeing?

Disallow: /includes/
Disallow: /scripts/

-Andy