What should I block with a robots.txt file?

[Travis-W]

Hi Mozzers,

We're having a hard time getting our site indexed, and I have a feeling my dev team may be blocking too much of our site via our robots.txt file.

They say they have disallowed php and smarty files.

Is there any harm in allowing these pages?

Thanks!

[Andy.Drinkwater]

I'm not too hot on Smarty, but doesn't this generate the HTML templates?

However, this shouldn't cause a problem because the files that are being generated are html so as long as they have done this right, it should be fine.

Do you want to ping me the robots file or URL over and I will have a look for you?

Andy

[Travis-W]

Hi Andy, here you go: www.consumerbase.com/robots.txt

I know we want to block the .html files, but I am unsure about the other folders.

I guess I would need to know for certain from my programmers that none of our content is in there?