Is it terrible to not have robots.txt ?

[StandUpCubicles]

I was under the impression that you really should have a robots.txt page, and not having one is pretty bad. However, hubspot (which I'm not impressed with) does not have the capability of properly implementing one. Will this hurt the site?

[john4math]

It won't hurt the site.  You only need one if you want to disallow parts of your site to search engines, or disallow different search bots.  If you don't have any pages or directories to disallow, I wouldn't worry about it.

[AU-SEO]

Technically you don't need one.

However, It is so easy to put one in place (usually) and I would consider it a best practice. Like developing an application without taking notes on implementation. What happens when someone new comes along to work with it. They will ask the same question. Not having one and deliberately allowing the robots to crawl all are two inherently different things.

[ShaMenz]

Hi Jaycie,

Google's view of the issue is that you should have a robots.txt file in order to eliminate the risk of your web host dealing with requests in an unexpected way and returning something strange.

Matt Cutts talked about robots.txt in this Webmaster Help Video last month.

Hope that helps,

Sha

[crvw]

And for reference, this Search Engine Land column from 2009 provides a pretty concise treatment of how to utilize our friend the robots exclusion protocol: A Deeper Look At Robots.txt

[StandUpCubicles]

Thank you everyone! Great stuff