Not using a robot command meta tag

[STPseo]

Hi SEOmoz peeps.

Was doing some research on robot commands and found a couple major sites that are not using them.

If you check out the code for these:

http://www.amazon.com

http://www.zappos.com

http://www.zappos.com/product/7787787/color/92100

http://www.altrec.com/

You fill not find a meta robot command line.

Of course you need the line for any noindex, nofollow, noarchive pages.

However for pages you want crawled and indexed, is there any benefit for not having the line at all?

Thanks!

[StalkerB]

Doesn't matter if the line's not there if you don't want the robots to do anything. Not including it just assumes the position that they can do whatever they're doing ie Index, Follow

All of those sites do have a few things in robots.txt though (and the meta is really for people that can't edit robots.txt)

http://www.amazon.com/robots.txt

http://zappos.com/robots.txt

http://www.altrec.com/robots.txt

[Malarowski]

Barry is spot on. You either have the meta robot line, or you do your robots.txt. I prefer to use the txt because it shows me all the rules I impose on the website at a glance and doesn't make me go and search each page individually if anything needs to be changed. Imagine having a website with 20000 pages where each is individually adjusted. If you suddenly need to change something it will take you forever. With a robots.txt it will literally take seconds.