Does using robots.txt to block pages decrease search traffic?

[nicole.healthline]

I know you can use robots.txt to tell search engines not to spend their resources crawling certain pages.

So, if you have a section of your website that is good content, but is never updated, and you want the search engines to index new content faster, would it work to block the good, un-changed content with robots.txt? Would this content loose any search traffic if it were blocked by robots.txt? Does anyone have any available case studies?

[KeriMorgret]

You certainly don't want to block this content!

One thing I'd consider is the if-modified-since header, or other headers. Here are two articles that explain more about the concept of using headers to tell the search engines " this hasn't changed, don't bother crawling it". I haven't personally used this, but have read about it in many places.

http://www.feedthebot.com/ifmodified.html

http://searchengineland.com/how-to-improve-crawl-efficiency-with-cache-control-headers-88824

[nicole.healthline]

Agreed, that is a better solution, but, I am still wondering if you block something with robots.txt, will that lead to a decrease in traffic? What if we have some duplicate content that is highly trafficked, if we block it with robots.txt, will the traffic numbers change?

[KeriMorgret]

If you block the pages from being crawled, you are also telling the search engines to not index the pages (they don't want to include something they haven't looked at). So yes, the traffic numbers from organic search will change if you block the pages in robots.txt.