Robots.txt vs noindex

[94501]

I recently started working on a site that has thousands of member pages that are currently robots.txt'd out.

Most pages of the site have 1 to 6 links to these member pages, accumulating into what I regard as something of link juice cul-d-sac.

The pages themselves have little to no unique content or other relevant search play and for other reasons still want them kept out of search.

Wouldn't it be better to "noindex, follow" these pages and remove the robots.txt block from this url type? At least that way Google could crawl these pages and pass the link juice on to still other pages vs flushing it into a black hole.

BTW, the site is currently dealing with a hit from Panda 4.0 last month.

Thanks! Best... Darcy

[LesleyPaone]

I think what I am going to say is going to sound like it is going against the grain, but it really isn't. I have noticed in some places if you want an active community, you reward your members. Look at how moz does their forum, they don't really noindex the pages, but once you hit a point they psuedo drop the nofollow off of your profile link (it could be argued whether they really do). But the point is reward your members that are active. I would set up some automatic noindex tag in the header that grabbed the users post numbers. Then you can noindex all of the spammers and have prominent members shown in the search. If it were me that is how I would do it. I have a PA of 49 on my profile in one forum I regular, I have seen the stats, it is regularly an entry page to the forum. Another member has a 64 on a 93 domain, his is used a lot more than mine for entry as well. Think of it this way, if someone is googling my name, the second result is http://screencast.com/t/jIx7a4hcWV  Moz's forum. 2nd search results still get a lot of clicks.

[94501]

HI Lesley,

Thanks for the thoughts. I don't see this as a real option for a number of reasons, including but not limited to that there are 50,000 profiles, most with very little information. The members of this site are 95% busy professionals who aren't trying to advance their career via their profile. So, there'd be some privacy concern and the potential for tens of thousands of low content/highly templated pages. Not really a search dream come true!

Also, converting it into a system where different levels of profile completeness are acknowledged would not really resonate with this community nor would it be near the top of our engineering priorities.

What I really want to get clear on is how best to keep them search invisible while not losing link value into a robots.txt'd black hole. Really just looking for confirmation if, with those goals, "noindex, follow" and remove from robots is the way to go. I'm pretty sure it is, but would like to hear more about that.

Thanks... Darcy

[Saijo.George]

Since you said  " The pages themselves have little to no unique content or other relevant search play and for other reasons still want them kept out of search. " I would use  meta robots "noindex, follow"

[94501]

Hi Saijo,

Thanks for the response. Do you think that would yield the benefit I'm looking for of recapturing that lost link juice?

Do you think there'd be any downside to the switcheroo from robots.txt to noindex, follow?

Best... Darcy

[Saijo.George]

if you add the meta noindex, follow tag , it will keep the page out of the SERP but allows pagerank to flow through them to other pages.

See this interview of Matt Cutts for more info : http://www.stonetemple.com/articles/interview-matt-cutts.shtml