Free Article Previews (Free Registration Wall)

[Yun]

My site is www.optionmonster.com It has a few different areas of "Free" and "Premium" content.

1. We have a first click free program for Google visitors, where they can read the first news article in full.

2. Other visitors and subsequent news article views have a content preview, which includes headline and first paragraph of content, and then asks for a name and email address in order to consume the rest of the free news.

3. We have premium content (paid), and all of those folders are currently on our robots.txt file

My questions are as follows....

a. How much content is needed in front of the "pay/reg wall" in order to still get proper juice from linkbacks/pagerank crawls

b. Should I make the premium content a robots meta noindex tag, or because you can't see the article at all without logging in, is a proper place to include on robots.txt

[RyanKent]

I would not include your premium area in the robots.txt file at all. Google will recognize that they are unable to access the content, and it won't get indexed. There is no need to block it further.

For your other content, as long as you are showing Google the same exact information as you are showing to other visitors, you are fine. If you show Google more then you do to other visitors, then your site can be blacklisted.

How much content is needed in front of the "pay/reg wall" in order to still get proper juice from linkbacks/pagerank crawls

There are many factors involved. There isn't an answer like "400 words and you are good". These type of setups can be tricky. If you have a forum, then you can allow visitors to see the first thread but not any of the replies. By limiting content based on a number of words, you will be cutting off some articles, and it is possible a small article can be viewed in it's entirety.

It is more important your key words are in the content. If I linked to an article with the anchor text of 1980 Corvette, and your article started off discussing cars but you cut it off before discussing the 1980 Corvette, then that link can be devalued.

[Yun]

So you wouldn't No index FOLLOW those pages, in case someone links to them?

[RyanKent]

So you wouldn't No index FOLLOW those pages, in case someone links to them?

I am not sure what I said to offer that impression.

By "those pages" I am guessing you are referring to your premium content? If so, it really doesn't matter what you do with the content as Google can't get to it due to the robots.txt file. Even if you change the meta tag, Google wont be able to see the change.

Yes, it is possible they could try to follow a link but since this is premium content, they wont be able to access it.

Perhaps I am not clear on your Free vs Premium division. In short, any content which you do not desire to be indexed should contain the NoIndex tag, but that only applies if a crawler has access to the content.

[Yun]

Okay. But since GoogleBots can't access the premium regardless, is it redundant to have on the robots.txt file? I thought the best practice was to have almost nothing on the robots.txt file per (http://www.seomoz.org/blog/serious-robotstxt-misuse-high-impact-solutions)

So my question comes down to if google and other bots will hurt you if you have premium sections that they try to crawl but can't, vs ignoring them because they are on the robots.txt

[RyanKent]

Since GoogleBots can't access the premium regardless, is it redundant to have on the robots.txt file?

You are indeed correct the best practice is to minimize your robots.txt file. That is why the very first sentence in my initial response was the suggestion to NOT include your premium sections in the robots.txt file

Will google and other bots will hurt you if you have premium sections that they try to crawl but can't?

No.