Indexation of content from internal pages (registration) by Google

[guitarslinger]

Hello,

we are having quite a big amount of content on internal pages which can only be accessed as a registered member.

What are the different options the get this content indexed by Google?

In certain cases we might be able to show a preview to visitors. In other cases this is not possible for legal reasons.

Somebody told me that there is an option to send the content of pages directly to google for indexation. Unfortunately he couldn't give me more details. I only know that this possible for URLs (sitemap). Is there really a possibility to do this for the entire content of a page without giving google access to crawl this page?

Thanks

Ben

[Keszi]

Hi Ben,

Maybe fetch as Google Bot can be a solution to your issue. But not 100% sure of this.

Gr.,

Istvan

[guitarslinger]

Hi Istvan,

"The Fetch as Googlebot tool lets you see a page as Googlebot sees it."

Since Googlebot has no access to the entire site (login required) it will probably not display anything (just tried it logged in and it would not display any of the content). How could this tool theoretically help us indexing the content of the internal page?

Ben

[Copstead]

I guess my questions is why would you want Google to index something that is only available to registered users?

In order for it to be indexed, it has to be open to everyone.

You will have to figure out what can be shown as a preview and what can't.  If you want something to be indexed, then you will have to create a separate section for your preview content (since Google won't index your protected content.)

[Keszi]

Fetch as Google Bot is submit to index. This is why I believe it should work with it.

[SEOExecutive20]

I found this information for you but you should definitely check that it doesn't break any of Google's guidelines before incorporating it to your website.

This is a simple code to allow bots to bypass the password on password protected pages

$allow_inside = ($is_logged_in) || substr_count($_SERVER['HTTP_USER_AGENT'],'Googlebot');

http://davidwalsh.name/google-password-protected-areas

The reference post is older, so this code could have been updated

[guitarslinger]

As I mentioned we have 2 cases.

In the first case, we can show a preview.

In the second case we can only show the content to a certain audience (which is a legal question). So the free registration is a legal requirement. Still people will be looking for it via Google. Since the content found on those pages is useful for a fairly large audience so why wouldn't we want Google to index the pages. Of course without Google knowing that there is relevant content on those pages, they will neither index nor propperly rank those pages.

[guitarslinger]

Thanks, I will check wether this meets the legal requirements (see my reply to Brents answer).

[Copstead]

The issue is that Google won't and shouldn't index pages that are restricted.

This is best for user experience.  Most people won't sign in to view the content.

You basically have to create two sites.  One that is visible to all, and Google where you show or preview a bit.  then the other that is protected.