Should our social network put all of our member profiles in the site map?

[JoeCotellese81]

We're working on some changes to our site and want to add a site map file to give some hints to search engines on our member profiles. Our site has millions of member profiles. I want to ensure that Google indexes them.

Is there a benefit to submitting millions of profiles across multiple site maps?

Is it better to pick a sample of those and submit them? Most member profiles have links to other member profiles.

Is there any benefit to taking a single site map of 50k links and then changing it at some frequency so that all of the member profiles are eventually indexed?

Thanks.

[Malarowski]

Benefiting or not, make sure users have the ability to turn this off to avoid privacy problems.

It might create a mess though if they link back and forth.

[JoeCotellese81]

Yes, we have the concept of public / private profiles. Private profiles aren't included in the index.

[john4math]

I would definitely do this.  Since you have so many profiles, you'll want to have one main sitemaps linking to smaller sitemaps with the profile URLs.  One of the sites I work on has several million activities created by users that we put into sitemaps.  We automatically generate our sitemaps files frequently and add the new activities created.  We include a last modified date for each activity as well so the search bot will know if anything has changed since the last time it's indexed it.

I would create a sitemap system where all of your profiles could be found, and by including the last modified date, you can leave it up to the searchbot as to whether or not a profile has been updated and needs to be re-indexed.  There are a couple other properties you could use listed on http://www.sitemaps.org/protocol.php as well.

[akaigotchi]

Totally agree with mike and john.

[JoeCotellese81]

This was the approach we were considering. However, what gave me pause was looking at other social networks are doing.

• LinkedIn only has 50k profiles.
• Meez has multiple index files and looks like they are indexing many of their profiles
• Facebook and MySpace don't appear to use Sitemaps (although they do have a crawlable member directory)
• WeeWorld has 500 links

The point is there doesn't seem to be any consistency.

If we end up creating a human readable directory ala Facebook does that achieve the same thing?