For large sites, best practices for pages hidden behind internal search?

[vlevit]

If a website has 1M+ pages, with most of them being hidden behind an internal search, what's the best way to get pages included in an engine's index?

Does a direct clickpath to those pages need to exist from the homepage or other major hub pages on the site?

Is submitting an XML sitemap enough?

[Keszi]

Hi vlevit,

The best practice would be to exist a direct path of flow from index page. Something like: index -> category(filter) -> subcategory(filter) -> page/product. But in some cases xml sitemaps can also help you in indexing.

BUT, beware with to large XML sitemaps, try to create more then one sitemap, group them as possible.

A few very good resources can be found under the next links:

http://www.seomoz.org/ugc/solving-new-content-indexation-issues-for-large-b2b-websites

http://www.seomoz.org/qa/view/29009/sitemaps-management-for-big-sites-tens-of-millions-of-pages

I hope it helpes,

Istvan

[ResslerMotors]

Make sure they are part of the actual structure of your website, not just part of search.  Meaning, you have to have links pointing at them.  Also, you will also want to make sure that those pages have value.

[vlevit]

This is what I was assuming, but was wondering if there was a clever way around creating direct click paths to those pages, while still maintaining their importance to the site. Thanks for the info.

[Everett]

Hello Vlevit,

If you can be more specific we may be able to be of more help. Google doesn't want you to show internal search result pages, but if this is a different type of situation it there may be an exception. Are these search result pages, product pages, category pages, content pages.... is it an eCommerce site, community, content site... ?

Generally speaking, 1M+ pages with no links going into them and content that is either sparce/thin or partially/fully duplicated on other similar pages (like a search for widgets and a search for green widgets showing overlapping content) is exactly the type of thing that will get you in hot water that would affect even the rankings of your home page.

Do you feel like your question has been answered or would you like to be more specific about your site and goals?

Cheers,

Everett

[vlevit]

Everett, thanks for your reply. I understand the problems of showing internal search pages. I'm not looking to have internal search results being indexed, just the pages that the results link to.  We're in eCommerce.

I was under the impression that there was a clever way to have the individual product pages indexed without establishing a direct click path, but best practices recommend otherwise.

Question answered. Thanks all for your help.

[Everett]

Hello Vlevit,

You could do several things. I recommend giving Google your product feed, which should accomplish your goals. Another possible solution would be to make those search pages noindex,follow so they don't end up getting indexed, but Google can still use them for discovery.

Thanks for explaining the situation.

Below is more on submitting product feeds. It is for Google Product Search, but I would imagine the "link" field where you put the URL to your product detail page will help those pages get indexed in the standard results:
http://support.google.com/merchants/bin/answer.py?hl=en&answer=188494#US

Everett