Magneto site with many pages

[ciznerguy]

just finsihed scan to a magento site.

off course I am getting thousand of pages that are dynamic.

search pages and other.

checking with site command on Google I see 154,000 results

which pages it is recommended to block?

some people are talking about blocking the search pages and some actually talking about allowing them?

any answer on this?

Thanks

[Benj25]

Hi there! When you say block, do you mean through your robots.txt?

[FuseoUK]

I've had similar problems with a few Magento sites.  This is a standard list I use in my robots.txt files (below.)  I hope it helps.

You don't have to include all the Magento folders like 'app' 'lib' 'var' and 'admin' t etc hey are just there to be thorough.

I think you'll get the idea.  I've brought the number of indexed pages down from half a million to just a few thousand using these.

Disallow: /*?
Disallow: /*.js$
Disallow: /*.css$
Disallow: /404/
Disallow: /admin/
Disallow: /api/
Disallow: /app/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalog/product_compare/
Disallow: /catalogsearch/
Disallow: /catalogsearch/advanced/
Disallow: /catalogsearch/term/
Disallow: /catalogsearch/term/popular/
Disallow: /cgi-bin/
Disallow: /checkout/
Disallow: /checkout/cart/
Disallow: /contacts/
Disallow: /contacts/index/
Disallow: /contacts/index/post/
Disallow: /customer/
Disallow: /customer/account/
Disallow: /customer/account/login/
Disallow: /downloader/
Disallow: /install/
Disallow: /js/
Disallow: /lib/
Disallow: /magento/
Disallow: /newsletter/
Disallow: /pkginfo/
Disallow: /private/
Disallow: /poll/
Disallow: /report/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /skin/
Disallow: /tag/
Disallow: /var/
Disallow: /wishlist/

[ciznerguy]

yes my thought is blocking through robot.txt

[ciznerguy]

Hi

sorry for responding late.

what was the affect on your results when u did block all those pages?

Thanks

[FuseoUK]

There was no effect on the rankings, no significant ups or downs.  But now when I do the site:www.domain.com command in Google, I see just the pages that I want Google to index.

It can only help in the long run I guess.