How to Disallow Tag Pages With Robot.txt

[monster99]

Hi i have a site which i'm dealing with that has tag pages for instant -

http://www.domain.com/news/?tag=choice

How can i exclude these tag pages (about 20+ being crawled and indexed by the search engines with robot.txt

Also sometimes they're created dynamically so i want something which automatically excludes tage pages from being crawled and indexed.

Any suggestions?

Cheers,

Mark

[DeanAndrews]

Hi Mark

If your using Wordpress then I would recommend SEO Yoast to resolve the tag issue. If not then I suggest you amend the robots.txt file to resolve.

Here is an example:

Disallow: /?tag=
Disallow: /?subcats=
Disallow: /*?features_hash=

NOTE:

Be very careful when blocking search engines. Test and test again!

[NakulGoyal]

I agree. I would suggest adding the noindex on the pages and letting the bots crawl them. Blocking them would prevent future crawl of these pages, but I am guessing you would also want to remove the existing pages.

Therefore add the noindex first, wait a few days and then add the disallow (Although technically if they are noindex, you don't really need the disallow).

[monster99]

Thanks, is there a way to test it out before actually implementing it with the site.

The site is non-wordpress aswell.

Cheers,

Mark

[NakulGoyal]

What CMS is it Mark ?

[monster99]

Hi Nakul, its Drupal

Mark