Blocking Dynamic URLs with Robots.txt

[AndrewY]

Background:

My e-commerce site uses a lot of layered navigation and sorting links.  While this is great for users, it ends up in a lot of URL variations of the same page being crawled by Google.  For example, a standard category page:

www.mysite.com/widgets.html

...which uses a "Price" layered navigation sidebar to filter products based on price also produces the following URLs which link to the same page:

http://www.mysite.com/widgets.html?price=1%2C250

http://www.mysite.com/widgets.html?price=2%2C250

http://www.mysite.com/widgets.html?price=3%2C250

As there are literally thousands of these URL variations being indexed, so I'd like to use Robots.txt to disallow these variations.

Question:

1. Is this a wise thing to do?  Or does Google take into account layered navigation links by default, and I don't need to worry.

2. To implement, I was going to do the following in Robots.txt:

User-agent: *

Disallow: /*?

Disallow: /*=

....which would prevent any dynamic URL with a '?" or '=' from being indexed.  Is there a better way to do this, or is this a good solution?

Thank you!

[TaitLarson]

I don't know if there's a good solution with robots.txt given your URL structure.  However, you could use the rel=canonical link tag in the header to force google to treat many of your URLs the same way.  This would help you avoid duplicate content penalties.

More on rel=canonical:

http://www.google.com/support/webmasters/bin/answer.py?answer=139394

http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

[AndrewY]

Tait,

Thanks for the answer.  I think the canonical tag would be ideal, but in terms of implementation, it would require some substantial code modification to the site / PHP code as I have a lot of categories, and adding this manually to each one would be very time consuming.

Would preventing the spiders from indexing any URLs with a "?" or "&" (which would only be dynamic URLs variations) cause any problems?  Or is this just not an ideal best practice?

Thanks!

[TaitLarson]

If you are happy with any URLs with query strings not being indexed your robots.txt will work fine.

Do any or your URLs with question marks in them have links to them?  If so you might want to be careful blocking google from indexing them.  I would think you'd lose the benefits those links would pass to your site.