Does It Really Matter to Restrict Dynamic URLs by Robots.txt?

[CommercePundit]

Today, I was checking Google webmaster tools and found that, there are 117 dynamic URLs are restrict by Robots.txt. I have added following syntax in my Robots.txt You can get more idea by following excel sheet.

#Dynamic URLs

Disallow: /?osCsidDisallow: /?q=

Disallow: /?dir=Disallow: /?p=

Disallow: /*?limit=

Disallow: /*review-form

I have concern for following kind of pages.

Shorting by specification:

http://www.vistastores.com/table-lamps?dir=asc&order=name

Iterms per page:

http://www.vistastores.com/table-lamps?dir=asc&limit=60&order=name

Numbering page of products:

http://www.vistastores.com/table-lamps?p=2

Will it create resistance in organic performance of my category pages?

[SanketPatel]

Hi,

Instead of blocking those URLs, You can use "URL parameter" setting in Google webmaster tool. You will get parameters like "?dir" & "?p" in it,  select appropriate option from that like what actually happens when this parameter come into picture.

[Cyrus-Shepard]

Robots.txt isn't the best solution for dynamic URLs. Depending on the type of URL, there are a number of other solutions available.

1. As blurbpoint mentions, Google Webmaster Tools allows you to specify URL handling. They actually do a decent job of this automatically, but also allow you the option to change the settings yourself.

http://support.google.com/webmasters/bin/answer.py?hl=en&answer=1235687

2. Identical pages with different parameters can create duplicate content, which is often best handled with canonical tags.

3. Parameters that result in pagination may require slightly nuanced solutions. I won't get into them all here but Adam Audette gives a good overview of pagination solutions here: http://searchengineland.com/the-latest-greatest-on-seo-pagination-114284

Hope this helps. Best of luck with your SEO!

[CommercePundit]

I am quite late to add my reply on this question. Because, I was busy to fix issue regarding dynamic URLs.

I have made following changes on my website.

1. I have re-write all dynamic URLs and make it static one exclude session ID and internal search option. Because, I have restricted both version via Robots.txt.
2. I have set canonical to near duplicate pages which Dr.Pete described in Duplicate content in post panda world.

I want to give one live example to know more about it.

Base URL: http://www.vistastores.com/patio-umbrellas

Dynamic URLs: It was dynamic but, I have re-write to make it static one. But canonical tag to base URL is available on each near duplicate pages which are as follow.

http://www.vistastores.com/patio-umbrellas/shopby/limit-100
http://www.vistastores.com/patio-umbrellas/shopby/lift-method-search-manual-lift
http://www.vistastores.com/patio-umbrellas/shopby/manufacturer-fiberbuilt-umbrellas-llc
http://www.vistastores.com/patio-umbrellas/shopby/price-2,100
http://www.vistastores.com/patio-umbrellas/shopby/canopy-fabric-search-sunbrella
http://www.vistastores.com/patio-umbrellas/shopby/canopy-shape-search-hexagonal
http://www.vistastores.com/patio-umbrellas/shopby/canopy-size-search-7-ft-to-8-ft
http://www.vistastores.com/patio-umbrellas/shopby/color-search-blue
http://www.vistastores.com/patio-umbrellas/shopby/finish-search-black
http://www.vistastores.com/patio-umbrellas/shopby/p-2
http://www.vistastores.com/patio-umbrellas/shopby/dir-desc/order-position

Now, I am looking forward towards Google crawling and How Google treat all canonical pages. I am quite excited to see changes in organic ranking with distribution of page rank in website. Thanks for your insightful reply.