Duplicate pages, overly dynamic URL’s and long URL’s in Magento

[WendyWuTours]

Hi there,

I’ve just completed the first crawl of my Magento site and SEOMOZ has picked up 1,000’s of duplicate pages, overly dynamic URL’s and long URL’s due to the sort function which appends URL’s with variables when sorting products (e.g. www.example.com?dir=asc&order=duration).

I’m not particularly concerned that this will affect our rankings as Google has stated that they are familiar with the structure of popular CMS’s and Magento is pretty popular.

However it completely dominates my crawl diagnostics so I can’t see if there are any real underlying issues.

Does anyone know a way of preventing this?

Cheers,
Al.

[mediabase]

Hi Almenzies, As you mentioned that SEOmoz  repots you by telling that there area 1000 of pages which are having the issues of duplicate content , so below is alink which solves the Duplicate content issues:

Solving the Duplicate Content Issues in Magento.

I hope that your query had been solved.

[WendyWuTours]

Hey Harald, Thanks for your response - I've come across that article whilst googling the issue, but it doesn't specifically deal with the duplicate URL's being crawled and being included in SEOmoz reports. As I say I'm not too worried about any negative impact here as I've implemented canonical URL's and I have a sitemap - however it ruins my SEOmoz crawl diagnostic report by creating 1,000's of errors. Cheers, Al.

[WendyWuTours]

Is it worth trying to tackle this programmatically e.g. if url includes dir= or limit= or order= then include a noindex meta tag on that page?

It’s easy to exclude these parameters in Google Webmaster tools, but again I’d really like to reduce the number of errors reported by seoMOZ as currently I have 10,000 errors due to duplicate content!

[Dr-Pete]

I'm actually a fan of selectively (programmatically) NOINDEX'ing like that. I find that the GWT parameter blocking doesn't always scale well. I'm running into a lot of clients trying to use it on 100s or 1000s (or millions, actually) of pages and Google is mostly ignoring it. Very frustrating.

We're working on features to let you ignore certain warnings/notices if you feel they don't apply, I but I do believe in being proactive about indexation issues. I think they matter a lot more than they used to, especially post-Panda.

I would double-check to see if there's a Magento plug-in to help, as this could be a common problem. Unfortunately, we don't have any Magento experts on-staff. I'll leave this open as a discussion question, in case any members have specific expertise.

[nageldecoratie]

You should use the Yoast Robots extension to fix almost all the duplicate content.

http://www.magentocommerce.com/magento-connect/yoast-metarobots.html

When using 2.0 Magento connect: http://connect20.magentocommerce.com/community/Yoast_MetaRobots

for 1.0 use: magento-community/Yoast_MetaRobots

Also use canonical URL. You can find this at the admin panel:

System - Configuration - Catalog - Canonical links for catagories

System - Configuration - Catalog - Canonical links for products