Duplicate URL errors when URL's are unique

[STP_SEO]

Hi All,

I'm running through MOZ analytics site crawl report and it is showing numerous duplicate URL errors, but the URLs appear to be unique. I see that the majority of the URL's are the same, but shouldn't the different brands make them unique to one another?

http://www.sierratradingpost.com/clearance~1/clothing~d~5/tech-couture~b~33328/

http://www.sierratradingpost.com/clearance~1/clothing~d~5/zobha~b~3072/

Any ideas as to why these would be shown as duplicate URL errors?

[DirkC]

There is long article on the dev blog how they determine whether pages are duplicates - check https://moz.com/devblog/near-duplicate-detection/ - it's quite technical stuff - but this is the part which might interest you:

"This leads to one of the questions we get asked a lot: Why do I see duplicate content warnings in the context of Custom Crawl for pages that I see as different. Ultimately, it’s always because of the same reason: because no dechroming is done, there is a small amount of unique content relative to the total content. One of the places where this crops up a lot is web stores, where there’s a large amount of chrome layout, but only a short product description associated with it."

Dechroming : removing things like navigation, footer, ..etc from the page (exact def. to be found in the article)

If you compare both pages - apart from the image & product title there isn't too much difference between them so the crawler sees only a very small % of content which is different and marks them as duplicates.

Dirk