Question on how MOZ identifies duplicate content issues

[IrvCo_Interactive]

I have a paginated gallery of images on one website. The images are all different across the pages, the titles of the pages are different and the captions on the photos are different. But Moz analytics flags these pages as having duplicate content?

Is the level of sensitivity around duplicate content in Moz Analytics the same as Google? What more could we do to make these pages not register as duplicate? Alternately, is there a way to mark these pages as not duplicate in Moz so that it stops reporting a high number of high priority issues which don't exist?

[KeriMorgret]

We don't know the exact sensitivity of Google's duplicate content algorithm. Ours is set to about 95% similar, and looks at the code of the page, not just how the page appears to the user.

Do you also have individual pages for these images? Perhaps the pagination pages themselves don't need to be indexed?

[IrvCo_Interactive]

Hi again,

On this issue, we don't have individual URLs for the images themselves, they pop up in a lightbox style overlay. However, I'm noticing that some of the URLs which Moz is flagging a duplicate are actually not the canonical version. Does the Moz crawler take the canonical tag value into account when flagging duplicate content?

[ErinMcCaul]

Hi there!

Thanks for reaching out. My name is Erin, and I'm on the Moz Help Team.

Campaigns have a 90% tolerance for duplicate content. This includes all the source code on the page and not just the viewable text. So if a URL is at least 90% similar in code to another URL, this warning will appear. Although the pages in question are may appear to be different on the front end, they are actually duplicates based on this percentage (at least the example URLs I checked in your campaigns.)

You can run your own tests using this tool: http://www.webconfs.com/similar-page-checker.php

We don't know what standard Google uses, but it's safe to say they are a bit more sophisticated than us - so you might be okay in this regard as long as you have a couple hundred words of unique text per page. Google won't say how much duplicate content is too much, so we like to be better safe than sorry.

Our crawler does recognize canonicals, so this might be happening because of where your canonicals tags are pointing. Pages are considered duplicates if their canonical tags point to different URLs. For example, DUPLICATE1 is considered a duplicate of DUPLICATE2 if the canonical tag for the first page is CANON1 while the canonical for the second URL is CANON2.

Since the canonical tags point to different pages it is assumed that CANON1 and CANON2 are likely to be duplicates themselves.

Here is how our system interprets duplicate content vs. rel canonical:

Assuming A, B, C, and D are all duplicates,

If A references B as the canonical, then they are not considered duplicates
If A and B both reference C as canonical, A and B are not considered duplicates of each other
If A references C as a canonical, A and B are considered duplicated
If A references C as canonical, B references D, then A and B are considered duplicates

For more information on using canonical tags, check out this great post by our very own Dr. Pete:
http://moz.com/blog/rel-confused-answers-to-your-rel-canonical-questions

Whew! That was a lot of information! If you still have questions, feel free to email us at help@moz.com with examples of any URLs you have questions about.

Happy Tuesday!

Erin