MozBot Finding Duplicate Pages That Aren't Duplicate

[amaray403]

I've been reviewing the technical audits for my campaign in Moz, and noticed I had a number of duplicate content issues that I'm not really sure how to address. When I click on the links of what the duplicates are, they are all different links that have different content/images.

Based on what I was seeing other's wrote in the forum, this could be because the code base is really the same between these pages, and many of these were using query parameters (I'm assuming that is why the code is almost exactly the same across these pages),

so example: website.com/tags/KEYWORD1?type=KEYWORD2 is a duplicate of website.com/tags/KEYWORD3?type=KEYWORD4

I was reading that I can use that URL Parameters area in google search console, but my search console says that the googlebot isn't experiencing issues, so I wasn't sure if that was the right move. I can't do the canonicals because these pages all have different content on them, and I know duplicate content is a big SEO issue, so I really wasn't sure what my next steps should be.

Thanks for the help!

[tawnycase]

Hi there! Tawny from Moz's help team here.

The best way to prevent our crawler from reporting duplicate content for pages you aren't concerned about and don't intend to change would be to block our crawler from these pages using the robots.txt file for the site. For example, it looks like most of the pages reported as duplicates include URL parameters, so you should be able to add a disallow directive for that parameter and any others to block our crawler from accessing them. It would look something like this:

User-agent: Rogerbot
Disallow: ?type

etc., until you have blocked all of the parameters that may be causing these duplicate content errors. You can also use the wild card user-agent * in order to block all crawlers from those pages, if you prefer.

Here is a great resource about the robots.txt file that might be helpful: https://moz.com/learn/seo/robotstxt

I'd recommend checking your robots.txt file in this handy Robots Checker Tool once you make changes to avoid any nasty surprises.

Let us know if we can help with anything else! Just drop us a line at help@moz.com and we'll do our best to get things straightened out for ya.