We are still seeing duplicate content on SEOmoz even though we have marked those pages as "noindex, follow." Any ideas why?

[cmaseattle]

We have many pages on our website that have been set to "no index, follow." However, SEOmoz is indexing them as duplicate content. Why is that?

[DanDeceuster]

SEOMoz is not a search engine index, it uses a crawler. If those pages are not blocked by the robots.txt file, then SEOMoz will crawl them. They ignore the noindex tag because they don't index anything. Search engines will honor the noindex tag and not index a page if you specify with the robots meta tag. However, to remove pages from the crawl, disallow them in the robots.txt.

[cmaseattle]

So, it indexes issues that Google does see and doesn't see. How do we differentiate between the two?

Additionally, what would be some suggestions as to what we should do?

[DanDeceuster]

Go to Google and search site:yourdomain.com and see if the pages in question come up. If so, Google has indexed them. If not, Google has not indexed them. Like SEOMoz, Google can crawl any page. Doesn't mean they will index the page. If you have noindexed a page, it should not be indexed by Google and should not be problematic for you.

[ShaMenz]

Hi Gary,

Great answer from Daniel.

One thing that you can do is to create a list of noindexed pages in excel, then add all pages identified by SEOmoz as duplicates and run a simple comparison in excel. This will identify any pages that do not match. You will easily see whether the new pages in the report can be ignored.

There is already a feature request in the works with the SEOmoz engineering team which will enable us to "turn off" pages that can be ignored (like those that are already noindexed). In the meantime, keeping track of the pages you can ignore is probably the best option.

You can keep track of progress by following updates on the Feature Request here.

Hope that helps,

Sha