Moz Campaign Duplicate Content Problem

[bjs2010]

I am chasing my tail on the duplicate content report on Moz.

My campaign never crawls more than 10000 pages - but yet, as time goes on, and just when I think I've got on top of duplicate content by removing it or disabling the pages, it peaks again and shows me other pages that have duplicate content issue.

These pages have remained the same during the last few weeks, and have been crawled so I dont understand why Rogerbot has not detected them before, "all in one go" - I seem to get dupe content appearing all the time with no changes to the website itself - its just Moz has not reported it till now.

Very frustrating. Help!

[G2W]

What cms or e-commerce platform are you using (I assume it's not a static site by the fact that you have over 10,000 pages)? You may want to look into some sort of canonical plugin; I've had a similar problem with Magento and after we installed the plugin duplicate content in the issues report took a drastic plunge.

[bjs2010]

Using Magento myself.

Our site has around 8500 crawlable pages - no more - Ive confirmed the same with Screaming Frog Spider.

So I think Rogerbot is crawling the whole site every time - but not recognising the dupe content all in one go.

Any other suggestions?

[G2W]

Sometimes it appears that Moz picks up on-site search queries as individual pages. That may be the case, and it would explain why new duplicate content flags keep popping up after the initial scan.

The canonical plugin I mentioned may take care of the issue if that's what it is.

http://yoast.com/tools/magento/canonical/

Worked for me, anyway.

Can you post a sample url that was flagged as a duplicate?

[bjs2010]

Thanks.

The URL's that Moz is reporting are clean and well formatted - No querystrings or anything, so they are reporting the clean URL as I want it to be indexed by Google. That's ok.

But there is Dupe on page content - text that has been copied and pasted, so it's a real issue - and its right in picking it up.

What I cant comprehend is why it's not reporting all of these at once, especially as its not reaching the crawl limit ever...

??

[ChiarynMiranda]

Hi There,

I responded to the ticket you submitted to the Help Team, but I am going to post the answer here, as well.

There are a few reasons why Roger might find different pages during different crawls. The way Roger crawls is by going to the homepage of the campaign and grabbing up to about 400 links on the same domain (or all of the links if there are less than 400) from the source code of that page and then we crawl through to those pages to diagnose them and to grab more links. The process continues on like that until we reach the campaign limit or until we no longer find any new pages. If there have been any changes to the links on a page or if we get any different server responses from any of the pages, it can change the architecture of the crawl, potentially leading the crawler to new pages. It is difficult to say more specifically why the crawler wouldn't find every page of the most recent crawl on the first crawl, since we don't keep the full crawl data after a new crawl is generated so I can't see the changes from crawl to crawl directly.

if you have an old crawl report CSV where the current duplicate content pages are not shown as duplicate content, you can email that to me in the support request and I will look into the issue further for you.

Chiaryn
Help Team Ninja