Diagnosing duplicate content issues

[jamesti]

We recently made some updates to our site, one of which involved launching a bunch of new pages. Shortly afterwards we saw a significant drop in organic traffic. Some of the new pages list similar content as previously existed on our site, but in different orders. So our question is, what's the best way to diagnose whether this was the cause of our ranking drop? My current thought is to block the new directories via robots.txt for a couple days and see if traffic improves. Is this a good approach? Any other suggestions?

[BTeubner]

Here's what I'd do.

First, check big stuff like:

• Did your URLs change? Is so, were redirects applied?
• Did your devs accidentally noindex or nofollow entire sections of the site? I've seen this happen a bunch.

If neither of the above, I'd next dig deep with analytics to try to figure out if there is a certain pages, type of pages (product page, articles, blog, etc), etc. that have taken the biggest hit.It may give you more direction. Could also:

• Review Webmaster tools
• Run a site scanner link ScreamingFrog and look for big errors

You could always try blocking new stuff but you never know how long it's gonna take google to come back and honor your changes.

[jamesti]

yeah, unfortunately we've checked all the items you listed and there's nothing obvious.  seems like blocking pages is the only option left..

[BTeubner]

Could always give it a whirl...

[Dr-Pete]

I hate to advocate full-scale blocking, but if you really took a hit, and you know the timeline coincided with the new content, it is possible. It might be better to scale back and re-roll out new content in chunks.

One warning - if this is a regular filter (you added a bunch of duplicates), Google should start re-ranking content as soon as the blocking kicks in (this may take weeks, not days). If this was Panda-related or more severe, though, it could take a month or more to see an impact. Not to be the bearer of bad news, but don't Robots.txt block the pages for 2 days, decide it didn't work, and unblock them.

A slightly less extreme approach would be to META NOINDEX all of the pages. That way, you could start to selectively lift the NOINDEX on content piece by piece. If you Robots.txt block all the new directories, it's going to be hard to re-introduce the content. You'll end up releasing the block all at once and potentially just having the same problem again.