Searching for duplicate internal content

[GrouchyKids]

As per this article http://www.seomoz.org/blog/fat-pandas-and-thin-content

Does anyone know of a way to search a site for internal duplicate content?

Justin

[RobertFisher]

Justin,

I haven't tried this with large sites: 5,000 pages, etc. but with smaller sites I use the site:examplesite.com and then a space with the text string I have concern about. This actually works quite well if you know generally what content you have a concern over. So, site:examplesite.com Four score and seven years ago. Put that in the search box and pages with that content will be displayed first or close to first.

If you are wanting something to search the entire site and then tell you what may be duped, you are likely going to have to go to a paid tool. Copyscape is the one I hear the most about and since I put it here I went and bought the premium. It is a nickel per search and you buy $10 worth of credits to begin. That gives you 200 searches to use over 12 months. I will run a search shortly and respond with a first result opinion.

Hope this helps

[RobertFisher]

Justin,

I just tried copyscape using an article I wrote that went a bit viral, regarding a medicine that was under severe scrutiny by the FDA. While we had two different sites at the time and it posted as a blog, it shows on both. Also, it showed (slightly altered...hmmmmm) on several others not in the least associated with our firm or our clients. So, on first blush it seems a reasonable tool for the cost.

best of luck, hope this helps.

[overtake]

Justin,

Simplest way do a site search in Google eg:

site:www.seomoz.org.

Navigate to the last page of results - you'll want to change the results to 100 in the search settings if you have a large-ish site.

Underneath the last result it will say something like:

In order to show you the most relevant results, we have omitted some entries very similar to the X already displayed.

If you like, you can repeat the search with the omitted results included.

Press that link.  The difference between your standard results and what this shows is most likely duplicate content. Then you just need to find those pages   Out of interest does anyone know a tool that scrapes these results and provides a csv of those links,  would be handy rather than having to manually check which is difficult at the best of times.

If you have query strings or duplicate links your pretty certain to have duplicate content of some kind particularly in my experience if you have an e-commerce site and you have various categories that tend to cover the same type of products.

[RobertFisher]

Overtake,

just saw this and was happy to see you put the note re repeating the search with omitted results. Excellent.

[GrouchyKids]

Thanks all...

So will Copyscape search your own site for internal dupes my understanding was that it would not?

I'm looking for a tool that identifies even small parts of a page with internal duplication, once identified I can make a judgement call as to what can / could be done in each instance.

Any ideas???

[RobertFisher]

Since it picks up any, it will pick up yours. If you are wanting something a bit more refined, I am not sure of any internal tool and would rely on the site: method.

[GrouchyKids]

Thanks Guys