API for testing duplicate content

[Sebes]

Does anyone know a service or API or php lib to compare two (or more) pages and to return their similiarity (Level-3-Shingles).

API would be greatly prefered.

[EricaMcGillivray]

While I don't know of an API that does that, you can set up your site using the SEOmoz tools and our Crawl Diagnostics section does look for Duplicate Cotent.

[Sebes]

Hey Erica,

thanks for your answer. What I need is a way to decide on-the-fly whether two pages are similar or not. If they are too similar I need to depublish or at least rel canonical one of those.

Best solution would be an API that takes 2 pages, but it seems as if I have to build it myself then.

Thanks for your efforts.