open source deduplication

Dedupe is an open source python library that quickly de-duplicates or links large sets of data. And you don’t need a powerful server to use it - we built it to run on a modern day laptop. https://github.com/datamade/dedupe

English