Pls suggest to improve performance in keyvaluedb

[Venkatesh T]

Dear Team,

First of all thanks for this wonderful library. I used KeyValueDB to deduplicate 150 million records considering Name,address and country. I configured minRelevance as 0.5 and maxSearchHits as 100. It was very fast initially. After reached 10 millions, it becomes slow. It took nearly 1 hr for processing 0.1 million records. Please suggest to improve the performance.

[JP de Vooght]

Hi Venkatesh? Did you eventually solve this?

If I understand correctly blocking is possible only with IDs. I was wondering if locality-sensitive hashing would be possible by looking at several attributes for blocking. I am still getting familiar with Duke so perhaps I missed something...

JP