why kylin choose hyperloglog over bloom filter for estimation calculation

[聪樊]

Dear All

Recently, we are trying to solve distinct count problem(like UV, DAU). I am investigated HyperLogLog and Bloom filter. I am wondering why kyin use hll ? Have you guys do any comparison ?

Thanks,
Lucas

[Antonios Chalkiopoulos]

HyperLogLog is suited for cardinality ( count distinct elements in a set )

Bloom Filter is for membership checks !

So HLL can be like a 10 KByte file , but bloom filters can be hundred of MBytes ( they server different purposes )

[聪樊]

Thanks Antonios

But I think both HyperLogLog and Bloom Filter could be used for membership checks and cardinality estimation. But HyperLogLog has better performance. 

see: https://github.com/echen/streaming-simulations/wiki/Cardinality-Estimation:-Bloom-Filter-vs.-HyperLogLog for details.