Understanding observations of memory used for HLLs

[Hemanth Yamijala]

Hi,

I am trying to understand the memory used for HLLs. We are trying to implement HLLs for counts of sets which could be large in numbers 1M+, but low cardinality, maybe 1K-1.5K. I understand from some threads here and other articles online that Redis has a sparse and dense representation for HLLs and this could cause memory to switch from low usage to high (upto 12KB). So, while the max memory can be bound, the fact that the cardinality is low is something significant and hence I was trying to test what an average case might be. Please bear with me if I am not interpreting something right in the details below.

I am running the test on Mac OSX and using Activity Monitor to measure the results. The test uses Jedis and generates 'n' keys and adds 'm' random numeric IDs for each key (using Java's random.nextInt(1000000000)). PFADD is called for one key, one user at a time. Same redis connection is used for all the calls.

My observation for 10K keys and 1.5K users per key (and no other keys in the redis instance) is that the 'Memory' field of the redis-server in activity monitor grows up to about 32MB steadily as keys are added. Then, if I leave the server on for a long time without any usage, it steadily comes down to 148KB. The detail view for this process also shows similar behaviour for the fields Real & Private memory size. 

Then, when I run another test to query these keys randomly, the memory size goes up again.

Additional details:

* The keys are all using sparse encoding.

* The size of the RDB file is 24M. This sort of correlates with the value mentioned in the comments of http://download.redis.io/redis-stable/src/hyperloglog.c for the said cardinality.

I am trying to understand why the memory value goes down when not being used, because my understanding is that Redis will hold everything in memory always. So, I expected that the mem size used will remain steady if there are no deletions. 

Is this an artifact of something clever happening for HLLs, or something I am not understanding when reading the memory used values, or in general something about how Redis manages memory?

Would appreciate any insights.

Thanks

Hemanth

[Hemanth Yamijala]

Sorry for sending too soon. I am using Dynamic Redis (for geoadd commands) 

redis_version:2.8.17-dynamic-0.5.2

Redis info for memory is:

# Memory

used_memory:32666160

used_memory_human:31.15M

used_memory_rss:6144000

used_memory_peak:32666160

used_memory_peak_human:31.15M

used_memory_lua:35840

mem_fragmentation_ratio:0.19

mem_allocator:libc