Understanding hash-max-zipmap-xxx values

[Ethan Collins]

I need a few clarification regarding hash-max-zipmap-[entries|values]
params. I was looking for some high level documentation on internal
data structures but didn't find one.

Let's say I have 2 fields in a hash where 1 field is max 3 bytes and
another field is approx 40 bytes, how should I define hash-max-zipmap-
value, 40 or (40 + 3) / 2? Reading the documentation, I feel that it
should be the former, that's 40 but given this example where one field
is substantially larger than the other, I am curious to know if hash
will waste space for this kind of data design.

Let's say my data looks something like:
stuff:<stuff_id> ->
item_list -> 'item1:item2:item3' [typical size: 40
bytes. Max size: ~950 bytes]
count -> 3

For the above, suppose I make my redis configuration as hash-max-
zipmap-entries 2 and hash-max-zipmap-value 950. For a configuration
like this, does redis allocate predefined chunks of 1190 bytes for
each hash, or does it store based on the actual size?

Thanks,
Ethan.

[Pieter Noordhuis]

Hi Ethan,

Your hunch is correct, that is: hash-max-zipmap-value is the length of
the largest value in a hash for the entire hash to be encoded as a
zipmap. Whenever a value is changed, or added to the hash, and exceeds
this length, the internal representation of the hash is changed to use
a hash table instead of a zipmap. The hash-max-zipmap-entries is
similar in that aspect, only with respect to the number of field/value
pairs stored in the hash.

These limits are only thresholds. Redis does not use them to
pre-allocate memory. If you have hash-max-zipmap-value set to 950 and
store a value of length 1, that will only consume the amount of memory
necessary to represent it, not 950 bytes. Hashes represented as zipmap
are always stored as compact as possible.

Cheers,
Pieter

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Ethan Collins]

Thanks. Now, when redis internally converts the zipmap to hash table
when we break the value/entry configuration, is there any statistics
kept which I can refer to identify the situation?

In the case of a hash table, do you allocate 1190 byte buckets then
(for my above config) ?

Thanks,
Ethan.

[Josiah Carlson]

Redis only ever stores based on actual size (modulo whatever the
malloc library over-allocates, modulo hash over-allocation with the
expectation of growth, etc.). To do anything else would be terribly
silly.

Regards,
- Josiah

[Ethan Collins]

Thanks. I have another query: Does redis try to involve compression
for a hash value especially when it finds that hash-max-zipmap-value
is high? (like 950) I am guessing not, but remember some earlier talks
on employing compression.

Thanks,
Ethan.

[Josiah Carlson]

Compression happens when it writes data to disk during a sync. It
keeps the values themselves in memory uncompressed so that when
someone requests the value, it can be simply copied or referenced as
part of the socket output buffering procedure.

- Josiah

[Josh]

These limits are only thresholds. Redis does not use them to pre-allocate memory... Hashes represented as zipmap are always stored as compact as possible.

Pieter, so I'm tempted to just set both hash-max-zipmap-value and hash-max-zipmap-entries to huge numbers so their effect as "limits" never comes into play. So what are the maximum values we could set, and what are the downsides of setting high values?  

[hva...@gmail.com]

Some time back, Instagram posted about tuning ziplists (the name of the parameters have changed) for a new function/feature they were creating.  They blogged about it on this page.

Their observation was:

Hashes in Redis are dictionaries that are can be encoded in memory very efficiently; the Redis setting ‘hash-zipmap-max-entries’ configures the maximum number of entries a hash can have while still being encoded efficiently. We found this setting was best around 1000; any higher and the HSET commands would cause noticeable CPU activity.

So they found the penalty of hash-max-ziplist-entries being too large was increased cpu consumption.

However, as you'll see by reading the rest of the blog post, they were sensitive to cpu resources because they wanted to run on an AWS server type that was fairly small in RAM.  AWS tends to scale both RAM and cpu together, so a server with less RAM also has less cpu.  They were also solving a rather specific problem (storing a number of integer values) that's likely different from your workload.  The version of Redis you're using is also different from theirs (in Oct 2011 the current version was 2.4.x), and that can also give you different results than theirs.

[Joshoooaaah]

Thanks for that!  Yes, I'd read that great Instagram article already.  1000 just seemed a bit arbitrary and I was seeking to understand WHY we should impose any limit at all.  Since asking the question here, I found this excellent resource and from that was able to answer this question which was the essence of mine.  

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/9qM9iSeRAA4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+unsubscribe@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.

Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.