redis memory usage

[vij]

i have a redis installation running in production. The dataset size is
currently 20G. As it is growing fast , i did some profiling in redis
data. Total there aree 20lack keys and using a script i fetched all
the keys and run debug command on each keys to get the approx memory
for each key. i have identified data worth 9G , no idea how the
remaining memory being used. I am using only hashes, no sets. Can you
please advice on how to identify the remaining data? Is there any
better profiling tool for redis which can give exact usage on each
keys?

-Vij

[Josiah Carlson]

Are you basing your data size information on both the data that you
see, as well as the data structure overhead? Are you accounting for
memory fragmentation and over-allocation for each block of data
requested by the allocator?

I would actually recommend going about it in the opposite direction.
Start a new Redis on an identical system (same architecture, same
version of Redis, same configuration file), create some hashes with
sizes on the order of magnitude of hashes you are actually seeing.
1. Start with an empty Redis, check memory use reported by INFO with
resident size.
2. Add your hash using 4 byte unique keys and values.
3. Re-check memory use reported by INFO with resident size.
4. Do this for hash sizes increasing by sqrt(2) until you are close to
the maximum size of your hash in production.

Because you know how much space each hash will take of a given size,
and you know the overhead of your actual data (4 byte key, 4 byte
value), you can extrapolate Redis' overhead for data structures,
memory over-allocation, and fragmentation. Re-calculate your expected
size in production, and your numbers should line up pretty well.

With the version of Redis I am using, I know that Redis uses roughly
80 bytes + my data size to store an entry in a hash. If I have a hash
of 1 million entries (a small hash for us), I know it's going to be 80
megs + whatever data I put in. Redis has improved memory use since we
deployed, so I would suggest you run the tests.

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Ethan Collins]

Thanks for mentioning about the 80b of extra overhead. I ran some
tests and found that the field+value size is directly proportional to
their size. This is because of the zipmap format. However, I don't
know how the hash 'key's are stored. I don't find memory usage linear
to their size. Do you know how hash keys are stored?

Regards,
Ethan.

[Didier Spezia]

If I'm not mistaken, for each hash entry, on a 64 bits platform,
you have a dictEntry object (size=24), containing pointers to:

- a robj (size=16) containing a pointer to the actual value
(size=9+length for strings)

- the key itself as created with sdsdup (size=9+length)

Now the allocator typically allocates more data than you need.
The minimum granularity for the glibc allocator is 32 bytes.
So if you have small keys and values, the minimum footprint
for a (string) entry in Redis is
32(dictEntry) + 32(robj) + value(32) + key(32) = 128

jemalloc and tcmalloc offer better granularity, so using them
can dramatically decrease the memory footprint if you have
plenty of small keys and values in non zipped objects.

Regards,
Didier.