I was wondering if there's a way to get back the memory usage of a given key. Keep in mind that the internal representation of a key can be very different from what the user gets back. So although there are a bunch of scripts that will sample the data and infer the memory usage of the various keys based on the values they get back, they cannot determine the actual memory used by the current encoding of the object.
For example, storing n number of small integers (to fit the size of the shared integers) doesn't really take any space to store the values, since all the integers are sharing a global pool of ints, so we only need to calculate the memory used to store the key.
Another reason to do that is to include the overhead in the memory usage. The overhead used to store the different types is different as well. Also strings that represent numbers that can fit in a 'long long' are converted internally, I'm repeating myself here, since this is part of the encoding.
So unless there's a way to do this in redis, what do you guys think about adding an extension to the 'OBJECT' command to return this information. I think that this will be useful. I'm willing to patch redis and send a pull request unless there were objections.
Thanks,
-- You received this message because you are subscribed to the Google Groups "Redis DB" group. To post to this group, send email to redis-db@googlegroups.com. To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
I think it could be useful information. I don't know how others feel, but I have a sense that augmenting the INFO command to take a key to return information about; more than just memory use, but also memory slop (size allocated by memory allocator, but not requested, or spare data pre-allocated by Redis for resize operations), and TTL, and any other information that may be useful.
... Of course if you can modify INFO to take a key as an argument, it may also make sense to write it to accept many keys as arguments.
On Wed, Mar 21, 2012 at 3:23 PM, John Shahid <jvsha...@gmail.com> wrote: > Hi everyone,
> I was wondering if there's a way to get back the memory usage of a > given key. Keep in mind that the internal representation of a key can > be very different from what the user gets back. So although there are > a bunch of scripts that will sample the data and infer the memory > usage of the various keys based on the values they get back, they > cannot determine the actual memory used by the current encoding of the > object.
> For example, storing n number of small integers (to fit the size of > the shared integers) doesn't really take any space to store the > values, since all the integers are sharing a global pool of ints, so > we only need to calculate the memory used to store the key.
> Another reason to do that is to include the overhead in the memory > usage. The overhead used to store the different types is different as > well. Also strings that represent numbers that can fit in a 'long > long' are converted internally, I'm repeating myself here, since this > is part of the encoding.
> So unless there's a way to do this in redis, what do you guys think > about adding an extension to the 'OBJECT' command to return this > information. I think that this will be useful. I'm willing to patch > redis and send a pull request unless there were objections.
> Thanks,
> -- > You received this message because you are subscribed to the Google Groups "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
-- You received this message because you are subscribed to the Google Groups "Redis DB" group. To post to this group, send email to redis-db@googlegroups.com. To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Its a great idea to display the memory usage, but I had a slightly different approach that I wanted to discuss.
Can we guesstimate the in-memory size of an object from its serialized length as its stored in the dump.rdb? If yes, we can parse the rdb and generate a tree like report of memory usage across all keys in all the databases. In other words, instead of saying "user:123 takes 100 bytes", we can say "user:* takes 500MB". We could do interesting things like "Show me 100 objects using maximum memory".
I think we can arrive at fairly representative numbers from the serialized length -
1. *Strings* : A string has two ints - len and free, and then a character buffer. So, to get the in-memory size of a string, the formula would be "number of bytes in string + 8 bytes for the 2 ints". Compressed strings in the rdb have the uncompressed length, so that's easy. And we'd have to do special handling for ints in the string - but that's not bad. 2. *Zipmaps, Ziplists, Intsets* - are serialized AS-IS in the rdb, so we can get the exact in-memory size 3. *Lists* - The list has two pointers - head and tail. Each list entry has two pointers, previous and next. Each entry also has a value, whose size is variable. So, if we encounter a linkedlist in the rdb, we can estimate its in-memory size as : in-memory-size = (2 * number of elements + 2) * sizeof pointer + total size of all strings in the list 4. *Sets, Dictionary* - These are a bit more tricky. I haven't investigated deeply, but I am guessing we can come up with a heuristic to arrive at in-memory size given the number of elements and the size of each key/value pair. 5. *Sorted Set* - If we have a heuristic for dictionary, sorted set would be easy because its largely a linked list and a dictionary.
Of course, the caveat is that the memory usage would be approximate, and could be off by a few bytes, and we would have to decide whether we are using 32 bit or 64 bit pointers, and maybe there are a few other things I am missing/
Given all this, do you think such a tool would be useful?
--Sri
On 22 March 2012 21:20, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> I think it could be useful information. I don't know how others feel, > but I have a sense that augmenting the INFO command to take a key to > return information about; more than just memory use, but also memory > slop (size allocated by memory allocator, but not requested, or spare > data pre-allocated by Redis for resize operations), and TTL, and any > other information that may be useful.
> ... Of course if you can modify INFO to take a key as an argument, it > may also make sense to write it to accept many keys as arguments.
> Regards, > - Josiah
> On Wed, Mar 21, 2012 at 3:23 PM, John Shahid <jvsha...@gmail.com> wrote: > > Hi everyone,
> > I was wondering if there's a way to get back the memory usage of a > > given key. Keep in mind that the internal representation of a key can > > be very different from what the user gets back. So although there are > > a bunch of scripts that will sample the data and infer the memory > > usage of the various keys based on the values they get back, they > > cannot determine the actual memory used by the current encoding of the > > object.
> > For example, storing n number of small integers (to fit the size of > > the shared integers) doesn't really take any space to store the > > values, since all the integers are sharing a global pool of ints, so > > we only need to calculate the memory used to store the key.
> > Another reason to do that is to include the overhead in the memory > > usage. The overhead used to store the different types is different as > > well. Also strings that represent numbers that can fit in a 'long > > long' are converted internally, I'm repeating myself here, since this > > is part of the encoding.
> > So unless there's a way to do this in redis, what do you guys think > > about adding an extension to the 'OBJECT' command to return this > > information. I think that this will be useful. I'm willing to patch > > redis and send a pull request unless there were objections.
> > Thanks,
> > -- > > You received this message because you are subscribed to the Google > Groups "Redis DB" group. > > To post to this group, send email to redis-db@googlegroups.com. > > To unsubscribe from this group, send email to > redis-db+unsubscribe@googlegroups.com. > > For more options, visit this group at > http://groups.google.com/group/redis-db?hl=en.
> -- > You received this message because you are subscribed to the Google Groups > "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to > redis-db+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/redis-db?hl=en.
-- You received this message because you are subscribed to the Google Groups "Redis DB" group. To post to this group, send email to redis-db@googlegroups.com. To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
I hacked together a script to generate a memory report across all keys by parsing redis dump.rdb file. The output is a csv file with the columns - database number, data type, key, memory in bytes, encoding.
The memory usage is based on some reverse calculations, and are therefore approximate. In general, the actual memory used will be slightly higher than what is reported.
--Sri
On 22 March 2012 23:42, Sripathi Krishnan <sripathi.krish...@gmail.com>wrote:
> Its a great idea to display the memory usage, but I had a slightly > different approach that I wanted to discuss.
> Can we guesstimate the in-memory size of an object from its serialized > length as its stored in the dump.rdb? If yes, we can parse the rdb and > generate a tree like report of memory usage across all keys in all the > databases. In other words, instead of saying "user:123 takes 100 bytes", we > can say "user:* takes 500MB". We could do interesting things like "Show me > 100 objects using maximum memory".
> I think we can arrive at fairly representative numbers from the serialized > length -
> 1. *Strings* : A string has two ints - len and free, and then a > character buffer. So, to get the in-memory size of a string, the formula > would be "number of bytes in string + 8 bytes for the 2 ints". Compressed > strings in the rdb have the uncompressed length, so that's easy. And we'd > have to do special handling for ints in the string - but that's not bad. > 2. *Zipmaps, Ziplists, Intsets* - are serialized AS-IS in the rdb, so > we can get the exact in-memory size > 3. *Lists* - The list has two pointers - head and tail. Each list > entry has two pointers, previous and next. Each entry also has a value, > whose size is variable. So, if we encounter a linkedlist in the rdb, we can > estimate its in-memory size as : > in-memory-size = (2 * number of elements + 2) * sizeof pointer + > total size of all strings in the list > 4. *Sets, Dictionary* - These are a bit more tricky. I haven't > investigated deeply, but I am guessing we can come up with a heuristic to > arrive at in-memory size given the number of elements and the size of each > key/value pair. > 5. *Sorted Set* - If we have a heuristic for dictionary, sorted set > would be easy because its largely a linked list and a dictionary.
> Of course, the caveat is that the memory usage would be approximate, and > could be off by a few bytes, and we would have to decide whether we are > using 32 bit or 64 bit pointers, and maybe there are a few other things I > am missing/
> Given all this, do you think such a tool would be useful?
> --Sri
> On 22 March 2012 21:20, Josiah Carlson <josiah.carl...@gmail.com> wrote:
>> I think it could be useful information. I don't know how others feel, >> but I have a sense that augmenting the INFO command to take a key to >> return information about; more than just memory use, but also memory >> slop (size allocated by memory allocator, but not requested, or spare >> data pre-allocated by Redis for resize operations), and TTL, and any >> other information that may be useful.
>> ... Of course if you can modify INFO to take a key as an argument, it >> may also make sense to write it to accept many keys as arguments.
>> Regards, >> - Josiah
>> On Wed, Mar 21, 2012 at 3:23 PM, John Shahid <jvsha...@gmail.com> wrote: >> > Hi everyone,
>> > I was wondering if there's a way to get back the memory usage of a >> > given key. Keep in mind that the internal representation of a key can >> > be very different from what the user gets back. So although there are >> > a bunch of scripts that will sample the data and infer the memory >> > usage of the various keys based on the values they get back, they >> > cannot determine the actual memory used by the current encoding of the >> > object.
>> > For example, storing n number of small integers (to fit the size of >> > the shared integers) doesn't really take any space to store the >> > values, since all the integers are sharing a global pool of ints, so >> > we only need to calculate the memory used to store the key.
>> > Another reason to do that is to include the overhead in the memory >> > usage. The overhead used to store the different types is different as >> > well. Also strings that represent numbers that can fit in a 'long >> > long' are converted internally, I'm repeating myself here, since this >> > is part of the encoding.
>> > So unless there's a way to do this in redis, what do you guys think >> > about adding an extension to the 'OBJECT' command to return this >> > information. I think that this will be useful. I'm willing to patch >> > redis and send a pull request unless there were objections.
>> > Thanks,
>> > -- >> > You received this message because you are subscribed to the Google >> Groups "Redis DB" group. >> > To post to this group, send email to redis-db@googlegroups.com. >> > To unsubscribe from this group, send email to >> redis-db+unsubscribe@googlegroups.com. >> > For more options, visit this group at >> http://groups.google.com/group/redis-db?hl=en.
>> -- >> You received this message because you are subscribed to the Google Groups >> "Redis DB" group. >> To post to this group, send email to redis-db@googlegroups.com. >> To unsubscribe from this group, send email to >> redis-db+unsubscribe@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/redis-db?hl=en.
-- You received this message because you are subscribed to the Google Groups "Redis DB" group. To post to this group, send email to redis-db@googlegroups.com. To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com. For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
