rdb -c memory for AOF file

[Alexander Gladysh]

Hi, list!

I have an .aof file from 2.4.14, and I would like to generate a memory
report for that file. I.e what keys consume memory.

There is redis-rdb-tools[1] project that can do that for RDB files. Is
there something that can do it for AOFs?

(I know that I can convert aof to rdb by loading it to Redis. It is
rather cumbersome for larger DBs.)

Thanks,
Alexander.

[1] — https://github.com/sripathikrishnan/redis-rdb-tools

[Salvatore Sanfilippo]

Hello Alexander,

such a tool is almost impossible to build because AOF can be any
combination of Redis commands, the only way to go from AOF to the
memory representation of keys and values is to load the file into
Redis, otherwise such a tool should basically simulate much of the
Redis internals in order to work.

If you have your AOF instance running "live", you can use redis-cli
--big-keys in order to have a clue about your large keys without any
need to perform a conversion.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>



--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Salvatore Sanfilippo]

Sorry it's --bigkeys not --big-keys.

[Alexander Gladysh]

Hello, Salvatore,

Thank you for your help!

On Sun, Oct 7, 2012 at 5:23 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> Hello Alexander,
>
> such a tool is almost impossible to build because AOF can be any
> combination of Redis commands, the only way to go from AOF to the
> memory representation of keys and values is to load the file into
> Redis, otherwise such a tool should basically simulate much of the
> Redis internals in order to work.

Makes sense.

Then the next question: can I write a simple script that uses Redis to
convert AOF to RDB? As I see it now, I have to feed .aof to a new
redis-server process, wait until it loads, connect to it, and issue a
BGSAVE. I, of course, will do that manually, but I would like
something that is more convenient to automate.

> If you have your AOF instance running "live", you can use redis-cli
> --big-keys in order to have a clue about your large keys without any
> need to perform a conversion.

Yes, I have one.

Please help me interpret what I see (or just point me to the docs):

$ redis-cli --bigkeys

# Press ctrl+c when you have had enough of it... :)
# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys
# in order to reduce server load (usually not needed).

[string] zzz:live_cluster_count | biggest so far with size 1
^C
$ redis-cli -n 2 --bigkeys

# Press ctrl+c when you have had enough of it... :)
# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys
# in order to reduce server load (usually not needed).

[string] zzz:chain:49 | biggest so far with size 6157
[string] zzz:chain:1 | biggest so far with size 22163
[ set] zzz:partners | biggest so far with size 16
[ hash] zzz:chainsv | biggest so far with size 10159
[ set] zzz:dates | biggest so far with size 45
[string] zzz:chains | biggest so far with size 209720
[ set] zzz:categories | biggest so far with size 386
^C
$ redis-cli -n 3 --bigkeys

# Press ctrl+c when you have had enough of it... :)
# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys
# in order to reduce server load (usually not needed).

[ hash] zzz:category:9:1346616000:uniq | biggest so far with size 1
[ hash] zzz:category:(?):1349035200:all | biggest so far with size 7
[ hash] zzz:stat:6:1347739200:uniq | biggest so far with size 17
[string] zzz:cps:1:1347393600:lb | biggest so far with size 10
[ hash] zzz:cps:6:1347652800:all | biggest so far with size 54928
[string] zzz:stats:3:1347220800:lb | biggest so far with size 5410
[string] zzz:stats:3:1347480000:lb | biggest so far with size 5438
[string] zzz:cps:2:1347307200:lb | biggest so far with size 357312
[string] zzz:cps:6:1348603200:lb | biggest so far with size 5150046
[ hash] zzz:cps:6:1348689600:all | biggest so far with size 79787
[string] zzz:cps:6:1349121600:lb | biggest so far with size 5978019
[ hash] zzz:cps:6:1349294400:all | biggest so far with size 82397
[string] zzz:cps:6:1349208000:lb | biggest so far with size 6273410
[ hash] zzz:cps:6:1349121600:all | biggest so far with size 83285
[ hash] zzz:cps:6:1349208000:all | biggest so far with size 87557
^C
$ redis-cli -n 4 --bigkeys

# Press ctrl+c when you have had enough of it... :)
# You can use -i 0.1 to sleep 0.1 sec every 100 sampled keys
# in order to reduce server load (usually not needed).

[ hash] services:info:zzz1871 | biggest so far with size 14
[string] zzz:payload_global_counter:417 | biggest so far with size 5
[string] zzz:payload_global_counter:383 | biggest so far with size 6
[ set] services:running | biggest so far with size 33
^C

* * *

My problem:

I have this in info:

db0:keys=1,expires=0
db2:keys=61,expires=0
db3:keys=6323,expires=0
db4:keys=52,expires=6

...and .aof file, after rewrite, weights 2GB.

I would like to figure out what takes so much space and optimize that away.

Thanks again,
Alexander.

[Salvatore Sanfilippo]

On Sun, Oct 7, 2012 at 3:40 PM, Alexander Gladysh <agla...@gmail.com> wrote:
> Hello, Salvatore,
>
> Thank you for your help!

You are welcome!

> Then the next question: can I write a simple script that uses Redis to
> convert AOF to RDB? As I see it now, I have to feed .aof to a new
> redis-server process, wait until it loads, connect to it, and issue a
> BGSAVE. I, of course, will do that manually, but I would like
> something that is more convenient to automate.

In Redis the two persistence methods can be enabled / used at the same
time without issues.
Basically any time you have a server configured with AOF persistence,
you can still just call "BGSAVE" to generate an RDB file.

> Please help me interpret what I see (or just point me to the docs):

Basically redis-cli scans keys at random, and shows you the biggest
key found so far, of any given type.
It is useful if you have your "leak" in a single huge key, or just a
few. If your memory is used because of suboptimal usage among many
keys it is not so useful.

But this looks interesting:

> [ hash] zzz:category:9:1346616000:uniq | biggest so far with size 1
> [ hash] zzz:category:(?):1349035200:all | biggest so far with size 7
> [ hash] zzz:stat:6:1347739200:uniq | biggest so far with size 17
> [string] zzz:cps:1:1347393600:lb | biggest so far with size 10
> [ hash] zzz:cps:6:1347652800:all | biggest so far with size 54928
> [string] zzz:stats:3:1347220800:lb | biggest so far with size 5410
> [string] zzz:stats:3:1347480000:lb | biggest so far with size 5438
> [string] zzz:cps:2:1347307200:lb | biggest so far with size 357312
> [string] zzz:cps:6:1348603200:lb | biggest so far with size 5150046
> [ hash] zzz:cps:6:1348689600:all | biggest so far with size 79787
> [string] zzz:cps:6:1349121600:lb | biggest so far with size 5978019
> [ hash] zzz:cps:6:1349294400:all | biggest so far with size 82397
> [string] zzz:cps:6:1349208000:lb | biggest so far with size 6273410
> [ hash] zzz:cps:6:1349121600:all | biggest so far with size 83285
> [ hash] zzz:cps:6:1349208000:all | biggest so far with size 87557
> ^C

You have many hashes with a decent number of elements apparently.

The size is actually "number of elements" for aggregate data types. So
many hashes have 10k elements, like zzz:cps:6:1349208000:all

This is probably your problem.

Cheers,
Salvatore

[Alexander Gladysh]

On Sun, Oct 7, 2012 at 5:47 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> On Sun, Oct 7, 2012 at 3:40 PM, Alexander Gladysh <agla...@gmail.com> wrote:
>> Then the next question: can I write a simple script that uses Redis to
>> convert AOF to RDB? As I see it now, I have to feed .aof to a new
>> redis-server process, wait until it loads, connect to it, and issue a
>> BGSAVE. I, of course, will do that manually, but I would like
>> something that is more convenient to automate.
>
> In Redis the two persistence methods can be enabled / used at the same
> time without issues.
> Basically any time you have a server configured with AOF persistence,
> you can still just call "BGSAVE" to generate an RDB file.

I know. But that is risky and slow — there is not much memory left on
that machine (that is the reason why I am digging into this).

So, I have .aof that I downloaded, and will analyze locally.

>> Please help me interpret what I see (or just point me to the docs):
>
> Basically redis-cli scans keys at random, and shows you the biggest
> key found so far, of any given type.

What confuses me is this:

>> [string] zzz:cps:6:1348603200:lb | biggest so far with size 5150046
>> [ hash] zzz:cps:6:1348689600:all | biggest so far with size 79787

I assumed that the size is in bytes. 79787 is clearly smaller than
5150046, so whole thing just looks weird.

(I read below and realized that this I'm wrong.)

Can the output be clarified in this matter? I.e. maybe write units
explicitly (and, if possible, write size in bytes as well).

> It is useful if you have your "leak" in a single huge key, or just a
> few. If your memory is used because of suboptimal usage among many
> keys it is not so useful.

Well, AOF size almost matches memory usage, so, I think, this is not the case.

> But this looks interesting:
>
>> [ hash] zzz:category:9:1346616000:uniq | biggest so far with size 1
>> [ hash] zzz:category:(?):1349035200:all | biggest so far with size 7
>> [ hash] zzz:stat:6:1347739200:uniq | biggest so far with size 17
>> [string] zzz:cps:1:1347393600:lb | biggest so far with size 10
>> [ hash] zzz:cps:6:1347652800:all | biggest so far with size 54928
>> [string] zzz:stats:3:1347220800:lb | biggest so far with size 5410
>> [string] zzz:stats:3:1347480000:lb | biggest so far with size 5438
>> [string] zzz:cps:2:1347307200:lb | biggest so far with size 357312
>> [string] zzz:cps:6:1348603200:lb | biggest so far with size 5150046
>> [ hash] zzz:cps:6:1348689600:all | biggest so far with size 79787
>> [string] zzz:cps:6:1349121600:lb | biggest so far with size 5978019
>> [ hash] zzz:cps:6:1349294400:all | biggest so far with size 82397
>> [string] zzz:cps:6:1349208000:lb | biggest so far with size 6273410
>> [ hash] zzz:cps:6:1349121600:all | biggest so far with size 83285
>> [ hash] zzz:cps:6:1349208000:all | biggest so far with size 87557
>> ^C
>
> You have many hashes with a decent number of elements apparently.
>
> The size is actually "number of elements" for aggregate data types. So
> many hashes have 10k elements, like zzz:cps:6:1349208000:all
>
> This is probably your problem.

Looks like that. Will investigate that.

Thank you!
Alexander.

[Alexander Gladysh]

On Sun, Oct 7, 2012 at 5:40 PM, Alexander Gladysh <agla...@gmail.com> wrote:

> [ hash] zzz:chainsv | biggest so far with size 10159

It is this guy. Several GBs of strings here.

Thanks again,
Alexander.

[Salvatore Sanfilippo]

On Sun, Oct 7, 2012 at 3:54 PM, Alexander Gladysh <agla...@gmail.com> wrote:

> I know. But that is risky and slow — there is not much memory left on
> that machine (that is the reason why I am digging into this).

If you have AOF on, you are probably rewriting the AOF file. BGSAVE is
identical, it forks and so forth, so if the rewrite process is working
well, BGSAVE will also work well. However you have the alternative of
just using SAVE: it will block but not additional memory will be used
at least...

> What confuses me is this:
>
>>> [string] zzz:cps:6:1348603200:lb | biggest so far with size 5150046
>>> [ hash] zzz:cps:6:1348689600:all | biggest so far with size 79787

> I assumed that the size is in bytes. 79787 is clearly smaller than
> 5150046, so whole thing just looks weird.

The size is in number of elements for aggregate types, in bytes for
strings, but it searches the biggest key independently for every data
type, so the hash is smaller than the string, but was the biggest hash
found so far.

> (I read below and realized that this I'm wrong.)
>
> Can the output be clarified in this matter? I.e. maybe write units
> explicitly (and, if possible, write size in bytes as well).

Yes this is a good idea.

[Alexander Gladysh]

On Sun, Oct 7, 2012 at 8:07 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> On Sun, Oct 7, 2012 at 3:54 PM, Alexander Gladysh <agla...@gmail.com> wrote:
>
>> I know. But that is risky and slow — there is not much memory left on
>> that machine (that is the reason why I am digging into this).
>
> If you have AOF on, you are probably rewriting the AOF file. BGSAVE is
> identical, it forks and so forth, so if the rewrite process is working
> well, BGSAVE will also work well.

It is not working very well, sadly. It swaps and is quite slow (as
expected with so little physical memory left). Need to fix monitoring
to catch such things earlier.

But, fortunately, that chainsv key is mostly disposable (need to
switch from hash to keys and add EXPIRE). So I'll just dump it
somewhere and DEL it.

> However you have the alternative of
> just using SAVE: it will block but not additional memory will be used
> at least...

Did not know that, thanks!

Alexander.