Best approach for save and RAM size?
124 views
Skip to first unread message
PanMan
Jan 9, 2012, 8:01:20 AM1/9/12
to Redis DB
Hi!
I use Redis in data processing: We have a file of about 90 GB that we
need to process every week. Before starting I flush the whole DB, so
it starts clean. I have written a script which processes the file, and
increments counters in several sorted sets. After the file processing
is done (which takes a few hours), I get the data I want from Redis,
mostly by getting some ranges from the sorted sets.
The total amount of memory used by Redis is around 14GB. I have this
running on a machine which has 24 GB of memory, and is currently only
used for this purpose.
However, during the adding of the keys, the machine becomes really
slow, and is running out of memory (and swap). At that time there are
2 Redis processes, both using 14 GB, which doesn't fit in the 24GB.
I assume this is because Redis forks itself while saving.
I currently have this as my save config:
save 900 1
save 300 10
save 60 10000
Since these saves happen in the background, the memory size is
doubled, which causes my issue. Does the SAVE command have the same
memory usage? From the docs it seems it would happen in the same
process? http://redis.io/commands/save
I wouldn't mind if I would have to call SAVE manually, and wait for it
to finish, if that means the memory needed is less (since there is no
forking). I could just call it in my processing script, and wait to
save, once every 30 min or so.
Would that work? Any other pointers to keep the process within my
current machine limits?
I use Redis in data processing: We have a file of about 90 GB that we
need to process every week. Before starting I flush the whole DB, so
it starts clean. I have written a script which processes the file, and
increments counters in several sorted sets. After the file processing
is done (which takes a few hours), I get the data I want from Redis,
mostly by getting some ranges from the sorted sets.
The total amount of memory used by Redis is around 14GB. I have this
running on a machine which has 24 GB of memory, and is currently only
used for this purpose.
However, during the adding of the keys, the machine becomes really
slow, and is running out of memory (and swap). At that time there are
2 Redis processes, both using 14 GB, which doesn't fit in the 24GB.
I assume this is because Redis forks itself while saving.
I currently have this as my save config:
save 900 1
save 300 10
save 60 10000
Since these saves happen in the background, the memory size is
doubled, which causes my issue. Does the SAVE command have the same
memory usage? From the docs it seems it would happen in the same
process? http://redis.io/commands/save
I wouldn't mind if I would have to call SAVE manually, and wait for it
to finish, if that means the memory needed is less (since there is no
forking). I could just call it in my processing script, and wait to
save, once every 30 min or so.
Would that work? Any other pointers to keep the process within my
current machine limits?
Dvir Volk
Jan 9, 2012, 8:22:29 AM1/9/12
to redi...@googlegroups.com
that should work perfectly, I do it when I'm pressed with RAM and I'm not on an instance that actually server users.
also, not sure if it will affect it, but did you set vm.overcommit_memory = 1 ?
it's a must as well.
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Dvir Volk
System Architect, DoAT, http://doat.com
Didier Spezia
Jan 9, 2012, 8:29:37 AM1/9/12
to redi...@googlegroups.com
Hi,
this is actually a bit more complex than what you described.
When the BGSAVE is running, another process is forked, and the OS
will share the pages between the two Redis processes using a COW
mechanism. Only the pages which are updated during the dump
operation will be duplicated by the OS.
So even if you see two 14 GB processes, it does not mean 28 GB
of memory are actually required. If you update a lot of differerent
values during the dump, the memory consumption will be between
14 GB and 28 GB. The more different updated values, the higher
the memory consumption. Usually, only a limited number of pages
are modified during a dump.
On top of that, you also have the dump file itself. The dump is stored on
a filesystem which is also cached by the OS. When a large file
is written, the filesystem may push the OS to swap some memory
out (stealing it from the processes) to increase the cache. On some
OS (Linux for instance) there is no way to limit the size of the filesystem
cache.
If you use Redis for a batch, I would suggest to comment out the save lines
in the configuration file to deactivate background dumps, and use the SAVE
command to dump the file.
Regards,
Didier.
ivan babrou
Jan 9, 2012, 8:33:25 AM1/9/12
to redi...@googlegroups.com
Do you really need to save your data while filling your redis? It's better to process your damn-big-file, then BGSAVE.
It will not copy memory, you'll stay in your 14GB. Redis double memory usage because of changed keys. If the key is changed, then redis fork for saving data will duplicate it (2x memory).
How about that?
--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
Reply all
Reply to author
Forward
0 new messages