does redis need twice the memory I actually think it does? I believe
it spawns another process to save to disk, does that mean it actually
copies the memory and I should always assume if I have 16 gigs of ram
8 is the max for a single redis process?
It use copy-on-write <http://en.wikipedia.org/wiki/Copy-on-write>, so kernel will takes care of using minimal RSS for changed pages during fork. You may want to tweak vm.overcommit_memory, if your dataset close to physical memory. I don't have such Redis setup, so at least theoretically.
If you can not overcommit_memory for some reason, but still keep as large as memory Redis setup. One choice is using RDB, disable BG_SAVE, and dump database manually via SAVE. But it's will blocking any read / write and may take minutes if you have 16G data. Another way is using AOF, and use multiple redis (to use the multiple cores), then schedule log rewrite yourself, then you can make sure only one redis are copy-on-write fork itself at the same time.
> does redis need twice the memory I actually think it does? I believe > it spawns another process to save to disk, does that mean it actually > copies the memory and I should always assume if I have 16 gigs of ram > 8 is the max for a single redis process?
> -- > You received this message because you are subscribed to the Google Groups > "Redis DB" group. > To post to this group, send email to redis-db@googlegroups.com. > To unsubscribe from this group, send email to > redis-db+unsubscribe@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om> > . > For more options, visit this group at > http://groups.google.com/group/redis-db?hl=en.
Even then, the forked save process consumes some additional RAM,
though not as much as 100% of the parent. I've found with a Redis
database of about 10GB, saves can take up to 2GB extra RAM, but I only
found that by trial and error and imagine it's very dependent on the
usage profile.
On Oct 3, 8:42 am, Favo Yang <favoy...@gmail.com> wrote:
> It use copy-on-write <http://en.wikipedia.org/wiki/Copy-on-write>, so kernel
> will takes care of using minimal RSS for changed pages during fork. You may
> want to tweak vm.overcommit_memory, if your dataset close to physical
> memory. I don't have such Redis setup, so at least theoretically.
> If you can not overcommit_memory for some reason, but still keep as large as
> memory Redis setup. One choice is using RDB, disable BG_SAVE, and dump
> database manually via SAVE. But it's will blocking any read / write and may
> take minutes if you have 16G data. Another way is using AOF, and use
> multiple redis (to use the multiple cores), then schedule log rewrite
> yourself, then you can make sure only one redis are copy-on-write fork
> itself at the same time.
> On Sun, Oct 3, 2010 at 11:54 AM, BeagleGuy <jimi...@gmail.com> wrote:
> > based on this screen shot:http://www.dropmocks.com/mBvx1
> > does redis need twice the memory I actually think it does? I believe
> > it spawns another process to save to disk, does that mean it actually
> > copies the memory and I should always assume if I have 16 gigs of ram
> > 8 is the max for a single redis process?
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Redis DB" group.
> > To post to this group, send email to redis-db@googlegroups.com.
> > To unsubscribe from this group, send email to
> > redis-db+unsubscribe@googlegroups.com<redis-db%2Bunsubscr...@googlegroups.c om>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/redis-db?hl=en.
• if you have a lot of updates, and they span a big set of pages this will affect memory usage. • Also virtualized hardware like EC2 can add some noise in this process.
On Oct 3, 2010 12:01 AM, "BeagleGuy" <jimi...@gmail.com> wrote:
does redis need twice the memory I actually think it does? I believe it spawns another process to save to disk, does that mean it actually copies the memory and I should always assume if I have 16 gigs of ram 8 is the max for a single redis process?
-- You received this message because you are subscribed to the Google Groups "Redis DB" group. To post to this group, send email to redis-db@googlegroups.com. To unsubscribe from this group, send email to redis-db+unsubscribe@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om> . For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
> does redis need twice the memory I actually think it does? I believe > it spawns another process to save to disk, does that mean it actually > copies the memory and I should always assume if I have 16 gigs of ram > 8 is the max for a single redis process?
If your system supports copy-on-write (COW) virtual memory, no. It will only duplicate pages of the save-to-disk process that are written to by the main redis-server process.
You screenshot shows a very small SHR (memory shared with other processes), but I don't know if that column counts COW, or only shared libraries. I believe it is the latter. I don't of a way to see how much of a process space is being shared via COW though.
