Redis memory use when RDB

[Zerg ling]

When RDB,  child thread would be forked which double the memory ues.

Howere, linux use "copy-on-write" when "fork" .

Which one is right?

Or, in the worst condition ,  we need double of parent-thread's memory  for RDB with copy-on-write?

Sorry for bad English.

[Itamar Haber]

Hi,

Redis uses copy-on-write on Linux for `BGSAVE`/snapshotting, and the recommendation to provision double (or more - see next paragraph) RAM is indeed intended to address the worst-case scenario.

Note that during a background save, besides the fork's memory consumption, the main process may experience unrelated memory pressure due to any number of reasons besides data set size increase (e.g. replication and client buffers). These also may also require worst-case provisioning.

That said, monitoring is key here - after accumulating some milage with your Redis deployment and the load/traffic that your application generates, you'll be able to make a more educated decision as to exactly how much extra RAM you want to have.

Cheers,

Itamar

[Greg Andrews]

Redis does indeed fork a child process to perform the background save.  However, the copy-on-write behavior in Linux (and other operating systems) means that the parent Redis process and the child Redis process *share* the vast majority of the parent process's RAM.  The memory pages are only copied to make double the consumption when the parent process or the child process changes (writes) the memory page.

It's very useful to understand how it works, so you can interpret the memory consumption behavior you see on your servers, and can design ways to avoid trouble.  I'm numbering the points below, not because they in an important sequence, but because referencing a particular point by number is easier.

1. When the child process is forked from the parent, they share almost all the memory.  They share it - the memory consumption is not 2.x.  More like 1.01x - just a small amount more.
2. The child process never writes to the Redis keys/values in memory, it only reads from them and writes to disk.  So the child does not cause the memory consumption to increase beyond a small amount for buffering the disk writes.
3. If the parent process is not receiving any write/increment commands during the time the child is saving to disk, the parent process won't write to the keys/values either, and the memory consumption won't grow.
4. If the parent process receives a few write/increment commands during the time the child is saving to disk, the parent will write a few memory pages (usually 4k bytes per page), and the memory consumption will grow a little.
5. If the parent process receives many write commands during the time the child is saving to disk, but the commands are for only a few keys, the memory consumption will rise until the pages with those keys have been copied and then the consumption will stop rising.  The two processes will each have a different copy of those few pages, and there's nothing causing more pages to be copied.
6. If the parent process receives many write commands during the the time the child is saving to disk, and the commands are for many, many different keys, the memory consumption will rise a lot more.

Notice I kept saying "during the time the child is saving to disk".  If that time is long because writing to disk is slow, then there's more opportunity for the parent to receive commands that change more keys, and more opportunity for the memory consumption to grow.  If that time is short because disk i/o is fast, then there's less opportunity for changes to the database and less growth of the memory.  So: having fast disk when you're using snapshots is a good thing.

As was already mentioned, if conditions on your server make the child take longer to save to disk, having enough memory to handle the growth in memory is a good thing.

Finally, consider the option of saving to disk on a replica server that's only used for backups, and which you can disconnect the Redis process from the master just before saving to disk.  Since there are no changes coming in, the memory won't grow while the child process is saving to disk.  Then re-connect the Redis process to the master until it's time for the next snapshot.

And there's the other persistence approach: Write to the append-only file (AOF), which doesn't fork a child process and try to firehose the entire database to disk in one action.  It streams the update commands to the append-only file, which doesn't have the risk of doubling the memory consumption.