Redis memory configuration for high writes

[Chandra]

Hi Team,

Need suggestion on how much max memory setting, should i keep.

I have a system of with heavy writes i.e. 1 Million Ops and the clusters configuration is 4 Cores and 16 GB.

Created 10M + 10S with AOF enabled and RDB at night time.

Sometimes seeing the RDB Cow is going beyond 5 GB.

Please help me in finding a formula or suggestion on how one should set the max memory of redis and also how much should we spare for the systems considering such a huge scale.

[Sotto Voce]

In a scenario where a Redis server constantly has a lot of write activity, you generally don't want that same process to save snapshots to RDB.  The reason is the writes create memory consumption via the COW (copy-on-write) feature.  You need very fast disk i/o and a huge amount of unused memory to keep the COW activity from hitting the limits of the server's memory.

One useful technique is to replicate the database to another server that's set aside for saving the snapshots (no other read or write activity).  On the replica server the write activity is much lower (only the replication updates) so the COW memory growth is slower.

Another approach is to "shard" your database among multiple Redis servers.  The rate of writes is lower on each of the servers, so the COW growth is slower.

Saving to an Append-Only File (AOF) can remove the COW activity, since a child process is not being spawned that tries to save the whole database in a single snapshot.

None of the above ideas is perfect - they don't solve the write/COW issue neatly and cleanly without introducing potential problems of their own.  They're things to consider and test with your data and write/read patterns.

[Chandra]

In general what would be the guiding principle for setting up the max memory like some x% of system memory.. what would be that X.