Peculiar Redis Memory Usage

[Tyler Sullens]

Hi, 

I’m trying to understand a (seemingly) peculiar issue I’m having where our Redis used memory seems to be about 2x the size of our actual key space. 

We are running 3.2.9 in a master / slave configuration. Our redis memory use varies between ~20GB and 28GB depending on day of week and time of day. The strangeness I see however is that restarting a redis instance will reduce this to about half of whatever the redis used memory was at the time of restart, and inspecting the .rdb dump of a BGSAVE will indicate that the actual key space seems to again be about %50 of our total redis used memory (which makes sense, a restart essentially being SYNCing a BGSAVE from the master). 

Typically within 12-24 hours the restarted instance is again at the same used memory levels as its peers. 

Our fragmentation ratio remains at a low level (1.03 at time of this email).

We do write temporary keys to slaves as part of a pipeline process, but take great care to issue deletes at the end. I have also observed this behavior from a slave that has no clients using it and is only replicating from the master.

I understand the mechanics of redis re-using memory allocated to it from the OS and that an .rdb dump does not accurately reflect the redis used memory (should be within %10 though right?), but it is strange to me that we should see redis using such a high amount of memory (again, seems like 2x the actual size of the “live” key space).

I do not think this is a problem with redis but is probably something we are doing wrong, and am wondering if anyone has any suggestions or or information to help me identify if this is actually a real problem or expected, and maybe what I can do to figure this out.

Tyler Sullens

DevOps Engineer

NPR Digital Media

[Tuco]

Hi, 

you mentioned "We do write temporary keys to slaves as part of a pipeline process" 

Aren't slaves meant to be read only?

[hva...@gmail.com]

What does the output of the INFO command say with regard to fragmentation ratio after the process has been running 12-24 hours?

My guess is that your pattern of key creation and deletion prevents the kernel from taking all the memory back when Redis frees it.  From near the end of the Memory Optimization documentation page https://redis.io/topics/memory-optimization :

Redis will not always free up (return) memory to the OS when keys are removed. This is not something special about Redis, but it is how most malloc() implementations work. For example if you fill an instance with 5GB worth of data, and then remove the equivalent of 2GB of data, the Resident Set Size (also known as the RSS, which is the number of memory pages consumed by the process) will probably still be around 5GB, even if Redis will claim that the user memory is around 3GB. This happens because the underlying allocator can't easily release the memory. For example often most of the removed keys were allocated in the same pages as the other keys that still exist.

As it says, this will happen with any long-running process that uses memory in large chunks and small portions.  This could be the cause of the memory usage patterns you're seeing.

[Tyler Sullens]

We see a fragmentation ratio between 1.00 and 1.20 on any given instance at any given time.

I understand the memory optimization that is at work here, and I agree that our usage pattern is what’s at cause here… I was hoping to confirm that it is a usage pattern problem and maybe some idea on how to identify the pattern.

Tyler Sullens

DevOps Engineer

NPR Digital Media

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.