Redis memory fragmentation ratio reached 5000 !!!

[张新峰]

Hi, all

   I noticed a very strange phenomenon during my test against redis. Our version of Redis is 2.8 and the max-memory is 13G(The physical memory is 15G).

First we write a lot of 1-2K KV pairs with expiration 1 hour or 1 day into Redis until the it begin to evict and all the memory is used up. After that, the fragmentation ratio is about 1.03-1.04 which is very satisfying. 

Then, we stop our load and redis begin to expire all the stored keys and the used memory goes down. But strange thing happened, the RSS didn't go down at all!  This makes the memory fragmentation ratio begin to go up until after 1 day all the keys are expired. At that time, the memory fragmentation reached about 500. Then after a while, the RSS finally begin to go down a little bit and rise again?! Until the memory fragmentation reached 5000 at the peak, then the RSS dropped dramatically. Memory usage graph is like this:

The memory fragmentation graph is like this: 

Here are my questions:1. We all know and afraid of OOM killer. Though during "normal" test cases, the memory fragmentation ratio looks quite normal.  But in case we see high memory fragmentation ratio, can we do something other than just kill the redis instance?  2. Why is the RSS remain over 14G for so such a long time? Hours after we didn't insert any data into redis, the RSS remains the same. Only after 1 day, RSS begin to drop.  3. Why did RSS go up again? We didn't insert any data into redis at that time and RSS went up? Looks very bizarre to me.

[Josiah Carlson]

To answer your question 1:

There are things that you can try, but ultimately you do the equivalent of restarting Redis: flush all of your data, kill all your connections.

To answer your question 2:

I believe this is related to behavior of the underlying memory allocator, jemalloc. You can read a really good overview here: https://www.facebook.com/notes/facebook-engineering/scalable-memory-allocation-using-jemalloc/480222803919

To make a long story short; when allocating memory, jemalloc requests a large chunk of memory it calls an arena, and services smaller requests from those arenas. An arena can only be freed back to the OS when all of the memory that had been allocated from the arena is freed. You are seeing high memory fragmentation ratios as your data expires because there is still some data allocated from a large number of your arenas that just hasn't been freed, so jemalloc can't release that arena back to the OS. As a few last keys get expired, or as those connections recycle, those arenas are slowly accessed and freed, eventually leading to low RSS (assuming that the later downward movement RSS and fragmentation ratio is related to freeing, and not swapping Redis to disk).

Also, looking at your Redis "used memory" and "used memory RSS" numbers, it looks like Redis was partially swapped out just before Thursday 00:00.

To answer your question 3:

RSS is what Redis discovers by examining the contents of /proc for the Redis process. When I said that Redis looked like it was partially swapped out, it was because you saw a drop in RSS then a rise in RSS, but didn't have a similar rise in Redis used memory. This tells me that Redis was swapped out (reducing RSS) and then gradually swapped back in when it needed to expire or handle requests for certain keys. Eventually it either frees all of the data back to the OS, or is completely swapped out. To tell which, I'd need more information from Redis and from other commands running on your server.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.