Re: How does Redis release memory?

[Salvatore Sanfilippo]

Hello,

Redis always uses the peak memory usage during it's life, unless you
*restart* it from scratch.
So basically if you want to use 5 GB for 10 seconds, and all the other
day you use 100 MB, you need 5 GB of memory free always as after the
peak memory usage it will never release the memory again.

This is not going to change in the future for a number of technical reasons.

Cheers,
Salvatore

On Thu, Mar 14, 2013 at 3:49 PM, Andras Zimmer <andras...@gmail.com> wrote:
> First and foremost: thanks for Redis! It's versatile and capable -- meets
> pretty much all of our DB-in-memory needs. Having used it for a while I
> have a few questions about its memory allocation (or more precisely:
> release) behavior.
>
> Background: Our application uses Redis on Ubuntu 12.04:
> redis_version:2.6.10; mem_allocator: jemalloc-3.2.0. We have three major use
> cases for the Redis DB:
>
> keys that have list values and that expire in 1-2 days, there are a ton of
> these
> keys that get DEL'd at the end of their useful life (usually less than an
> hour) -- they are of various types and they are also numerous
> a handful of keys that live for a long time (mostly lists and sets) -- but
> their content change heavily
>
> After weeks of uptime I realized that the amount of memory allocated to
> Redis slowly but constantly grew. I know that the "live" (non-expired,
> non-deleted keys (along with content)) must not require even close to 100MBs
> of memory -- but Redis was using nearly 800MBs. I double checked and there
> were only a few "leftover" objects in the DB that I DEL'd. No surprise,
> nothing changed.
>
> I then checked and realized that maxmemory was not set. So I set it from
> redis-cli with 'config set maxmemory 629145600' to 600MB in the hopes that
> Redis will free 200MB. I had to stop the server a few hours later anyway so
> I didn't do a thorough test but it looked like to me that Redis did not in
> fact free up any memory; altough it seemed not to allocate more. Thus comes
> question #1:
>
> Q1: What is Redis's memory release policy if maxmemory is manually configed
> to a smaller amount than what is already allocated by Redis? (And as a
> corrolary: is there any other way to force Redis to "shrink" rest of
> restarting it?)
>
> Having read docs (at least trying -- better late than never :)) I'm still
> unsure of a few things:
>
> Q2: When does Redis release memory allocated for items that get DEL'd: at
> near DEL time, at the time it would re-use the memory (so effectively it is
> reluctant to ever give up on it), at approaching maxmemory, or at some other
> time?
>
> Q3: What happens (memory-wise) with list and set items that get removed from
> keys (when the keys themselves don't get removed)? Do they get freed
> individually or only when the key itself is removed? (See use case 3
> above.)
>
> Q4: Am I right in that Redis will get rid of expired items only when using
> 'maxmemory-policy volatile-lru' (the default). The doc says: "remove the
> key with an expire set using an LRU algorithm": so might it be possible that
> Redis removes items that have their expiry set but not yet expired? If so
> does Redis prefer actually expired items when evicting keys?
>
> I would appreciate any help with these -- and I'm sorry in advance if they
> are documented but I was too sloppy in my reading.
>
> Regards,
>
> Andras
>
> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>



--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Dvir Volk]

Doesn't flushing the dbs clear memory?

[Salvatore Sanfilippo]

On Thu, Mar 14, 2013 at 7:37 PM, Dvir Volk <dvi...@gmail.com> wrote:

> Doesn't flushing the dbs clear memory?

Basically depends on the allocator and its exact version, and if you
have still a few keys in other DBs.
If there are enough fragmented objects around there is no allocator
that will be able to release memory, and many allocators including
jemalloc are more optimized to provide better performances and low
fragmentation than releasing memory.

Cheers,
Salvatore

[Andras Zimmer]

Thank you, Salvatore.

So, I understand that Redis (esp. w/ jemalloc) reuses memory it once allocated it from the OS.  That is: deleted items (keys, values) and removed items in "collections" (lists, sets) will automatically and internally get reallocated to new requests but never freed to the OS.

The only thing left for me is to make sure that 'maxmemory-policy volatile-lru' will only ever evict expired keys (and not as per the wording of the docs keys with expiration set).

Andras

PS: If you don't mind I'd add these pieces of info to .../topics/memory-optimization. Someone else may also found it useful.

[Andras Zimmer]

By looking at the source (:)) and more closely at the docs it is clear that Redis will evict expired keys with positive TTL; since it automatically DEL's expired ones. IMHO it also deserves a word of clarification in the docs.

Anyway, I'm all set.  (In my particular case I should have a larger-than-necessary maxmemory for Redis and not allow key eviction at all.)

Andras