                                                                                                                                                                                                                                                                
Received-SPF: pass (google.com: domain of aglad...@gmail.com designates 209.85.216.50 as permitted sender) client-ip=209.85.216.50;
MIME-Version: 1.0
In-Reply-To: <CA+XzkVdRMQiDGDL=mqNYeaP27281TwyZsXKm-pOgeMXYU+0...@mail.gmail.com>
References: <CABZR_7C=vig2BCps1wiwzYkE3G38g3FhQSNnxZBM5m_ZmQ-...@mail.gmail.com>
 <CABZR_7DFY3u+9vcLo5saB6f4-HYWfYU3DanEafzxdPbJg07...@mail.gmail.com>
 <CAMrLSE7w0Qe9aLQFKKSOzVM_ZBMZq1K+1EKhYxHMdZ=Xs2B...@mail.gmail.com>
 <CABZR_7AhLRDPsQVYQtSuDOm-1gzJiKLC52Kb0LKJ01PLM57...@mail.gmail.com>
 <CA+XzkVf1grVvhq1YQnkEV+QtNHNcK8em2oort_QOeLZSt7c...@mail.gmail.com>
 <CABZR_7B_=4H0kMdvFB7fygyYMkgKbxZ_Zkdq9NsQU25efUd...@mail.gmail.com>
 <CA+XzkVccHmnVS4Nxm_dS8hU-mzCQYWUKu0o0M=_oNCx6DF7...@mail.gmail.com>
 <CABZR_7AEbkqNKnrjiWHDZ9e+AUmgzCA+HPHRGVRZNJk0-dO...@mail.gmail.com> <CA+XzkVdRMQiDGDL=mqNYeaP27281TwyZsXKm-pOgeMXYU+0...@mail.gmail.com>
From: Alexander Gladysh <aglad...@gmail.com>
Date: Wed, 24 Oct 2012 13:48:39 +0400
Message-ID: <CABZR_7ADOQ-i99tov4bLzVuxXTQsCah_0t9m_x8zP0c4Kze...@mail.gmail.com>
Subject: Re: Reported Redis 2.4.14 memory usage grows dramatically after first bgrewriteaof
To: redis-db@googlegroups.com
Content-Type: text/plain; charset=UTF-8

On Wed, Oct 24, 2012 at 1:43 PM, Salvatore Sanfilippo <anti...@gmail.com> wrote:
>
>
> On Wed, Oct 24, 2012 at 11:18 AM, Alexander Gladysh <aglad...@gmail.com>
> wrote:
>>
>>
>> Fair enough. Maybe this should be mentioned in the docs?
>
>
> Maybe, but actually to make it simpler for the user probably there is
> another trick:
>
> diff --git a/src/rdb.c b/src/rdb.c
> index fd9fcac..b7ce885 100644
> --- a/src/rdb.c
> +++ b/src/rdb.c
> @@ -1010,6 +1010,8 @@ void startLoading(FILE *fp) {
>  /* Refresh the loading progress info */
>  void loadingProgress(off_t pos) {
>      server.loading_loaded_bytes = pos;
> +    if (server.stat_peak_memory < zmalloc_used_memory())
> +        server.stat_peak_memory = zmalloc_used_memory();
>  }
>
>
> Testing this code so that the peak memory is reported correctly while
> loading and committing it into 2.6 / unstable.

Thanks!

>> BTW, if you look at the original post, you'll see that INFO with drop
>> in the memory is actually taken while Redis was loading
>
>
> Not sure about what you mean, but this is what happens:
>
> Redis forks
> The child rewrites the AOF
> in the meantime the main process accumulate writes in memory
> At the end accumulated writes are flushed
>
> So under heavy write if disk is slow you start using additional memory (2.6
> is better in this regard btw).
>
> After the rewrite terminates the memory is released. If this is not in the
> doc we need to add it... but the good news is that in the latest days I
> spent a number of hours writing a Redis design document that explains in
> details a lot of stuff about Redis internals. However this info should also
> be present in /topics/persistence IMHO.

Thank you for the explanation.

I mean that I screwed up and misinterpreted what I saw. "Memory jump" I saw:

used_memory_human:655.01M

to

used_memory_human:239.86M

to

used_memory_human:643.12M

was actually because Redis was still loading in the middle. So, 239.86
MB instead of 600+MB is just partially loaded data. I should have
waited until everything is loaded before taking INFO.

Sorry for the noise again,
Alexander.