Reported Redis 2.4.14 memory usage grows dramatically after first bgrewriteaof

[Alexander Gladysh]

Hi, list!

Is this expected behaviour? (Full info output below.)

$ redis-cli info

used_memory:686632584
used_memory_human:654.82M
used_memory_rss:640655360
used_memory_peak:1138350024
used_memory_peak_human:1.06G
mem_fragmentation_ratio:0.93

$ redis-cli bgrewriteaof
Background append only file rewriting started

$ redis-cli info

used_memory:686830080
used_memory_human:655.01M
used_memory_rss:640655360
used_memory_peak:1138350024
used_memory_peak_human:1.06G
mem_fragmentation_ratio:0.93

So far so good.

$ sudo service redis-server restart
Stopping redis-server: redis-server.
Starting redis-server: redis-server.

$ redis-cli info

used_memory:251516032
used_memory_human:239.86M
used_memory_rss:235659264
used_memory_peak:0
used_memory_peak_human:0B
mem_fragmentation_ratio:0.94

Note 3x drop after restart.

$ redis-cli bgrewriteaof
Background append only file rewriting started

$ redis-cli info

used_memory:674364888
used_memory_human:643.12M
used_memory_rss:616386560
used_memory_peak:674356352
used_memory_peak_human:643.12M
mem_fragmentation_ratio:0.91

...And we're back again with 3x growth.

Thanks,
Alexander.

$ redis-cli info
redis_version:2.4.14
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.5.2
process_id:21686
uptime_in_seconds:13
uptime_in_days:0
lru_clock:802602
used_cpu_sys:1.25
used_cpu_user:6.63
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:31
connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:674364888
used_memory_human:643.12M
used_memory_rss:616386560
used_memory_peak:674356352
used_memory_peak_human:643.12M
mem_fragmentation_ratio:0.91
mem_allocator:jemalloc-2.2.5
loading:0
aof_enabled:1
changes_since_last_save:3953784
bgsave_in_progress:0
last_save_time:1350203296
bgrewriteaof_in_progress:1
total_connections_received:3481
total_commands_processed:5560
expired_keys:0
evicted_keys:0
keyspace_hits:229
keyspace_misses:869
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:119206
vm_enabled:0
role:master
aof_current_size:549726816
aof_base_size:549494407
aof_pending_rewrite:0
aof_buffer_length:0
aof_pending_bio_fsync:0
db0:keys=1,expires=0
db2:keys=62,expires=0
db3:keys=7964,expires=0
db4:keys=53,expires=6

[Alexander Gladysh]

Any input?

[CharSyam]

Do you use redis with heavy write?

and that time, it is possible,

when redis execute  bgwrite, it forks

at that time it uses much memory.

and check your period that try to save rdb.

and how much memory do you use on your system?

2012/10/24 Alexander Gladysh <agla...@gmail.com>

Any input?

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Alexander Gladysh]

On Wed, Oct 24, 2012 at 12:31 PM, CharSyam <char...@gmail.com> wrote:
> Do you use redis with heavy write?

Here are the gory details:
https://groups.google.com/d/msg/redis-db/vSAvnYVtX9w/NRYlBwOrCTsJ

> and that time, it is possible,
>
> when redis execute bgwrite, it forks
>
> at that time it uses much memory.
>
> and check your period that try to save rdb.
>
> and how much memory do you use on your system?

RDB persistence is disabled in our configuration.

Also: would memory consumed by Redis *fork* be noted in INFO?

Thanks,
Alexander.

[Salvatore Sanfilippo]

Hi Alexander,

is the memory usage still there *after* the rewrite has terminated?

Thanks

Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Alexander Gladysh]

On Wed, Oct 24, 2012 at 1:00 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> Hi Alexander,
>
> is the memory usage still there *after* the rewrite has terminated?

Will double-check that. Meanwhile: peak memory is not updated while loading DB:

$ redis-cli info
redis_version:2.4.14
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
gcc_version:4.5.2

process_id:27332
uptime_in_seconds:6
uptime_in_days:0
lru_clock:889251
used_cpu_sys:1.32
used_cpu_user:4.92
used_cpu_sys_children:0.00
used_cpu_user_children:0.00
connected_clients:2

connected_slaves:0
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

used_memory:1137193424
used_memory_human:1.06G
used_memory_rss:1106731008
used_memory_peak:0
used_memory_peak_human:0B
mem_fragmentation_ratio:0.97
mem_allocator:jemalloc-2.2.5
loading:1
aof_enabled:0
changes_since_last_save:2987997
bgsave_in_progress:0
last_save_time:1351069791
bgrewriteaof_in_progress:0
total_connections_received:2524
total_commands_processed:0
expired_keys:0
evicted_keys:0
keyspace_hits:0
keyspace_misses:0
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:0
vm_enabled:0
role:master
loading_start_time:1351069791
loading_total_bytes:1314418876
loading_loaded_bytes:1046694276
loading_loaded_perc:79.63
loading_eta_seconds:1
db0:keys=1,expires=0
db2:keys=65,expires=0
db3:keys=6614,expires=0

[Salvatore Sanfilippo]

On Wed, Oct 24, 2012 at 11:11 AM, Alexander Gladysh <agla...@gmail.com> wrote:

> Will double-check that. Meanwhile: peak memory is not updated while loading DB:

That's expected behaviour, it is sampled at runtime, not a precise measure.

Cheers,
Salvatore

[Alexander Gladysh]

On Wed, Oct 24, 2012 at 1:13 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> On Wed, Oct 24, 2012 at 11:11 AM, Alexander Gladysh <agla...@gmail.com> wrote:
>
>> Will double-check that. Meanwhile: peak memory is not updated while loading DB:
>
> That's expected behaviour, it is sampled at runtime, not a precise measure.

Fair enough. Maybe this should be mentioned in the docs?

BTW, if you look at the original post, you'll see that INFO with drop
in the memory is actually taken while Redis was loading:

$ redis-cli info

used_memory:251516032
used_memory_human:239.86M
used_memory_rss:235659264

used_memory_peak:0
used_memory_peak_human:0B
mem_fragmentation_ratio:0.94

Oops. :-(

So, everything is OK. Sorry for the noise.

Alexander.

[Salvatore Sanfilippo]

On Wed, Oct 24, 2012 at 11:18 AM, Alexander Gladysh <agla...@gmail.com> wrote:

Fair enough. Maybe this should be mentioned in the docs?

Maybe, but actually to make it simpler for the user probably there is another trick:

diff --git a/src/rdb.c b/src/rdb.c

index fd9fcac..b7ce885 100644

--- a/src/rdb.c

+++ b/src/rdb.c

@@ -1010,6 +1010,8 @@ void startLoading(FILE *fp) {

 /* Refresh the loading progress info */

 void loadingProgress(off_t pos) {

     server.loading_loaded_bytes = pos;

+    if (server.stat_peak_memory < zmalloc_used_memory())

+        server.stat_peak_memory = zmalloc_used_memory();

 }

Testing this code so that the peak memory is reported correctly while loading and committing it into 2.6 / unstable.

 

BTW, if you look at the original post, you'll see that INFO with drop
in the memory is actually taken while Redis was loading

Not sure about what you mean, but this is what happens:

Redis forks

The child rewrites the AOF

in the meantime the main process accumulate writes in memory

At the end accumulated writes are flushed

So under heavy write if disk is slow you start using additional memory (2.6 is better in this regard btw).

After the rewrite terminates the memory is released. If this is not in the doc we need to add it... but the good news is that in the latest days I spent a number of hours writing a Redis design document that explains in details a lot of stuff about Redis internals. However this info should also be present in /topics/persistence IMHO.

Cheers,

Salvatore

[Alexander Gladysh]

On Wed, Oct 24, 2012 at 1:43 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
>
>
> On Wed, Oct 24, 2012 at 11:18 AM, Alexander Gladysh <agla...@gmail.com>
> wrote:
>>
>>
>> Fair enough. Maybe this should be mentioned in the docs?
>
>
> Maybe, but actually to make it simpler for the user probably there is
> another trick:
>
> diff --git a/src/rdb.c b/src/rdb.c
> index fd9fcac..b7ce885 100644
> --- a/src/rdb.c
> +++ b/src/rdb.c
> @@ -1010,6 +1010,8 @@ void startLoading(FILE *fp) {
> /* Refresh the loading progress info */
> void loadingProgress(off_t pos) {
> server.loading_loaded_bytes = pos;
> + if (server.stat_peak_memory < zmalloc_used_memory())
> + server.stat_peak_memory = zmalloc_used_memory();
> }
>
>
> Testing this code so that the peak memory is reported correctly while
> loading and committing it into 2.6 / unstable.

Thanks!

>> BTW, if you look at the original post, you'll see that INFO with drop
>> in the memory is actually taken while Redis was loading
>
>
> Not sure about what you mean, but this is what happens:
>
> Redis forks
> The child rewrites the AOF
> in the meantime the main process accumulate writes in memory
> At the end accumulated writes are flushed
>
> So under heavy write if disk is slow you start using additional memory (2.6
> is better in this regard btw).
>
> After the rewrite terminates the memory is released. If this is not in the
> doc we need to add it... but the good news is that in the latest days I
> spent a number of hours writing a Redis design document that explains in
> details a lot of stuff about Redis internals. However this info should also
> be present in /topics/persistence IMHO.

Thank you for the explanation.

I mean that I screwed up and misinterpreted what I saw. "Memory jump" I saw:

used_memory_human:655.01M

to

used_memory_human:239.86M

to

used_memory_human:643.12M

was actually because Redis was still loading in the middle. So, 239.86
MB instead of 600+MB is just partially loaded data. I should have
waited until everything is loaded before taking INFO.

Sorry for the noise again,
Alexander.

[Salvatore Sanfilippo]

On Wed, Oct 24, 2012 at 11:48 AM, Alexander Gladysh <agla...@gmail.com> wrote:

Sorry for the noise again,

Not noise at all, we are here to fix issues and to signal issues is the best thing one can do to improve Redis reliability. It's part of the game that there are false positives.

Thanks :)

[CharSyam]

Because of Alexander, Salvatore 

I can understand redis more ^^

thank you.

2012/10/24 Salvatore Sanfilippo <ant...@gmail.com>

--