Master and slave memory usage

[Jon Keating]

Hi,

I currently have 2 servers of Redis 2.1.7 in production. The master
has 200 persistent connections to it, and the slave has none. The
slave is currently used as a standby server if the master goes down.

Here is the memory usage of master and slave:

Master:
used_memory:1273673596
used_memory_human:1.19G
used_memory_rss:942362624

Slave:
used_memory:77738616
used_memory_human:74.14M
used_memory_rss:80416768

The master's memory keeps growing, so I am a bit concerned about it.

What could be the cause for this? Let me know if there is any other
information needed.

Thanks,
Jon

[Salvatore Sanfilippo]

Hello Jon,

please post at minimum full INFO output for both instances and as much
details as you can about your setup.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Jon Keating]

Hi Salvatore,

Thanks for the reply. My setup is a simple master/slave with the slave
as a standby server. The master has 200 persistent connections and the
slave has 0 connections. For monitoring, I have Nagios querying each
server by issuing a INFO command. The data can be lost (it is just a
cache), so there is no saving to disk happening. I hope that is enough
of a description that you need.

The version was before the 2.2 branch was made, so it still is shown
as 2.1.7

Since it is 32-bit, I'm going to have to swap it out with a 64-bit
compiled version... I didn't expect it to use up this much memory so
quickly. It has been running for one week.

Master INFO:
redis_version:2.1.7
redis_git_sha1:bbac56c2
redis_git_dirty:0
arch_bits:32
multiplexing_api:epoll
process_id:12333
uptime_in_seconds:534369
uptime_in_days:6
lru_clock:1466614
used_cpu_sys:124.04
used_cpu_user:201.31
used_cpu_sys_childrens:0.00
used_cpu_user_childrens:0.00
connected_clients:201
connected_slaves:1
blocked_clients:0
used_memory:1645726672
used_memory_human:1.53G
used_memory_rss:1157341184
mem_fragmentation_ratio:0.70
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:5709742
bgsave_in_progress:0
last_save_time:1293394517
bgrewriteaof_in_progress:0
total_connections_received:2328
total_commands_processed:104147824
expired_keys:1501
keyspace_hits:88994604
keyspace_misses:8679022
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master
db0:keys=179942,expires=287


Slave INFO:
redis_version:2.1.7
redis_git_sha1:bbac56c2
redis_git_dirty:0
arch_bits:32
multiplexing_api:epoll
process_id:12237
uptime_in_seconds:534475
uptime_in_days:6
lru_clock:1466627
used_cpu_sys:16.87
used_cpu_user:14.16
used_cpu_sys_childrens:0.00
used_cpu_user_childrens:0.00
connected_clients:2
connected_slaves:0
blocked_clients:0
used_memory:79148740
used_memory_human:75.48M
used_memory_rss:80916480
mem_fragmentation_ratio:1.02
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:5712732
bgsave_in_progress:0
last_save_time:1293394517
bgrewriteaof_in_progress:0
total_connections_received:1814
total_commands_processed:5714544
expired_keys:0
keyspace_hits:0
keyspace_misses:0
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:slave
master_host:dros-db01.int.sekaicamera.com
master_port:6379
master_link_status:up
master_last_io_seconds_ago:0
master_sync_in_progress:0
db0:keys=180392,expires=287

Thanks, and Happy New Year :)

Jon

> Salvatore 'antirez' Sanfilippohttp://invece.org

[Pieter Noordhuis]

Hi Jon,

Happy new year to you too.

Something weird is definitely going on, as the fragmentation ratio is < 1. This should always be >= 1 because the internal allocation counters will always count less than the actual used memory (RSS). Next thing: I notice there are -- as expected -- a lot less command executions on the slave than on the master. Also it has only 2 connected clients (being the replication link and Nagios). But: are you sure there were no writes against the slave? For instance: LPUSH goes to the master and RPOP goes to the slave. This would cause memory consumption on the master to always grow, while the slave will be stable (as long as the list lengths are stable).

Could you, just to be sure, create a dump of the dataset on the master (BGSAVE) and load it into a fresh redis-server? This will give us the memory footprint of your data without being influenced by anything else.

Cheers,

Pieter

[Salvatore Sanfilippo]

Agree 100% with Pieter except that it's ok to have negative
fragmentation ratio, this means there is some large piece of never
used data on the master that is never accessed, so it was swapped on
disk reducing the RSS figure to an amount that is less than the
allocated bytes.

My guess is that the most probable cause here is writes against the
slave (different number of keys, also)

And I would suggest exactly the same first step inspection that is to
BGSAVE on master and load the resulting dump into a fresh instance to
check what the used memory really is.

Another thing that comes in mind is if you have some special encoding
settings (for instance number of elements per ziplist) very different
in two instances. Definitely something that could produce the effect
we are seeing but I tend to think it's a different matter.

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo

[Jon Keating]

Hi Pieter and Salvatore,

Just wanted to say that the keys are the same on the master and
server. I had to manually type the ssh command, and that explains the
difference in keys in the master and slave. I just ran it via pssh,
and it was exactly the same.

I confirmed the settings, and the slave is not getting any write or
read commands.

Also, the only commands I use are SET and GET.

Would the number of elements per ziplist affect these commands? The
servers should be on nearly identical configurations, minus the
replication part.

I will get back with you guys after I perform a BGSAVE and put the
dataset on a fresh server.

Thanks,
Jon

[Jon Keating]

Hi Pieter and Salvatore,

Here is the INFO command from redis running on a new server. The
binary and config file are exactly the same as the master server. The
dataset was acquired via BGSAVE via the master, as requested. As you
can see, the dataset is fairly small 60M compared to the 1.66G and
growing each second of the master.

My gut reaction is that it has something to do with having persistent
connections. I made the timeout 0 to make sure that connections would
not get reset. Anyways, let me know what you suggest next.

redis_version:2.1.7
redis_git_sha1:bbac56c2
redis_git_dirty:0
arch_bits:32
multiplexing_api:epoll

process_id:1147
uptime_in_seconds:77
uptime_in_days:0
lru_clock:1470496
used_cpu_sys:0.52
used_cpu_user:0.23
used_cpu_sys_childrens:0.00
used_cpu_user_childrens:0.00
connected_clients:1
connected_slaves:0
blocked_clients:0
used_memory:63233544
used_memory_human:60.30M
used_memory_rss:70922240
mem_fragmentation_ratio:1.12

use_tcmalloc:0
loading:0
aof_enabled:0

changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1293967608
bgrewriteaof_in_progress:0
total_connections_received:1
total_commands_processed:0

expired_keys:0
keyspace_hits:0
keyspace_misses:0
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0

role:master
db0:keys=221670,expires=284

Thanks,
Jon

On Sun, Jan 2, 2011 at 7:27 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

[Jon Keating]

I appreciate the time you take to look into this.
If there is anymore information you need in help debugging this,
please let me know.

Thanks,
Jon

[Pieter Noordhuis]

Hi Jon,

Did you load the master from an existing RDB (that might have been larger than your current working set)? If so, and a lot of keys were expired/deleted in the mean time, this would explain the larger memory usage. When everything is loaded into memory and is later purged, there can be leftover allocations that prevent releasing them back to the OS. This would also explain the <1 fragmentation ratio, because in normal operations, it is likely that Redis touches nearly all memory pages. However, when there are many pages that cannot be reclaimed by the OS and thus are not used, the OS can mark them as unused and swap them to disk.

Cheers,

Pieter

[Jon Keating]

Hi Pieter,

No, the server was started fresh with 0 keys. I watched with concern
as the memory usage grew quickly and continues to grow at a rate of
200-300MB per day. As can be seen with the latest INFO output, the
data set is under 100MB but currently is using 2.2GB of memory in
production, with no sign of slowing down.

Should I do some testing to see if it really is persistent connections
that are the cause? If not, what else can I do?

Thanks,
Jon

> > > >>> > > redis-db+u...@googlegroups.com<redis-db%2Bunsubscribe@googlegroups.c om>
> > .

> > > >>> > > For more options, visit this group
> > > >>> > > athttp://groups.google.com/group/redis-db?hl=en.
>
> > > >>> > --
> > > >>> > Salvatore 'antirez' Sanfilippohttp://invece.org
>
> > > >>> > "We are what we repeatedly do. Excellence, therefore, is not an

> > act,...
>
> read more »

[Salvatore Sanfilippo]

Jon this is a pretty strange condition actually.
It's very strange (but not impossible) that we have a leak using such
simpler commands.
If it's not a leak, this kind of problem may result from a replication
link that is not consumed by the master, but this does not seem to be
the case here.

Another thing that can cause such a problem is a client doing
something strange like sending a MULTI, and then a lot of commands,
but never calling EXEC.

A simpler thing causing this problem is when clients don't read the
output from the server.
I'll add more information about this condition in the info output in
the 2.2 branch so that if you can upgrade we can see if that's the
case. I'll do the same also for the replication link, since when the
link is not fast enough compared to the rate at which the master
receives command this can also happen.

While the version you are using is pretty old, there are no critical
bugs I can find in the changelog after your release. To get more
information please can you retry to check with 'redis-cli monitor'
what commands are executed?

Cheers,
Salvatore

> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"We are what we repeatedly do. Excellence, therefore, is not an act,

[Jon Keating]

Hi Salvatore,

I can run it against valgrind and see if there is a leak, but I
imagine you have done so in a test or others would experience the same
problem. So probably not likely.

The only commands I run are (in order of usage):

GET
SET
SETEX
DEL

And I verified this by running monitor.

The client library I use is hiredis, and I check the reply of each
command. Is there something special that must be done when using the
hiredis library?

I only noticed this issue in production, but our test environment
doesnt get much traffic so it is hard to notice. Let me know which
commit I should base my testing off of and I'll try to get around to
it tomorrow or the day after. (nearly bedtime here ;)

Thanks,
Jon

> >> > > >>> expired_keys:0...
>
> read more »

[Salvatore Sanfilippo]

Hello Jon,

I wonder if you have any news about this, or if there is a way for us
to investigate this further.

We are going to release Redis 2.2 as stable, and this is the only
issue we have about it.
While we can't consider this a blocking bug as there is no clearly
identified bug, and may a matter of configuration or other problems
related to writing against the slave, I definitely want to make some
effort to track it.

To explore a few possibilities:

This kind of asymmetries are possible if you have a master -> slave
setup where you use the new maxmemory key eviction policies. With
maxmemory deletions of evicted keys are not transmitted, as in theory
you may want to have different max memory settings in the master and
slave, and since random deletion of keys can only apply to caches,
there are no consistency concerns.

But what happens If I forget to set maxmemory in one instance, master
or slave? that the memory will go higher and higher. Just an example.

I just added two new info fields in INFO output (both in 2.2 branch
and unstable) that show the max input and output buffer lengths, so we
can also check if it's a matter of a client not reading fast enough,
or a master <-> slave link that is not fast enough so more and more
data is queued on the master.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware