Understanding Redis swap (memory) usage

[Bobby]

We are currently running Redis 2.9.0 on a server running Ubuntu 10.04
(Lucid) with 96 GB of physical memory and ~2 GB of swap (our standard
server configuration). Even when the server has a significant amount
of physical memory free, we are seeing Redis using a large percentage
of swap. Lowering the swappiness value has had little effect and we
see high swap usage at times even in our smaller Redis
"instances" (using less than 1/3 of the physical memory).

Can anyone offer advice for troubleshooting/understanding Redis' swap
usage? Additionally, are there any best practices related to memory
and swap configuration for Linux servers running Redis? Any
information, recommendations, suggestions, etc. would be greatly
appreciated.

Below is various information from one of our more heavily used
"instances":
redis 127.0.0.1:6379> info
# Server
redis_version:2.9.0
redis_git_sha1:cf1eefa4
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:1024
tcp_port:6379
uptime_in_seconds:3307015
uptime_in_days:38
lru_clock:207466

# Clients
connected_clients:294
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:55222713624
used_memory_human:51.43G
used_memory_rss:61091758080
used_memory_peak:59999302880
used_memory_peak_human:55.88G
mem_fragmentation_ratio:1.11
mem_allocator:tcmalloc

# Persistence
loading:0
aof_enabled:1
changes_since_last_save:2238482
bgsave_in_progress:0
last_save_time:1323275491
bgrewriteaof_in_progress:0

# Diskstore
ds_enabled:0

# Stats
total_connections_received:7451280
total_commands_processed:3049406036
expired_keys:0
evicted_keys:0
keyspace_hits:1114377651
keyspace_misses:604930416
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:2955454

# Replication
role:master
connected_slaves:0

# CPU
used_cpu_sys:18459.63
used_cpu_user:35874.96
used_cpu_sys_childrens:33731.82
used_cpu_user_childrens:14570.56

# Cluster
cluster_enabled:0

# Keyspace
db0:keys=104316996,expires=0

----------

root@<hostname>:~# free -m
total used free shared buffers
cached
Mem: 96871 95146 1725 0 12
32157
-/+ buffers/cache: 62976 33895
Swap: 1953 1850 103

----------

root@<hostname>:~# ./getswap.sh | sort -n -k 5 | grep redis
PID=1024 - Swap used: 1885560 - (redis-server )

----------

root@<hostname>:~# cat /etc/redis.conf
daemonize yes
pidfile /var/run/redis.pid
port 6379
timeout 10
loglevel notice
logfile /var/log/redis.log
databases 1
rdbcompression yes
dbfilename kwi.rdb
dir /redis_data
slave-serve-stale-data yes
maxmemory-policy volatile-random
maxmemory-samples 15
appendonly yes
#appendonly no
appendfsync no
no-appendfsync-on-rewrite yes
diskstore-enabled no
diskstore-path redis.ds
cache-max-memory 0
cache-flush-delay 0
hash-max-zipmap-entries 512
hash-max-zipmap-value 64
list-max-ziplist-entries 512
list-max-ziplist-value 64
set-max-intset-entries 512
activerehashing no
#save 300 10

[Didier Spezia]

Hi,

there are multiple possible explanations, but I don't know which one

applies to your system.

1) ccNUMA configuration

Memory on a ccNUMA machine is directly attached to the CPU sockets.

The memory is therefore managed by the system on a per NUMA node basis.

By default, there is some affinity between the process (Redis) running on a

CPU, and the allocated memory. Your memory consumption looks unbalanced

since your largest instance is more than half the total memory of the machine.

Since the system has no free memory on the NUMA node your big Redis instance

runs on, it tries to increase the free list by swapping some memory associated

to this NUMA node (even if you have plenty of memory still available for the

other nodes).

Here is a good MySQL article about this problem (it applies to Redis as well).

http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/ 

Running Redis from numactl and activating interleaving memory policy could help

in that case.

2) Filesystem cache pressure

AOF is active and background saving is sometimes applied. So read/write operations

occur on the filesystem. The output of the free command shows that 1/3 of the

memory is eaten by the filesystem cache, which means those I/O operations are

quite intensive.

With Linux, there is no way to limit the size of the filesystem cache. On I/O intensive

systems, the cache will grow and possibly trigger swapping. On some older kernel versions,

it occurred even when the swappiness parameter was set to zero. I think it has improved on

more recent kernel versions, but I don't know from which version (Ubuntu 10.04 is based

on 2.6.32).

For RDB dumps, one way to decrease this effect would be to patch Redis and use

fsync + posix_fadvise in order to limit the filesystem cache consumption at save time.

However, I'm not so sure it would work for AOF, because the AOF file is written and

then read again during rewrite. So if the AOF file is not anymore in the filesystem cache,

it will probably have a serious consequence on the performance of the background

rewrite operations.

My advice would be to try to migrate to a more recent kernel version and set swappiness

to zero.

Best regards,

Didier.

[Jak Sprats]

Hi Didier,

nice writeup and sweet article

thanks,
jak

On Dec 7, 7:56 pm, Didier Spezia <didier...@gmail.com> wrote:
> Hi,
>
> there are multiple possible explanations, but I don't know which one
> applies to your system.
>
> 1) ccNUMA configuration
>

>Hi Memory on a ccNUMA machine is directly attached to the CPU sockets.

> The memory is therefore managed by the system on a per NUMA node basis.
> By default, there is some affinity between the process (Redis) running on a
> CPU, and the allocated memory. Your memory consumption looks unbalanced
> since your largest instance is more than half the total memory of the
> machine.
>
> Since the system has no free memory on the NUMA node your big Redis instance
> runs on, it tries to increase the free list by swapping some memory
> associated
> to this NUMA node (even if you have plenty of memory still available for the
> other nodes).
>
> Here is a good MySQL article about this problem (it applies to Redis as

> well).http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-...