unreasonable high memory used in redis cluster 3.0.7 looking for help

[Tomasen Sen]

Hi. We are using redis cluster on one of our project, with 8 nodes, running on CoreOS, and each have 192G memory.

The data contained in this redis cluster should only about 1GB. Here is INFO MEMORY on one healthy node:

used_memory:1244971880

used_memory_human:1.16G

used_memory_rss:1279311872

used_memory_peak:1256963704

used_memory_peak_human:1.17G

used_memory_lua:36864

mem_fragmentation_ratio:1.03

mem_allocator:jemalloc-3.6.0

But some nodes's memory usage just going top of roof after few days. e.g. here is INFO MEMORY of one salve today:

# Server

redis_version:3.0.7

redis_git_sha1:00000000

redis_git_dirty:0

redis_build_id:179c6547c1df4964

redis_mode:cluster

os:Linux 4.3.6-coreos x86_64

arch_bits:64

multiplexing_api:epoll

gcc_version:4.9.2

process_id:1

run_id:2727e6e307e0961e99a9c9b5810944eae452df00

tcp_port:6379

uptime_in_seconds:612951

uptime_in_days:7

hz:10

lru_clock:3075227

config_file:/etc/redis.conf

# Clients

connected_clients:4

client_longest_output_list:1729179

client_biggest_input_buf:0

blocked_clients:0

# Memory

used_memory:44777498352

used_memory_human:41.70G

used_memory_rss:29916381184

used_memory_peak:44777498352

used_memory_peak_human:41.70G

used_memory_lua:36864

mem_fragmentation_ratio:0.67

mem_allocator:jemalloc-3.6.0

# Persistence

loading:0

rdb_changes_since_last_save:10273

rdb_bgsave_in_progress:0

rdb_last_save_time:1462693000

rdb_last_bgsave_status:ok

rdb_last_bgsave_time_sec:9

rdb_current_bgsave_time_sec:-1

aof_enabled:1

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:5

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_last_write_status:ok

aof_current_size:1662694450

aof_base_size:842431475

aof_pending_rewrite:0

aof_buffer_length:0

aof_rewrite_buffer_length:0

aof_pending_bio_fsync:0

aof_delayed_fsync:0

# Stats

total_connections_received:51680

total_commands_processed:242806449

instantaneous_ops_per_sec:352

total_net_input_bytes:282172384758

total_net_output_bytes:33110317539

instantaneous_input_kbps:423.14

instantaneous_output_kbps:0.03

rejected_connections:0

sync_full:0

sync_partial_ok:0

sync_partial_err:0

expired_keys:0

evicted_keys:0

keyspace_hits:0

keyspace_misses:0

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:578043

migrate_cached_sockets:0

# Replication

role:slave

master_host:10.1.59.6

master_port:6379

master_link_status:up

master_last_io_seconds_ago:0

master_sync_in_progress:0

slave_repl_offset:281902109694

slave_priority:100

slave_read_only:1

connected_slaves:0

master_repl_offset:0

repl_backlog_active:0

repl_backlog_size:16777216

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0

# CPU

used_cpu_sys:3336.90

used_cpu_user:5477.22

used_cpu_sys_children:4032.05

used_cpu_user_children:51680.57

# Cluster

cluster_enabled:1

# Keyspace

db0:keys=1480627,expires=0,avg_ttl=0

 

we are using rdb/bgsave and aof. and the memory usage just going back to normal(1GB) after we restart this redis instance.

Everything else seems to be normal, no special event happened with the memory spike. and there is also no zombie bgsave process in the ps auxww.

Can anyone give us suggestions on how to fix or debug this issue?

here is our redis.conf

daemonize no

pidfile /var/run/redis.pid

tcp-backlog 4096

timeout 0

tcp-keepalive 0

loglevel notice

logfile "/var/log/redis/redis.log"

databases 16

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

dir /var/lib/redis/

slave-serve-stale-data yes

slave-read-only yes

repl-diskless-sync no

repl-diskless-sync-delay 5

# repl-ping-slave-period 10

# repl-timeout 60

repl-disable-tcp-nodelay no

repl-backlog-size 16mb

# repl-backlog-ttl 3600

rename-command FLUSHDB ""

rename-command FLUSHALL ""

rename-command KEYS ""

rename-command SAVE ""

rename-command CONFIG 1cdb34bd67330d1e67f2b512ffef525095205645

# maxclients 10000

maxmemory 180gb

maxmemory-policy noeviction

maxmemory-samples 5

appendonly yes

appendfilename "appendonly.aof"

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

aof-load-truncated yes

lua-time-limit 5000

slowlog-log-slower-than 10000

slowlog-max-len 128

latency-monitor-threshold 0

notify-keyspace-events ""

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

hll-sparse-max-bytes 3000

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

# cluster settings

cluster-enabled yes

cluster-config-file nodes-6379.conf

cluster-node-timeout 5000

cluster-slave-validity-factor 10

cluster-migration-barrier 1

cluster-require-full-coverage yes 

[George Chilumbu]

What kind of data types are you mostly using? 

If you guys are using sets much, then memory usage will be an issue for sure. Another thing is to reduce the size of elements in hashed sets, lists and sorted sets. Read this page :https://github.com/sripathikrishnan/redis-rdb-tools/wiki/Redis-Memory-Optimization

Also, the maxmemory is set too high...about 93%. Maybe reduce it to 80% first and see how that goes. Then to 70% again until you think performance is at normal.

Another thing to do is make sure to disable kernel feature transparent huge pages, which will affect greatly both memory usage and latency in a negative way according to the documentation. This is accomplished with the following command: echo never > /sys/kernel/mm/transparent_hugepage/enabled.

Have you looked at this site:

http://redis.io/topics/admin