Redis crashing

[Boštjan Božič]

Hello,

I am using Redis version 3.2.9. I am experiencing issue of server crashing. I believe issue might be that I ran out of memory, as Redis is basically taking all memory resources. I have RDB snapshot enabled, so after each crash, DB is restored, which means I am out of memory again. What I see also that bg_save status is stuck on error.

I wonder what would be best follow-up on this? Delete snapshot and reimport all data, adding EX parameter to keys?

Below is output of INFO:

# Server

redis_version:3.2.9

redis_git_sha1:00000000

redis_git_dirty:0

redis_build_id:3374d6ee561e7f16

redis_mode:standalone

os:Linux 3.10.0-514.6.1.el7.x86_64 x86_64

arch_bits:64

multiplexing_api:epoll

gcc_version:4.8.5

process_id:43804

run_id:48077b1da4eeb5a2a52addf2b1283476a4fe33df

tcp_port:6379

uptime_in_seconds:1931

uptime_in_days:0

hz:10

lru_clock:2088562

executable:/opt/redis/bin/redis-server

config_file:/etc/redis/6379.conf

# Clients

connected_clients:31

client_longest_output_list:0

client_biggest_input_buf:0

blocked_clients:0

# Memory

used_memory:32067766736

used_memory_human:29.87G

used_memory_rss:30606123008

used_memory_rss_human:28.50G

used_memory_peak:32069111304

used_memory_peak_human:29.87G

total_system_memory:33560993792

total_system_memory_human:31.26G

used_memory_lua:37888

used_memory_lua_human:37.00K

maxmemory:0

maxmemory_human:0B

maxmemory_policy:noeviction

mem_fragmentation_ratio:0.95

mem_allocator:jemalloc-4.0.3

# Persistence

loading:0

rdb_changes_since_last_save:17080

rdb_bgsave_in_progress:1

rdb_last_save_time:1528813287

rdb_last_bgsave_status:err

rdb_last_bgsave_time_sec:1474

rdb_current_bgsave_time_sec:32

aof_enabled:0

aof_rewrite_in_progress:0

aof_rewrite_scheduled:0

aof_last_rewrite_time_sec:-1

aof_current_rewrite_time_sec:-1

aof_last_bgrewrite_status:ok

aof_last_write_status:ok

# Stats

total_connections_received:8535

total_commands_processed:3558946

instantaneous_ops_per_sec:2961

total_net_input_bytes:164792133

total_net_output_bytes:58411194

instantaneous_input_kbps:119.50

instantaneous_output_kbps:18.25

rejected_connections:0

sync_full:0

sync_partial_ok:0

sync_partial_err:0

expired_keys:0

evicted_keys:0

keyspace_hits:13533

keyspace_misses:2853119

pubsub_channels:0

pubsub_patterns:0

latest_fork_usec:306428

migrate_cached_sockets:0

# Replication

role:master

connected_slaves:0

master_repl_offset:0

repl_backlog_active:0

repl_backlog_size:1048576

repl_backlog_first_byte_offset:0

repl_backlog_histlen:0

# CPU

used_cpu_sys:147.12

used_cpu_user:404.23

used_cpu_sys_children:26.50

used_cpu_user_children:27.80

# Cluster

cluster_enabled:0

# Keyspace

db0:keys=268297184,expires=0,avg_ttl=0

And this is current config file:

# General

daemonize yes

pidfile /var/run/redis/6379.pid

dir /redis/data/6379

port 6379

bind 0.0.0.0

timeout 0

tcp-keepalive 0

tcp-backlog 511

loglevel notice

logfile ""

syslog-enabled yes

syslog-ident redis_6379

syslog-facility USER

databases 16

 

# Snapshotting

save 900 1

save 300 10

save 60 10000

stop-writes-on-bgsave-error yes

rdbcompression yes

rdbchecksum yes

dbfilename dump.rdb

 

# Replication

slave-serve-stale-data yes

slave-read-only yes

repl-disable-tcp-nodelay no

slave-priority 100

min-slaves-max-lag 10

# Security

# Limits

maxclients 10000

maxmemory-policy noeviction

 

# Append Only Mode

appendonly no

appendfilename "appendonly.aof"

appendfsync everysec

no-appendfsync-on-rewrite no

auto-aof-rewrite-percentage 100

auto-aof-rewrite-min-size 64mb

 

# Lua

lua-time-limit 5000

 

# Slow Log

slowlog-log-slower-than 10000

slowlog-max-len 128

 

# Event Notification

notify-keyspace-events ""

 

# Advanced

hash-max-ziplist-entries 512

hash-max-ziplist-value 64

list-max-ziplist-entries 512

list-max-ziplist-value 64

set-max-intset-entries 512

zset-max-ziplist-entries 128

zset-max-ziplist-value 64

activerehashing yes

client-output-buffer-limit normal 0 0 0

client-output-buffer-limit slave 256mb 64mb 60

client-output-buffer-limit pubsub 32mb 8mb 60

hz 10

aof-rewrite-incremental-fsync yes

Thank you and best regards,

Bostjan

[Shen Longxing]

According to the info message, you set maxmemory to 0. It means no limit to the used memory for Redis.

When forking a child process to save rdb file, probably an oom occurred. According  to the oom killer algorithm, child processes are killed first.
You can check dmesg or /var/log/messages to identify this.

I recommend you to set like this:
maxmemory  50%(or above) of host memory, because of the COW
maxmemory-policy volatile-lru   if you want to evict expired key

在 2018年6月12日星期二 UTC+8下午11:06:31，Boštjan Božič写道：

[Boštjan Božič]

Hello Shen,

thank you for provided information. Yes, this is what I did in the end. I had to kill Redis process first though, as it was in infinite loop.

Best regards,

Bostjan