Redis throws “error on connection” error on client for 2-5 minutes while under high loads

[Mehmet KARCI]

Hi,

Redis throws “on connection” error on client for 2-5 minutes while under high loads. (20k req/s) We are not sure but this error might be occuring during bgsave. What might be the cause for this?

Server: Centos 6, 128 GB ram

Redis info:

    Server
redis_version:3.0.2
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:6be7fc9e6b88f79
redis_mode:standalone
os:Linux 2.6.32-642.1.1.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.7
process_id:21837
run_id:f548cab1cc4709613556465e525013e9d9b77b59
tcp_port:6379
uptime_in_seconds:1758702
uptime_in_days:20
hz:10
lru_clock:13752806
config_file:/etc/redis.conf

 Clients
connected_clients:599
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:13949126752
used_memory_human:12.99G
used_memory_rss:14990618624
used_memory_peak:14741865368
used_memory_peak_human:13.73G
used_memory_lua:36864
mem_fragmentation_ratio:1.07
mem_allocator:jemalloc-3.6.0

 Persistence
loading:0
rdb_changes_since_last_save:277877
rdb_bgsave_in_progress:1
rdb_last_save_time:1473370426
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:157
rdb_current_bgsave_time_sec:111
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1
aof_last_bgrewrite_status:ok
aof_last_write_status:ok

 Stats
total_connections_received:4336483
total_commands_processed:12253702741
instantaneous_ops_per_sec:4705
total_net_input_bytes:663766223026
total_net_output_bytes:1093793460368
instantaneous_input_kbps:255.10
instantaneous_output_kbps:352.74
rejected_connections:15946278
sync_full:0
sync_partial_ok:0
sync_partial_err:0
expired_keys:141120423
evicted_keys:0
keyspace_hits:7492668655
keyspace_misses:1506892131
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:725818
migrate_cached_sockets:0

 Replication
role:master
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0

 CPU
used_cpu_sys:223517.16
used_cpu_user:117783.39
used_cpu_sys_children:100744.66
used_cpu_user_children:1130717.50

 Cluster
cluster_enabled:0

 Keyspace
db0:keys=21447129,expires=9637026,avg_ttl=3602204180

Best Regards,

[The Real Bill]

Redis is rejecting the connections according to the stats there. I suggest looking at the server's logs to see why.

[The Real Bill]

[hva...@gmail.com]

Your Redis info output has two interesting lines:

rdb_last_bgsave_time_sec:157
rdb_current_bgsave_time_sec:111

The last successful background save took 2 minutes and 37 seconds.  When you collected the INFO output, it was in the middle of another background save that had been running for 1 minute 51 seconds.  There's 12G of data being written to disk at about 5GB per minute, which is a decent 85MB / second (for a cloud server).  Your background saves are taking about the same amount of time that you see Redis becoming unresponsive.

What do your other graphs of that server tell you is going on at those same times?  What is the CPU utilization?  The memory utilization?  The disk i/o usage?   Do the periods of connection trouble match the times that Redis is saving to disk?  If you make Redis stop saving to disk for a while (a few hours) do the connection troubles go away?  And re-appear when Redis starts saving to disk again?

The trouble you're seeing may or may not be connected to the background saves.  The above are just the things to investigate in order to find out.