Redis connection timeout

[MonoSlon]

Hello,

We are facing with redis connection problem.

There are two Redis instances in master-slave mode, master is for
reads and writes, slave is only for reads, and there are some backends
with php-fpm.
Sometimes php code throws exception "Uncaught exception
'RedisException' with message 'Can't connect to...".

We've made simple php script wich does a loop with "fsockopen" and
"fclose" to redis master's ip:port, and it shows now errors if timeout
in fsockopen is set to 3 seconds or more.

There is no packet loss in network and no big latencies, so it looks
like sometimes Redis accepts connections slowly, but these exceptions
don't correlate with redis dump saving.

Each backend has about 700 sockets in all states, redis master has
about 500 sockets (we've set net.ipv4.tcp_fin_timeout=1,
net.ipv4.tcp_tw_reuse=1, net.ipv4.tcp_tw_recycle=1,
net.ipv4.tcp_syncookies=0), so it's not port exhausting problem, and
it's not phpredis library issue.

Redis version is 2.2.4

Here is top output from master:
load average: 0.56, 0.50, 0.46
Cpu(s): 1.0%us, 1.2%sy, 0.0%ni, 94.7%id, 0.0%wa, 0.2%hi,
2.8%si, 0.0%st
Mem: 98976736k total, 20475568k used, 78501168k free, 692872k
buffers
Swap: 8193140k total, 0k used, 8193140k free, 18268592k
cached

And output of INFO:

info
$3843
redis_version:2.2.4
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:19119
uptime_in_seconds:528386
uptime_in_days:6
lru_clock:1379016
used_cpu_sys:32113.32
used_cpu_user:39985.89
used_cpu_sys_childrens:32491.37
used_cpu_user_childrens:5841.61
connected_clients:166
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:634889136
used_memory_human:605.48M
used_memory_rss:810725376
mem_fragmentation_ratio:1.28
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:11055
bgsave_in_progress:0
last_save_time:1314024370
bgrewriteaof_in_progress:0
total_connections_received:1410361443
total_commands_processed:620282294
expired_keys:9605312
evicted_keys:0
keyspace_hits:490248753
keyspace_misses:75359054
hash_max_zipmap_entries:512
hash_max_zipmap_value:64
pubsub_channels:0
pubsub_patterns:0
vm_enabled:1
role:master
vm_conf_max_memory:20000000000
vm_conf_page_size:64
vm_conf_pages:268435456
vm_stats_used_pages:0
vm_stats_swapped_objects:0
vm_stats_swappin_count:0
vm_stats_swappout_count:0
vm_stats_io_newjobs_len:0
vm_stats_io_processing_len:0
vm_stats_io_processed_len:0
vm_stats_io_active_threads:0
vm_stats_blocked_clients:0

[Greg Andrews]

I don't know if this is related to the problem you're experiencing, but your info command output had two lines that seem odd:

uptime_in_seconds:528386
total_connections_received:1410361443

That indicates an average of 2760 connections per second, which is a lot.

  -Greg

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Pavel Shevaev]

> That indicates an average of 2760 connections per second, which is a lot.

I'm also experiencing a somewhat similar behavior with redis-2.2.12 on
Centos 5.6. I see lots of errors from PHP backends which can't connect
to redis.

I graphed redis connections per second and commands per second during
such periods, here's the numbers:
* 230-250 conns/sec
* 2000-2700 cmds/sec.

How can I further diagnose the problem? Am I hitting some redis specific limits?

--
Best regards, Pavel

[Josiah Carlson]

You may be hitting system-level limits either incoming (to Redis, look into ulimit and/or the relevant Redis configuration option) or outgoing (your system isn't recycling closed network connections fast enough).

Both are likely caused by the fact that your code isn't re-using or pooling any of your Redis connections.

I don't know what it would take to do pooling using whatever client you are using.

 - Josiah

--

[Pavel Shevaev]

On Mon, Sep 12, 2011 at 10:32 PM, Josiah Carlson
<josiah....@gmail.com> wrote:
> You may be hitting system-level limits either incoming (to Redis, look into
> ulimit and/or the relevant Redis configuration option) or outgoing (your
> system isn't recycling closed network connections fast enough).
> Both are likely caused by the fact that your code isn't re-using or pooling
> any of your Redis connections.

> I don't know what it would take to do pooling using whatever client you are
> using.

You are right, I'm using non-persistent connections and it looks like
the redis client library(https://github.com/nicolasff/phpredis) I'm
using does support some sort pooling(via pconnect method). I'm going
to try it, thanks for the tip! I'm just wondering what are the
possible negative consequences of this...

--
Best regards, Pavel

[Josiah Carlson]

There are some potential corner cases with errors in parsing, etc., causing errors for later calls, but that would be a problem that others would have seen and reported.

Use pooling. It's the right answer, and it's the only way to make anything like this scale (mysql, postgresql, memcache, etc., all use connection pooling).

Regards,

 - Josiah

Best regards, Pavel

[Pavel Shevaev]

> Use pooling. It's the right answer, and it's the only way to make anything
> like this scale (mysql, postgresql, memcache, etc., all use connection
> pooling).

Yep, thanks again. I remember I was thinking about this before but
stumbled upon some MySQL related articles where folks claimed that
persistent connections were evil if used improperly.

--
Best regards, Pavel

[Josiah Carlson]

Toothpicks are evil if used improperly. So use them (and your connections to MySQL, Redis, etc., ) properly. I'm know there are articles about Redis and proper php connection pooling (and Google did not disappoint when I searched).

Regards,

 - Josiah

[Pavel Shevaev]

> Toothpicks are evil if used improperly. So use them (and your connections to
> MySQL, Redis, etc., ) properly.

Yep, you are right :) After switching to persistent redis connections
the server handles much heavier load spikes without any glitches.
Thanks again.

Best regards, Pavel