Issue 500 in redis: connection timeout

[re...@googlecode.com]

Status: New
Owner: ----
Labels: Type-Defect Priority-Medium

New issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

What version of Redis you are using, in what kind of Operating System?
2.2
What is the problem you are experiencing?
We have a redis server running with our webserver (which is in same LAN)
connects to redis using phpredis module. At normal condition it works fast,
when there are more than 100 connections open on redis port we have found
network timeouts randomly. I have put two scripts on webserver for testing

1. Php script which connect to tcp port 6379 every 10 secs and log when
connection cant establish on 5 secs.
2. php script which connects using predis client and send ping to the redis
server, this also run every 10 secs.

Both scripts timeout almost at same time. There is nothing in the logs. Can
you please provide some steps to pin point the exact issue here?

What steps will reproduce the problem?

Do you have an INFO output? Please past it here.

No

If it is a crash, can you please paste the stack trace that you can find in
the log file or on standard output? This is really useful for us!

Please provide any additional information below.

-Vij

[re...@googlecode.com]

Comment #1 on issue 500 by pcnoordh...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

A number of issues can cause this. For instance, Redis is overloaded and
cannot server newly connected clients at that moment (which is why the INFO
output is helpful here). It might as well be a PHP issue, or an issue
related to your network configuration (is the Redis host itself
reachable?). Another cause might be: Redis is configured with a very low
timeout, and closes the connections when there is no activity (this needs
to be explicitly configured, the default timeout is 300s), although
disconnecting timed out clients shows up in the log.

[re...@googlecode.com]

Comment #2 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Thanks. redis server itself is reachable it seems.
Can you give me some troubleshooting steps? Can we set the loglevel to
debug on the fly?
We are using the default timeout only, and every script is opening new
connection sending commands and then closing it.

-Vij

[re...@googlecode.com]

Comment #3 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Also, we are sending only 100s of requests per sec..not more. As per redis
docs server ready to handle more than 100k's of rquests

-vij

[re...@googlecode.com]

Comment #4 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

I have tried to run tcpdump to log the packet, when the connections were
dropped, server didnt send ack to the client. There is nothing else in the
dump file.
What may be the reason? Will it be in Linux side or in redis itself.

Thanks for your help

-Vij

[re...@googlecode.com]

Comment #5 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Tried to log the number of tcp connections on port 6379 every minute, and
when the issue happens always the connection count is more than 100
At normal times its less than 10

-Vij

[re...@googlecode.com]

Comment #6 on issue 500 by pcnoordh...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Maybe the PHP process is running out of file descriptors. Could you check
the fd limit for both the processes running Redis and PHP (ulimit -n)?

[re...@googlecode.com]

Comment #7 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

hmm..yea it is set to 1024 .. i am logging the total number of redis
connections
every minute and it looks like there were instances when there are 700+
open connections(using netstat) . I think it had hit the 1000 limits and
passed these errors.
I have set the timeout value to 300, in my scripts am not using same
connections for multiple operations. Value to 30 will be fine in that case?

-Vij

[re...@googlecode.com]

Comment #8 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Looks like issue is not due to the filedescriptior, i have logged number of
open files every 10 secs and it never reached limit of 1025

Can you please help me to find out what is the issue? I have tried to
analyze the tcpdump and found packets are not lost in network, server just
didnt sent ack for the new requests when issue happened. Please let me know
if u need more info ,

Thanks

-Vij

[re...@googlecode.com]

Comment #9 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

issue happens when the there are more than 100 established connections to
the redis server

[re...@googlecode.com]

Comment #10 on issue 500 by vijeeshk...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

redis_version:2.2.0
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:24829
uptime_in_seconds:249837
uptime_in_days:2
lru_clock:142324
used_cpu_sys:9589.27
used_cpu_user:2645.72
used_cpu_sys_childrens:731.97
used_cpu_user_childrens:134.06
connected_clients:1
connected_slaves:1
client_longest_output_list:0
client_biggest_input_buf:0
blocked_clients:0
used_memory:2235368128
used_memory_human:2.08G
used_memory_rss:3167801344
mem_fragmentation_ratio:1.42
use_tcmalloc:0
loading:0
aof_enabled:0
changes_since_last_save:2241
bgsave_in_progress:0
last_save_time:1301657468
bgrewriteaof_in_progress:0
total_connections_received:13739616
total_commands_processed:89080005
expired_keys:0
evicted_keys:0
keyspace_hits:70052629
keyspace_misses:5922576
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master

here is redis info o/p

[vij]

Sorry for opening an old thread again. i think it would be easier for
you to get the details of the issue than reading new thread for same
topic.

Our redis server issue which i described above happning again, earlier
it was causing due to an sunion of few one million record sets. We
have removed it and issue got fixed. But last few days we get the
same issue again. ie, redis failed to respond. it is only one or two
secs, and happeneed twice this week. Scripts are not able to make
connection in 3 secs and throws exception. We use php-redis module as
the client for redis. Wierd thing is, the connection fails when there
are 200+ open connections in the network to the redis port from the
webservers. I can understand the scripts may halt if there is a heavy
query being processed by redis and further reqiuests are in the queue.
But cant understand why it stop accepting network connections after
more than 200 open connections. Can you please suggest some ideas to
find the bottle necks . Thanks for your time

-Vij



On Apr 1, 4:33 pm, re...@googlecode.com wrote:
> Comment #10 on issue500by vijeeshk...@gmail.com: connection timeouthttp://code.google.com/p/redis/issues/detail?id=500

[re...@googlecode.com]

Comment #11 on issue 500 by pavelbar...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

we experience the same issue - with 2.4.6 redis

[re...@googlecode.com]

Comment #12 on issue 500 by g...@healthwarehouse.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

we also experience this problem. Our PHP application report
some "connection timeout" error.

The redis version is 2.4.10. Installed on Ubuntu 11.10 (oneiric) with 1.7Gb
of RAM. CPU load average is very low and there still plenty of free RAM.

[re...@googlecode.com]

Comment #13 on issue 500 by ad...@ktbt.ru: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Same proplem here...

redis-cli -a "*****" info
# Server
redis_version:2.5.11
redis_git_sha1:00000000
redis_git_dirty:0
os:Linux 2.6.32-220.el6.x86_64 x86_64
arch_bits:64
multiplexing_api:epoll
gcc_version:4.4.6
process_id:10664
run_id:0da5f47fff19894c203b8a1ce26eb6e2de2e8f48
tcp_port:6379
uptime_in_seconds:1469
uptime_in_days:0
lru_clock:1030772

# Clients
connected_clients:85
client_longest_output_list:79
client_biggest_input_buf:0
blocked_clients:0

# Memory
used_memory:1712614032
used_memory_human:1.59G
used_memory_rss:1764986880
used_memory_peak:1719926112
used_memory_peak_human:1.60G
used_memory_lua:30720
mem_fragmentation_ratio:1.03
mem_allocator:jemalloc-3.0.0

# Persistence
loading:0
rdb_changes_since_last_save:14274
rdb_bgsave_in_progress:0
rdb_last_save_time:1352485004
rdb_last_bgsave_status:ok
rdb_last_bgsave_time_sec:28
rdb_current_bgsave_time_sec:-1
aof_enabled:0
aof_rewrite_in_progress:0
aof_rewrite_scheduled:0
aof_last_rewrite_time_sec:-1
aof_current_rewrite_time_sec:-1

# Stats
total_connections_received:390790
total_commands_processed:8306950
instantaneous_ops_per_sec:6138
rejected_connections:0
expired_keys:197
evicted_keys:0
keyspace_hits:1431736
keyspace_misses:5722620
pubsub_channels:0
pubsub_patterns:0
latest_fork_usec:29199

# Replication
role:master
connected_slaves:1
slave0:94.75.227.108,43678,send_bulk

# CPU
used_cpu_sys:537.10
used_cpu_user:207.88
used_cpu_sys_children:126.56
used_cpu_user_children:393.94

# Keyspace
db0:keys=1925581,expires=1




So what can i do now?

[re...@googlecode.com]

Comment #14 on issue 500 by morganmc...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

I also see this issue with my redis instances hosted on ec2 a lot.

redis_version: 2.4.17
arch_bits: 64
multiplexing_api: epoll
gcc_version: 4.4.3
uptime_in_seconds: 1645462
uptime_in_days: 19
lru_clock: 1843244
used_cpu_sys: 76865.59
used_cpu_user: 23388.94
used_cpu_sys_children: 675.00
used_cpu_user_children: 1701.44
connected_clients: 302
connected_slaves: 0
client_longest_output_list: 2
client_biggest_input_buf: 0
blocked_clients: 3
used_memory: 684858696
used_memory_human: 653.13M
used_memory_rss: 860618752
mem_fragmentation_ratio: 1.26
mem_allocator: jemalloc-3.0.0
loading: 0
aof_enabled: 1
changes_since_last_save: 22412973
bgsave_in_progress: 0
last_save_time: 1360575666
bgrewriteaof_in_progress: 0
expired_keys: 339809
evicted_keys: 0
keyspace_hits: 359020680
keyspace_misses: 139879596
pubsub_channels: 0
pubsub_patterns: 0
latest_fork_usec: 126223
vm_enabled: 0
role: master
aof_current_size: 823029796
aof_base_size: 600604986
aof_pending_rewrite: 0
aof_buffer_length: 0
aof_pending_bio_fsync: 0
db0: keys=235044,expires=233870


Looking forward to a solution.
Please let me know what to look for in the output of info.

From,
Morgan

[Yiftach Shoolman]

For the EC2 case, if you are accessing the private-ip of your Redis instance (assuming a different instance than your app instance), and that instance is located on a different availability-zone (in the same region) --> there is a well know issue with AWS network regarding that (it's just the largest LAN in the world..). 

To solve it, just access the public-ip of your instance

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.

To post to this group, send email to redi...@googlegroups.com.

Visit this group at http://groups.google.com/group/redis-db?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.

--

Yiftach Shoolman
+972-54-7634621

[Felix Gallo]

Just don't forget to firewall your redis port very defensively if you ever take this advice...

F.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

[Yiftach Shoolman]

+1

--

Yiftach Shoolman
+972-54-7634621

[re...@googlecode.com]

Comment #15 on issue 500 by migu...@gmail.com: connection timeout
http://code.google.com/p/redis/issues/detail?id=500

Hello, Im having this issue in my website too, specially with high traffic
peaks (+7k online users). Apache error.log shows this:

PHP Fatal error: Uncaught
exception 'Predis\\Connection\\ConnectionException' with
message 'Connection timed out [tcp://127.0.0.1:6379]'

Everything is working fine (php, mysql, nginx...), but this error hangs the
page with error 500.

--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings