Issue 165 in redis: redis-server crashes with === ASSERTION FAILED === when connecting 10.000+ clients
re...@googlecode.com
Owner: ----
Labels: Type-Defect Priority-Medium
New issue 165 by henkpunt: redis-server crashes with === ASSERTION FAILED
=== when connecting 10.000+ clients
http://code.google.com/p/redis/issues/detail?id=165
What steps will reproduce the problem?
1. created script to open connections to redis, but not issue any commands
and stay connected
2. run the script to open more than 10.000 connections to redis
3. shortly after the 10.000th connecting client, redis crashes with
assertion failure:
25 Feb 15:04:31 . Accepted 127.0.0.1:58257
25 Feb 15:04:31 . Accepted 127.0.0.1:58258
25 Feb 15:04:31 . Accepted 127.0.0.1:58259
25 Feb 15:04:31 . Accepted 127.0.0.1:58260
25 Feb 15:04:31 . Accepted 127.0.0.1:58261
25 Feb 15:04:31 . Accepted 127.0.0.1:58262
25 Feb 15:04:31 . Accepted 127.0.0.1:58263
25 Feb 15:04:31 * === ASSERTION FAILED ===
25 Feb 15:04:31 * ==> ln != NULL
25 Feb 15:04:31 * (forcing SIGSEGV in order to print the stack trace)
25 Feb 15:04:31 * ======= Ooops! Redis 1.2.2 got signal: -11- =======
25 Feb 15:04:31 * redis_version:1.2.2
arch_bits:64
multiplexing_api:epoll
uptime_in_seconds:3
uptime_in_days:0
connected_clients:10235
connected_slaves:0
used_memory:3331241
used_memory_human:3.18M
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1267106668
bgrewriteaof_in_progress:0
total_connections_received:10235
total_commands_processed:0
role:master
25 Feb 15:04:31 * 1 redis-server 0x7f55a3faa040 keysCommand + -1548128208
25 Feb 15:04:31 * 2 redis-server 0x40926a _redisAssert + 58
25 Feb 15:04:31 * 3 redis-server 0x40df02 freeClient + 274
25 Feb 15:04:31 * 4 redis-server 0x40e01a createClient + 266
25 Feb 15:04:31 * 5 redis-server 0x41223a acceptHandler + 90
25 Feb 15:04:31 * ./redis-server(aeProcessEvents+0x195) [0x404325]
25 Feb 15:04:31 * ./redis-server(aeMain+0x1d) [0x40450d]
25 Feb 15:04:31 * ./redis-server(main+0xc2) [0x412812]
25 Feb 15:04:31 * /lib/libc.so.6(__libc_start_main+0xe6) [0x7f55a3f955a6]
25 Feb 15:04:31 * ./redis-server [0x403889]
this was by starting redis without any explicit configuration.
There is no apparent limit to the number of clients redis can handle from
the configuration file. maybe there is some hardlimit defined in the code?,
anyway crashing without any feedback on this limitation is not so nice...
I also checked that it was not a limitation of my Linux tcp config (such as
available ephemeral ports or some such), but using the following script I
was able to connect up to 60.000 clients on the same machine...
from concurrence import dispatch, Tasklet
from concurrence.io import Socket
N = 20000
def server():
clients = []
server = Socket.server(('localhost', 6379))
for i, client_socket in enumerate(server.accept_iter()):
print 'client accepted', i
clients.append(client_socket)
def client():
clients = []
for i in range(N):
print i
s = Socket.connect(('localhost', 6379))
clients.append(s)
print 'done', N
Tasklet.sleep(1000)
import sys
if sys.argv[1] == 'server':
dispatch(server)
else:
dispatch(client)
b.t.w. the client part was what i used to crash Redis.
--
You received this message because you are listed in the owner
or CC fields of this issue, or because you starred this issue.
You may adjust your issue notification preferences at:
http://code.google.com/hosting/settings
re...@googlecode.com
Comment #1 on issue 165 by henkpunt: redis-server crashes with ===
http://code.google.com/p/redis/issues/detail?id=165
i found the limitation it was in
ae.h:#define AE_SETSIZE (1024*10)
redefining it as 1024*64 let me have our much higher number of clients.
A better error message would be nice but is also unsafe, if people are
unaware of
this limitation and just compile the source, than it would be quite easy
for a rogue
client to crash the redis-server by just opening more than AE_SETSIZE
connections.
Redis probably just should refuse connections when the total is > than some
configurable amount and set AE_SETSIZE accordingly.
re...@googlecode.com
Status: Accepted
Comment #2 on issue 165 by antirez: redis-server crashes with === ASSERTION
FAILED === when connecting 10.000+ clients
http://code.google.com/p/redis/issues/detail?id=165
Hello Henkpunt,
indeed, there is a problem here, I think as easy fix is to initialize the
Redis
configuration so that the max-clients directive is automatically set to
AE_SETSIZE -
<fixed amount>, where the fixed amount is used in order to account for the
few file
descriptors opened.
When the configuration parser detects a max-clients directive that is > of
this
number, it can just exit with the error "to use more clients than N please
edit
AE_SETSIZE" or something alike.
Do you this this is acceptable?
re...@googlecode.com
Comment #3 on issue 165 by jaksprats: redis-server crashes with ===
http://code.google.com/p/redis/issues/detail?id=165
I got a similar bug at 26,000 concurrent requests w/ 2.0.0 stable
http://allinram.info/redis/concurrency/SEGV shows the bug ...
I edited AE_SETSIZE but still got the problem.
At 26,000 concurrent requests, this may be a linux configuration issue ..
any help would be appreciated
re...@googlecode.com
Comment #4 on issue 165 by pcnoordhuis: redis-server crashes with ===
http://code.google.com/p/redis/issues/detail?id=165
The cause of the assertion error is now fixed in both 2.0 and master. It
was caused by Redis not being able to add an event listener to the event
loop when this many clients are connected. The fix makes sure that the
connection is closed instead of crashing. This way, if Redis cannot accept
more connections, new incoming connections are simply dropped.
Leaving this ticket open because we might address setting AE_SETSIZE
dynamically.