JedisPool connection leakage?

[pablochacin]

Hi all

I've been experiencing a problem on which my application hangs after some two or three days running.
I've taken a core dump of the tomcat (6.0.36)  and after analyzing it, I found some inconsistencies. From the 200 thread defined in Tomcat I found that

- 93 were waiting on the borrowObject method of Apache commons Generic Pool, called from the redis.clients.utils.Util.Pool.getResource() method.
- 52 were waiting on java.net.SocketInputStream.read called from  redis.clients.util.RedisInputStream.fill()  method
The rest of the threads were waiting for work.

Now, I have the pool defined with 500 connections and Redis has a connection limit (maxclients) of 10000, so how it is possible that 93 threads were waiting for a connection?

The answer came after analyzing the heap dump. The GenericObjectPool shows that there are 500 active connections (_numActive field), but there are only
38 instances of Jedis Client class. That is, somehow the pool has a wrong counter for active connections. More over, it there are 38 connections, how can be
52 threads waiting on a jedis connection socket?

From this analysis, it seams to me that the JedisPool (or more likely, the apache pool) is leaking connections.
Interestingly, I've found some old reports of similar problems with apache's jdbc connector based on this same object pool.

I's this a know issue? If so, what would you suggest me to do, as I can't change Jedis Implementation.

Regards

Pablo

[Sam Hendley]

The fact that you have only 38 connection instances but 52 threads blocking on socket calls makes me suspicious that somehow connections are being accidentally shared between threads. Please double check that you are not accidentally sharing a connection between multiple threads (possibly with a wrapper object to provide the correct semantics). Can you give us some psedo code of your threading and how you are using the pool?

Sam

--
You received this message because you are subscribed to the Google Groups "Jedis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jedis_redis...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Pablo Chacin]

Hi Sam

My mistake. There were only 13 threads waiting on the sockets, not 52 (I got the information making a grep on the thread dump I counted each thread 4 times, apparently). Sorry. By the way, they all were on subscribe command.

The 93 threads waiting on the borrowObject are correct.

So, the "only" problem I see is that the GenericObjectPool says there are 500 active connections, when there are only 38.

Pablo

[pablochacin]

It turned out to be a failure in our application. A infrequently used transaction escaped our radar :-(

Something that surprised me (and was the origin of our mis-leaded suspicions about the pool) is that there's no reference to active sessions in the pool. I was expecting that the pool kept a reference to allow recovery of idle connections and ease the detection of this kind of leakage.

thanks for the help