Re: Could not get a resource from the pool
Ingvar Bogdahn
Hmm, your use case is advanced, so, no, it's not exactly obvious :-)
It maybe a pool exhaustion.
Try the following:
- set the maxIdle higher, try 10, and also set the minIdle to some
minimum, say 5.
- you seem to appreciate high availability... but maybe you exagerate.
Use either testOnBorrow or testOnReturn (prefer the first). Why then
also test while idle? Beware that you are loosing one TTL for each
single test!
- there is some setting that helped me once, exhaustion policy
something.... yep: try setWhenExhaustedAction(WHEN_EXHAUSTED_GROW)
--> this overrides maxActive
you can also set it to when_exhausted_fail, this may help to determine
if the pool indeed is exhausted.
hth,
ingvar
2011/10/1 Paul <asm...@hotmail.co.uk>:
> Hi,
>
> I'm sure the answer to this must be obvious but I can't find it! I've
> created a JedisPool with the following config:
>
> JedisPoolConfig config = new JedisPoolConfig();
> config.testOnBorrow = true;
> config.maxActive = 100;
> config.maxIdle = 5;
> config.testOnReturn = true;
> config.testWhileIdle = true;
> config.numTestsPerEvictionRun = 10;
> config.timeBetweenEvictionRunsMillis = 60000;
> config.maxWait = 3000;
>
> I share the pool between up to 100 concurrent worker threads that are
> created using:
>
> ExecutorService executor = Executors.newFixedThreadPool(100);
> ...
> executor.execute(myWorker);
>
> I've made sure that clients are always returned using returnResource.
>
> After a few minutes of heavy load though I start to see this exception
> continuously:
>
> redis.clients.jedis.exceptions.JedisConnectionException: Could not
> get a resource from the pool
> at redis.clients.util.Pool.getResource(Pool.java:22)
> at com.seodeepdive.http.LinkHeadRequest.run(LinkHeadRequest.java:184)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Caused by: java.util.NoSuchElementException: Could not create a
> validated object, cause: ValidateObject failed
> at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:
> 1226)
> at redis.clients.util.Pool.getResource(Pool.java:20)
>
> If there's nothing really daft in my code snippets above I was
> wondering if JedisPool was somehow conflicting within the
> ThreadSafeClientConnManager that I'm using to manage a http connection
> pool...
>
> Thanks for any help,
> Paul.
>
Paul
Setting when_exhausted_fail helped me track the problem down (I
hope!). I've been running for a while now with no more problems.
I think the issue as that although I had enough capacity for my worker
threads I hadn't taken into account the main thread that also required
a redis connection occasionally. So in rare cases I needed a pool of
101 (100 workers + 1 main thread).
Thanks for your help,
Paul.
On Oct 1, 8:10 pm, Ingvar Bogdahn <ingvar.bogd...@googlemail.com>
wrote:
>
> Hmm, your use case is advanced, so, no, it's not exactly obvious :-)
>
> It maybe a pool exhaustion.
>
> Try the following:
> - set the maxIdle higher, try 10, and also set the minIdle to some
> minimum, say 5.
> - you seem to appreciate high availability... but maybe you exagerate.
> Use either testOnBorrow or testOnReturn (prefer the first). Why then
> also test while idle? Beware that you are loosing one TTL for each
> single test!
> - there is some setting that helped me once, exhaustion policy
> something.... yep: try setWhenExhaustedAction(WHEN_EXHAUSTED_GROW)
> --> this overrides maxActive
>
> you can also set it to when_exhausted_fail, this may help to determine
> if the pool indeed is exhausted.
>
> hth,
> ingvar
>
Paul
pool issue again. Although it's actually a different issue now, so
some progress has been made. I'm now getting:
redis.clients.jedis.exceptions.JedisConnectionException: Could not get
a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:22)
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:124)
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:54)
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1657)
at redis.clients.jedis.JedisPool
$JedisFactory.makeObject(JedisPool.java:63)
at
org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:
1181)
at redis.clients.util.Pool.getResource(Pool.java:20)
... 4 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:
213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at redis.clients.jedis.Connection.connect(Connection.java:119)
Paul.
Ingvar Bogdahn
Search the forum, there was some trick with the SocketTimeout problem,
some setting of the socket.
btw, I posted nonsense, it's RTT not TTL: You loose a lot of round
trip times for all thoses tests you are doing. Try to disable 2 of the
3 different ones, that should be enough anyway. This may lower the
probability to get pool exhaustion or socket timeouts.
Don't have much other idea, except tweaking the pool settings. Be sure
to have a thorough read of the apache javadoc of GenericObjectPool
good luck
2011/10/3 Paul <asm...@hotmail.co.uk>: