com.datastax.driver.core.HostConnectionPool.awaitAvailableConnection blocks forever

[Senthil Kumar]

Hello Guys,

We are experiencing sluggishness in our cassandra queries.

Using JAVA jstack I did a thread dump of the process and noticed that there are 8 threads which seem permanently blocked 

on the awaitAvailableConnection call.

My understanding is that the block should only happen for a certain time till a connection frees up.

The server is not serving any requests (right now) what so ever, but I still see those blocked threads.

I would appreciate any insights into the matter.

cheers

Kumar

========

The thread dump is:

Thread 3191: (state = BLOCKED)

 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)

 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=175 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=1, line=836 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.util.concurrent.locks.AbstractQueuedSynchronizer$Node, int) @bci=67, line=870 (Compiled frame)

 - java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(int) @bci=17, line=1199 (Compiled frame)

 - java.util.concurrent.locks.ReentrantLock$FairSync.lock() @bci=2, line=224 (Compiled frame)

 - java.util.concurrent.locks.ReentrantLock.lock() @bci=4, line=285 (Compiled frame)

 - com.datastax.driver.core.HostConnectionPool.awaitAvailableConnection(long, java.util.concurrent.TimeUnit) @bci=4, line=296 (Compiled frame)

[Kevin Gallardo]

Hi Kumar,

Logically threads should be blocked by this call (HostConnectionPool.awaitAvailableConnection(long, java.util.concurrent.TimeUnit)) only when all connections to all Hosts are already full with pending requests. As explained here : http://datastax.github.io/java-driver/manual/pooling/#acquisition-timeout.

This block can be avoided by setting the poolTimeout to 0. In this case, rather than block, the driver will throw an Exception because it cannot send requests to Cassandra on any connection. 

Is there any chance you could give more information about how your pools are used when you observe those threads blocked ? Referring to this : http://datastax.github.io/java-driver/manual/pooling/#monitoring-and-tuning-the-pool. Also the number of "inFlight Requests" and the "maxRequestsPerConnection" you configured if you did configure it, would help. As well as the Java driver & Cassandra versions.

Hope that helps.

Cheers.

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

--

Kévin Gallardo.

kgdo.me

Software Engineer in Drivers and Tools Team, at DataStax.

[Senthil Kumar]

Kevin,

I really appreciate your insight!

I was amiss in saying that the threads were permanently blocked.

They certainly block for hours (so it seems).

Let me dig a bit further into the settings you mentioned.

cheers

Kumar