Cassandra Node is going down frequently and driver is showing NoHostAvailableException

[hemendra kumar]

We are getting very frequent alerts from opscenter that nodes are down but when we are checking through nodetool status it's showing all nodes are up. But application  is getting connection failure or NoHostAvailableException.

We observed below error from opscenter agent log: 
WARN [Thread-11] 2015-12-08 10:39:18,658 Cassandra operation queue is full, discarding cassandra operation 
WARN [Thread-11] 2015-12-08 10:39:18,658 12265 operations dropped so far. 
ERROR [cassandra-processor-5] 2015-12-08 10:39:20,461 Error when proccessing cassandra callcom.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried) 
ERROR [cassandra-processor-10] 2015-12-08 10:39:20,472 Error when proccessing cassandra callcom.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)

But we already have recommended settings in address.yaml file for handling this as below:

thrift_max_conns: 10 
async_pool_size: 10 
async_queue_size: 20000

Please provide your suggested solutions for this case.

Thanks,

Hemendra

[Andrew Tolbert]

Hi Hemendra,

Do you happen to know what version of OpsCenter you are using?   Older versions of the java driver can be pretty aggressive about marking hosts down.   The latest version of opscenter (5.2.2) includes java-driver 2.1.6, which includes changes that fixes the aggressive host down issues (JAVA-617 among others).

Thanks,

Andy

[hemendra kumar]

Hi Andrew,

We are using OpsCenter-5.1.0 and datastax-java-driver -2.1.5 . Is there any open issue with this java driver. Can we only upgrade java-driver to 2.1.6 without upgrading OpsCenter version. Please suggest.

Thanks,

Hemendra

[Andrew Tolbert]

So fortunately with 2.1.5 of the driver a change was reverted (JAVA-425) which makes the down detection less aggressive, however there is a bug that exists that was fixed in 2.1.6 (JAVA-617) which (in basic terms) would cause hosts to be marked down possibly much longer than they should.   Because of this I think you would have better success with OpsCenter 5.2.2.

Thanks,

Andy

[Andrew Tolbert]

Can we only upgrade java-driver to 2.1.6 without upgrading OpsCenter version. Please suggest.

I don't believe so since I think the OpsCenter agent jar is an 'uberjar' that contains all dependencies.   There might be a way to manipulate the classpath to put the java driver jar first so it is loaded ahead of the opscenter agent's embedded java driver code, but it's not something I've tried or would suggest. 

On Wednesday, December 9, 2015 at 7:42:53 AM UTC-6, hemendra kumar wrote: