NoHostAvailableException: All host(s) tried for query failed (no host was tried)

[Ankush Goyal]

Hi Guys,

We recently (this week) upgraded our java driver from 2.0.4 to 2.1.4 (we are running Cassandra version 2.0.12). We were using 2.0.4 for a long time (more than an year) in production and did not have this issue.
But, after upgrading we starting seeing intermittent  NoHostAvailableExceptions (that fail all the requests on client side):

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
	at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) ~[stormjar.jar:na]
	at com.datastax.driver.core.SessionManager.execute(SessionManager.java:530) ~[stormjar.jar:na]
	at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:566) ~[stormjar.jar:na]
	at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:119) ~[stormjar.jar:na]

We are having this issue in production, so would love to get some advice on how to triage this. What we know for sure:
* When errors start happening Cassandra cluster isn't under unusual load.
* We are able to reach Cassandra hosts (cluster) from client side - Not a connection issue.

Cheers,
-Ankush

[Vishy Kasaravalli]

What LoadBalancingPolicy are you using? Does a rolling bounce of your clients fix the issue? If so, java-driver has built up some bad state. 

Set this on one of your clients

log4j.logger.com.datastax.driver.core=TRACE

That will give valuable information on what may be happening. 

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Jeff Wehrwein]

We've seen this too, and I don't fully understand it but I think it's related to this: https://datastax-oss.atlassian.net/browse/JAVA-425

We turned the driver logging to DEBUG [1] and I see this log line right before the "no host was tried" exception:

2015-02-27 16:47:11,636 Hashed wheel timer #1 DEBUG com.datastax.driver.core.Connection - Defuncting connection to /x.x.x.x:9042
com.datastax.driver.core.exceptions.DriverException: Timed out waiting for server response
        at com.datastax.driver.core.RequestHandler.onTimeout(RequestHandler.java:570)
        at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:893)
        at com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)
        at com.datastax.shaded.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)
        at com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)
        at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at java.lang.Thread.run(Thread.java:745) 

Then there are hundreds of these lines where the driver seems to be closing perfectly good connections:

2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98290, inFlight=0, closed=true] closing connection
2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98290, inFlight=0, closed=true] closing connection
2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98291, inFlight=0, closed=true] closing connection
2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98291, inFlight=0, closed=true] closing connection
2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98292, inFlight=0, closed=true] closing connection
2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98292, inFlight=0, closed=true] closing connection
2015-02-27 16:47:11,838 Cassandra Java Driver worker-1311 DEBUG com.datastax.driver.core.Connection - Connection[/x.x.x.x:9042-98293, inFlight=0, closed=true] closing connection

And then the "no host was tried", all within 300 ms of the original timeout:

2015-02-27 16:47:11,868 ...

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
        at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)

        at com.datastax.driver.core.SessionManager.execute(SessionManager.java:530)
        at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:566)
        at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:119)

So I think a query is timing out on the driver side for some reason, despite the server apparently being fine.  Then the driver marks the server as down (as explained in JAVA-425), and thus the next query fails because the driver thinks all hosts are down.  What I can't figure out is why the query times out on the driver instead of on the server - both our driver [2] and server [3] timeouts are left as default, so the server should time out first.

We're using driver 2.1.4 and Cassandra 2.0.12.

Jeff

[1] http://www.datastax.com/documentation/developer/java-driver/2.1/java-driver/reference/debugLevelLogging.html

[2] 12 s, see SocketOptions.setReadTimeout()

[3] in cassandra.yaml:

read_request_timeout_in_ms: 5000

range_request_timeout_in_ms: 10000

write_request_timeout_in_ms: 2000

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Ankush Goyal]

Thanks Vishy.

We are using default LoadBalancing policy: DCAwareRoundRobinPolicy

Bouncing clients does not work. I think this is similar to what Jeff is experiencing..

[Ankush Goyal]

Thanks for sharing Jeff. Its helpful to know we are not alone :)
Will post if we find a cure for it.

[Ankush Goyal]

Hi Guys,

Wanted to share our findings:

It turned out that it was related to the query pattern in that particular client. We found that in certain cases that client would try to write a HUGE batch, which would not complete under driver side default timeout settings. That would cause driver to think that the node is down, and eventually it will end up marking all nodes as down.

Setting a limit on the batch size (by chunking) and using UNLOGGED batches seems to have solved the issue.

[Rick Bullotta]

Seems like a design flaw though, doesn't it?  A timeout is not in and of itself an accurate indication of a host down.  

[Sylvain Lebresne]

If you get drivers timeouts (not to be confused with cassandra/server timeouts) for large batches, this strongly suggests that you've set the driver timeout lower than your server timeout (either you've lower the driver timeout from its default, or you've bumped the server one without bumping the driver side one), which, as the javadoc indicates (http://www.datastax.com/drivers/java/2.1/com/datastax/driver/core/SocketOptions.html#setReadTimeoutMillis(int)), would be a mistake.

--

Sylvain

[Ankush Goyal]

Sylvian,

We use the default driver timeout (12 seconds for reads), and default on server side (5 seconds).

Not sure why we still saw this issue.

[Sylvain Lebresne]

Yeah, that is weird, especially since C* reject big batches right off the bat by default (more precisely any batch that is bigger than the batch_size_fail_threshold_in_kb which is 50kb by default). Only reason I can think of would be that your batches were big enough that C* wasn't even able to deserialize it properly within the driver side timeout, which is possible I suppose though kind of feel extreme. In any case, chunking your batches is the proper thing to do here, C* is not good with big batches (which is why there is a setting to reject them by default).

--

Sylvain

[Jeff Wehrwein]

Sylvain,

We're seeing the same behavior as Ankush (driver timeouts instead of server timeouts), but we're not using huge batches.  In the driver log we've seen the following error 84,000 times in the last 18 hours on one server that does a lot of C* queries:

2015-03-03 17:54:49,188 Hashed wheel timer #1 DEBUG com.datastax.driver.core.Connection - Defuncting connection to /x.x.x.x:9042

com.datastax.driver.core.exceptions.DriverException: Timed out waiting for server response
        at com.datastax.driver.core.RequestHandler.onTimeout(RequestHandler.java:570)
        at com.datastax.driver.core.Connection$ResponseHandler$1.run(Connection.java:893)
        at com.datastax.shaded.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:546)
        at com.datastax.shaded.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:446)
        at com.datastax.shaded.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:395)
        at com.datastax.shaded.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at java.lang.Thread.run(Thread.java:745) 

C* reject big batches right off the bat by default (more precisely any batch that is bigger than the batch_size_fail_threshold_in_kb which is 50kb by default)

I don't see anything about this setting in C* 2.0 (in the default config file or documentation).  There's batch_size_warn_threshold_in_kb, which defaults to 5kb, but is only a warning.  We get a few such warnings (a few hundred over those same 18 hours), but we're not far over the 5kb limit.

I checked our server log, which has default logging settings, and there are no ERRORs or WARNs that point to this problem.

It's not always completely clear which query caused the driver timeouts, but one suspect is this:

SELECT id, lastUpdatedAt FROM myTable WHERE id IN (?)

with a list of ~500-1000 ids, on this table:

CREATE TABLE myTable (
  id text,
  data text,
  lastupdatedat timestamp,
  PRIMARY KEY ((id))
)

I'm gathering from the documentation that we should probably be doing this query differently, but regardless the question remains: why does the driver time out instead of the server?

Thanks!

Jeff

[Andrew Tolbert]

Hi Jeff,

It's not always completely clear which query caused the driver timeouts, but one suspect is this:

with a list of ~500-1000 ids, on this table:

  id text,
  data text,
  lastupdatedat timestamp,
  PRIMARY KEY ((id))
)
I'm gathering from the documentation that we should probably be doing this query differently, but regardless the question remains: why does the driver time out instead of the server?

SELECT id, lastUpdatedAt FROM myTable WHERE id IN (?)

CREATE TABLE myTable (

Because id is the partition key, the coordinator node will make a single partition query for each of ~500-1000 partitions and not return until they've all been queried.   read_request_timeout_in_ms is the amount of time the coordinator will wait for a response from a replica (i.e. response to a single partition query).   Given that you are querying many partitions it sounds possible that this can take longer than 12 seconds (default driver read timeout) to get responses for all ids which would cause the driver timeouts to occur.

Since the driver supports many requests on a connection, it would be a good improvement to update your code to use a prepared statement and query each id individually.   This article discusses this topic under the "Case study: multi-partition query, a.k.a. “client-side SELECT...IN“" section.

Thanks,

Andy

[Jeff Wehrwein]

Hi Andy,

Thanks for your reply.  We've already made the change to the query (and we were already using prepared statements), but it still seems like an underlying design flaw in the driver or server code.  The driver feature implemented in JAVA-425 depends on the server timing out in its configured time, yet we've now discovered two different situations where it does not: Ankush's case with a large batch, and our case with a lot of coordinator work.  If driver timing out before server happens because of slow queries, then that's not an accurate way to conclude that the server is down.  So either the server should actually time out in the time specified, or the driver shouldn't assume that it does.

But given how it does work currently, it seems like the recommendation is "don't write slow queries".  While we certainly strive for that, we're still learning, and it's frustrating that the consequence of a mistake is that all queries fail.  Is there a better way to self-monitor for slow queries like this, so that the errors don't cascade into these opaque "no host was tried" errors that are difficult to trace?  Would it be advisable to increase the driver read timeout to something larger, so that slow queries don't blow up the whole driver?

Thanks!

Jeff

[Andrew Tolbert]

Hi Jeff,

But given how it does work currently, it seems like the recommendation is "don't write slow queries".  While we certainly strive for that, we're still learning, and it's frustrating that the consequence of a mistake is that all queries fail.  

I agree that this is not entirely user friendly.  It may not be immediately obvious why no hosts are available or why hosts are marked as down.  While a timeout exception will be thrown or set on the future of the request, it may not be obvious from monitoring that there are queries that are timing out (depending how you are handling exceptions in your code).  Maybe we could make it more obvious that the node is being marked down because the query timeout.  The term 'defuncting Connection' may not register to the user that the host is being marked down.

Is there a better way to self-monitor for slow queries like this, so that the errors don't cascade into these opaque "no host was tried" errors that are difficult to trace?  

Since an exception indicating the timeout occurred is thrown, you can explicitly check for that in your code.  In addition measuring response time of your queries might be a good step as well, for example whenever you receive a response to a request that took longer than 1 second (or whatever your desired ceiling) you log it.  This may help you proactively identify issues sooner.

Would it be advisable to increase the driver read timeout to something larger, so that slow queries don't blow up the whole driver?

I think you should always be mindful of what your queries are and how long they might take.  Large batch queries and selects with multiple partition keys will always be an order of magnitude slower then their single partition counterparts.   I would say "don't write slow queries" is better advice then "raise your read timeout".  Monitoring your response times as I suggested would be a good way of catching this before it becomes a bigger problem.  Setting a large read timeout could make the driver less susceptible to marking a host down, causing it to send requests to a host that is unresponsive, which will affect response latencies in your application.

Thanks,

Andy

[Rick Bullotta]

Forgive my ignorance, but are timeouts on a per-query/per-request basis?  If not, they definitely should be…

[Ankush Goyal]

Hi Guys,

My earlier post saying our exception got resolved after chunking up writes, was a false positive. We didn't see anything for 3 days (2 days were weekend, so less traffic). But, we are still seeing a lot of intermittent NoHostAvailableException (No host was tried).

At the moment, we are thinking of reverting back to earlier version (2.0.3 or 2.0.4) of driver that had other issues but at least wasn't marking all the hosts as down.

Would definitely like to get some feedback from datastax driver team regarding this (Seems like a serious bug). Since, as far as we can tell, we are following all the prescribed patterns.
Unfortunately we can't turn on DEBUG mode in prod due to excessive amount of logs driver would generate.

[Jeff Wehrwein]

Would definitely like to get some feedback from datastax driver team regarding this (Seems like a serious bug). Since, as far as we can tell, we are following all the prescribed patterns. 

+1.  Seems like a bad case of theory getting in the way of practice.

 

[Thomas Arnaud]

Hi,

We are having the same problem.

Could this be a threading issue ?

We have an application with a single session and some prepared statements that are shared between hundreds of threads.

At startup everything seems to be fine but after few minutes a few timeout start to appear and suddenly this "All host(s) tried for query failed (no host was tried)" is raised.

If we change the number of threads, few timeout appears much later.

And if there is a single thread, the issue never rises.

--

Thomas

[Olivier Michallat]

Ankush,

You mentioned chunking your batches. How many statements to they contain now? Are these statements on different partition keys?

--

Olivier Michallat

Driver & tools engineer, DataStax

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Olivier Michallat]

Thomas,

Have you customized any of the following:

- SocketOptions.setReadTimeoutMillis() on the driver side?

- *_timeout_in_ms in cassandra.yaml?

Do you have statements that use IN queries with a large number of partition keys?

--

Olivier Michallat

Driver & tools engineer, DataStax

[Tobias Hauth]

Hi Olivier,

I think the problem here is that no matter how bad the queries are this shouldn't cause the driver to mark a node as down. If other queries still work fine this is just the wrong behavior. The server should timeout before the client does. We have at least two cases now that show that this is not working. Are there any plans to fix the driver?

Thanks,

Tobias

[Thomas Arnaud]

Olivier,

We use default SocketOptions configuration.

Server side, the read and the counter_write timeouts have been increased to 10 seconds.

But we were trying tu tune server before this problem happened.

row_cache_size_in_mb: 2000 (was previously default value) + row cache propertiy update on a single table.
counter_cache_size_in_mb: 500 (was previously default value)
concurrent_reads: 64 (was previously 32)

The server have enough memory to hold such a configuration.
And according to comments in cassandra.yaml, the concurrent_reads could/should be much greater than 32 in our configuration (12 cores).

We tried to rollbacked and it seems ok now.
I don't understand why those settings brings timeouts.
And i don't understand why driver finally consider nodes down whereas they are all up and i have no timeout issue with cqlsh.

-- Thomas

[Fabrice Larcher]

Hello,

Here are some information about the issue that Thomas mentioned.

Summary : We use C* 2.1.0 and the Java driver 2.1.4. We have 5 nodes and a replication factor of 3. We recently added more threads and we experience the loss of connection earlier and more often.

Configuration : We rollbacked our modifications of 'row_cache_size_in_mb', 'counter_cache_size_in_mb', 'concurrent_reads' and use default values but the issue is still there. We also use default values of 'SocketOptions.setReadTimeoutMillis()' on the driver side and '*_timeout_in_ms' in 'cassandra.yaml'. We do not use any IN statement.

Here are some logs of the driver that seems to be correlated with our issue (we activated the logs at the TRACE level temporarily) :

[Rawdata Analyzer #143 [1]] TRACE 2015-03-08 15:37:08,624 [com.datastax.driver.core.Connection] - <Connection[/node1:9042-3, inFlight=8687, closed=false] writing request EXECUTE 0xbdeecae78a59b65047e4113fae87089b ([cl=LOCAL_QUORUM, vals=[java.nio.HeapByteBuffer[pos=0 lim=66 cap=66], java.nio.HeapByteBuffer[pos=0 lim=16 cap=16], java.nio.HeapByteBuffer[pos=0 lim=8 cap=8]], skip=false, psize=5000, state=null, serialCl=SERIAL])>

[Hashed wheel timer #1] DEBUG 2015-03-08 15:37:10,481 [com.datastax.driver.core.Connection] - <Defuncting connection to /node1:9042>

com.datastax.driver.core.exceptions.DriverException: Timed out waiting for server response

[Hashed wheel timer #1] DEBUG 2015-03-08 15:37:10,481 [com.datastax.driver.core.Cluster] - <Host /node1:9042 is Suspected>

[Hashed wheel timer #1] DEBUG 2015-03-08 15:37:10,482 [com.datastax.driver.core.Connection] - <Connection[/node1:9042-3, inFlight=8687, closed=true] closing connection>

[Hashed wheel timer #1] DEBUG 2015-03-08 15:37:10,482 [com.datastax.driver.core.Connection] - <Not terminating Connection[/node1:9042-3, inFlight=8687, closed=true]: there are still pending requests>

[Cassandra Java Driver worker-12] TRACE 2015-03-08 15:37:10,483 [com.datastax.driver.core.Connection] - <Connection[/node1:9042-5, inFlight=0, closed=false] Connection opened successfully>

[Cassandra Java Driver worker-12] TRACE 2015-03-08 15:37:10,483 [com.datastax.driver.core.Connection] - <Connection[/node1:9042-5, inFlight=0, closed=false] writing request STARTUP {CQL_VERSION=3.0.0}>

[New I/O worker #24] TRACE 2015-03-08 15:37:10,483 [com.datastax.driver.core.Connection] - <Connection[/node1:9042-5, inFlight=0, closed=false] request sent successfully>

[Hashed wheel timer #1] DEBUG 2015-03-08 15:37:10,489 [com.datastax.driver.core.RequestHandler] - <Error querying /node1:9042, trying next host (error is: com.datastax.driver.core.TransportException: [/node1:9042] Connection has been closed)>

[Hashed wheel timer #1] DEBUG 2015-03-08 15:37:10,492 [com.datastax.driver.core.RequestHandler] - <Error querying /node1:9042, trying next host (error is: com.datastax.driver.core.TransportException: [/node1:9042] Connection has been closed)>

...

Then we get "com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)". However, the corresponding node(s) still answer correctly with CQLSH and JMX shows non-zero values for 'connected-to' and 'open-connections'. After a restart of our application, the driver finds again connections to all nodes. 

Would you have any advice for us ? If there is a known-fix, could you please send me the patch ?

PS : By the way (even if not related with that issue), we experience some NullPointerException from the driver (the stacktrace is attached to the email). Should I create a JIRA issue ?

Many thanks,

Fabrice

[Andrew Tolbert]

Hi Fabrice and Thomas,

With regards to the timeouts you are seeing, are you monitoring the performance of your cassandra nodes?   What does the output of 'nodetool cfhistograms <keyspace> <table>' look like?  It would be interesting to see what your read/write latencies look like.  You could also look at the 'requests' timer metrics in JMX to see what kind of latencies you are observing there as well.   As you mention, when you increase the thread count of your application you encounter this issue earlier and more often, I'm wondering if you are possibly overloading your cluster in any way?  If you have requests that are consistently timing out, this would make it such that to the driver it appears all hosts are down, while you could create a separate session (i.e. using cqlsh) while this is happening and see seemingly good responsiveness.

PS : By the way (even if not related with that issue), we experience some NullPointerException from the driver (the stacktrace is attached to the email). Should I create a JIRA issue ?

It looks like what may be happening here is that the metadata associated with the PreparedStatement of your query is null.  I'm not exactly sure how this could possibly happen.    There is an assertion at ArrayBackedResultSet#67 to ensure the defs aren't null (but that will only exercised if assertions are enabled in your JVM via -ea).   Are you able to consistently produce this problem?   It might be nice to get a tcpdump to see if that helps understand or to get some examples of the kinds of queries you are making (and how you are making them, i.e. are you using prepared statements like i'm assuming?)

Thanks,

Andy

[Fabrice Larcher]

Hi,

Monitoring outputs :

Here is an output for a table (on which we noticed slow SELECTs from the point of view of our application) :

Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count

                              (micros)          (micros)           (bytes)                  

50%             0.00             12.80             57.39               103                 1

75%             0.00             17.70             67.25               103                 1

95%             0.00             29.26             89.21               103                 1

98%             0.00             54.38            201.29               446                 6

99%             0.00            106.34            359.78               642                10

Min             0.00              2.69             12.53                87                 0

Max             1.00         730227.05        3066389.24          17436917            263210

For another table, I get : 

nodetool: Unable to compute when histogram overflowed

(are we hitting the upper limit ?)

The JMX measures on driver side show higher latencies (please see jconsole screenchot attached). The maximum time is about 23 seconds. 

The C* nodes and the application servers are on the same hardware. We do not notice high network throughput. 

Configuration changes

Indeed, we put our C* under heavy load, and timeouts probably put our C* node down (from the driver point of view). I did not understand why the C* nodes could not come back soon after the requests' timeouts. I though at first that we were facing JAVA-642 when I noticed that the commit 0426b60c22 might be of interest ; but we experience the same issue with the version 2.1.5 of the driver. Then, I noticed that the default behavior is using a ExponentialReconnectionPolicy that could explain a long delay for a node to be reconnected. Finally, we made the following changes on driver side :

• Set the read timeout to 20 seconds (SocketOptions) 
  
• Set up a ConstantReconnectionPolicy with a delay of 2 seconds

The situation seems more stable, but we still experience some "no host was tried". We also have to discover why some requests are so slow ; the 'max' values of the histograms are very high compared to other values.

Questions

Is there a way to speed up the come-back of C* nodes (I am not sure to have seen any C* node returning to the UP state) ? Is there a driver's API that makes possible to "force" the state of nodes (we would notice by ourself if a C* node is really DOWN) ? 

The NullPointerException

We experience it few times per day on each node. We only use prepared statements. A statement of an occurrence is : "SELECT valCol FROM ks.table WHERE keyCol = ?;" where keyCol is a BIGINT and valCol a UUID. (This table is used as an 'index table').

Thanks,

Fabrice LARCHER

Senior Developer

Phone : 00 33 1 85 08 43 65

[Andrew Tolbert]

Hi Fabrice,

For another table, I get : 

nodetool: Unable to compute when histogram overflowed

This looks like CASSANDRA-8028 which is fixed in 2.1.3.  Looks like it has to do with there being large partitions present in the table you are trying to get histograms for.  Do you expect there to be large partitions for that table?

The JMX measures on driver side show higher latencies (please see jconsole screenchot attached). The maximum time is about 23 seconds. 

Those numbers don't look so good.  a 50th percentile of 1397 milliseconds is not very good (although everything is relative).  However, it appears that the timer for this metric starts before the driver has chosen the node to query (via the LoadBalancingPolicy), which makes me think what could be happening is that the load balancing policy is not returning the host to query quickly (meaning there are no 'UP' hosts available and it is waiting on 'SUSPECTED' (nodes that went down because of timeout) .

 I did not understand why the C* nodes could not come back soon after the requests' timeouts

Have you tried enabling tracing on your queries?   It might be nice to enable probabilistic tracing on one of your nodes using nodetool settraceprobability and see which of your requests are taking a long time and why (see this blog post for guidance).

Is there a way to speed up the come-back of C* nodes (I am not sure to have seen any C* node returning to the UP state) ? Is there a driver's API that makes possible to "force" the state of nodes (we would notice by ourself if a C* node is really DOWN) ? 

There is a method in PoolingOptions (accessible via cluster.getConfiguration().getPoolingOptions()) named 'refreshConnectedHosts'.  This forces the driver to refresh the status of all hosts.

What I really am starting to wonder and suspect is that you may be hitting JAVA-637.   I observed a while back that if you have all nodes go down near the same time while a bunch of timeouts are being retried, the executor that processes status events (UP, DOWN, ADDED, etc.) becomes severely backlogged by retry attempts that are waiting on 'suspected' hosts from the load balancing policy and time out.

I have an issue opened to more easily detect this issue via metrics (JAVA-626 - PR), but you can also detect it pretty easily by doing a thread dump (using jstack or something similar) and checking the status of your "Cassandra Java Driver worker-X" threads.  If they are all in a "TIMED_WAITING" state similar to below you could be encountering this issue.

"Cassandra Java Driver worker-0" nid=36 state=TIMED_WAITING
    - waiting on <0x1b072aaf> (a com.google.common.util.concurrent.ListenableFutureTask)
    - locked <0x1b072aaf> (a com.google.common.util.concurrent.ListenableFutureTask)
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:422)
    at java.util.concurrent.FutureTask.get(FutureTask.java:199)
    at com.datastax.driver.core.policies.DCAwareRoundRobinPolicy.waitOnReconnection(DCAwareRoundRobinPolicy.java:368)
    at com.datastax.driver.core.policies.DCAwareRoundRobinPolicy.access$100(DCAwareRoundRobinPolicy.java:56)
    at com.datastax.driver.core.policies.DCAwareRoundRobinPolicy$1.computeNext(DCAwareRoundRobinPolicy.java:310)
    at com.datastax.driver.core.policies.DCAwareRoundRobinPolicy$1.computeNext(DCAwareRoundRobinPolicy.java:279)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at com.datastax.driver.core.policies.TokenAwarePolicy$1.computeNext(TokenAwarePolicy.java:157)
    at com.datastax.driver.core.policies.TokenAwarePolicy$1.computeNext(TokenAwarePolicy.java:142)
    at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
    at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
    at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:102)
    at com.datastax.driver.core.RequestHandler$1.run(RequestHandler.java:179)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
      - java.util.concurrent.ThreadPoolExecutor$Worker@e1e012a

If that is indeed what is happening, it'd be interesting to see if disabling retries (using AlwaysIgnoreRetryPolicy) helps at all.  Obviously disabling retries is less than ideal, but it'd be interesting to see if it helps nonetheless  That could also lend credence to JAVA-637 being the problem.

We experience it few times per day on each node. We only use prepared statements. A statement of an occurrence is : "SELECT valCol FROM ks.table WHERE keyCol = ?;" where keyCol is a BIGINT and valCol a UUID. (This table is used as an 'index table').

Thanks for that information, nothing obvious sticks out there with regards to what could be causing that but I'll see if I can dig in and figure it out.

Andy

[Olivier Michallat]

However, it appears that the timer for this metric starts before the driver has chosen the node to query

 

Also, the driver-side measures include retries. We have a slow query logger coming up (JAVA-646), that should make it easier to detect timeouts.

Fabrice, could you open a JIRA issue for the NPE?

--

Olivier Michallat

Driver & tools engineer, DataStax

[Fabrice Larcher]

Hi Olivier,

I created JAVA-695 about the NPE.

​Fabrice

[Fabrice Larcher]

Hi,

Thanks for the feedback.

About the histogram's overflow, the table can have large partitions. We plan to use C* 2.1.4 later (as soon as it gets out in order to follow the advice of Rob).

We consider to enable tracing on some requests, thanks for the link.

About the method refreshConnectedHosts(), we are going to expose it from our application through JMX, so that we could test it.

As you suspected, we currently have many "Cassandra Java Driver worker-x" threads waiting with that stack trace. Then, we will try to lower the retry count. We use currently the class DowngradingConsistencyRetryPolicy and will use AlwaysIgnoreRetryPolicy instead (Note : the class 

​

AlwaysIgnoreRetryPolicy is a test class in the driver version 2.1.5, so I copy-pasted the code).

We set a particular instance of our application with far less load on the C* driver, but connected to the same C* cluster as other applications instances (that still send much more requests) up. We experienced after some hours, that the C* requests are also getting slower on that particular instance. It think that the test tend to confirm that the issue is indeed about nodes' status handling (and not about requests' capacity of the driver).

​Fabrice

[Andrew Tolbert]

Hi Fabrice,

About the method refreshConnectedHosts(), we are going to expose it from our application through JMX, so that we could test it.

Sounds good, interested to hear how that turns out.  Although, I suspect it may not be helpful, since it will utilize the same executor you are observing that gets backed up.

As you suspected, we currently have many "Cassandra Java Driver worker-x" threads waiting with that stack trace. Then, we will try to lower the retry count.

That is good to know, as far as I know, you are the first user (other than myself) to report seeing this.   I suspect it may be happening to more users, but it is non-trivial to catch unless you know to look for it or are actively looking at thread dumps.   It will be interesting to see if more users are encountering this same issue and explains others' problems in this thread.

We use currently the class DowngradingConsistencyRetryPolicy and will use AlwaysIgnoreRetryPolicy instead (Note : the class 

​

AlwaysIgnoreRetryPolicy is a test class in the driver version 2.1.5, so I copy-pasted the code).

My mistake, sorry about that!  At least the second I've recommended trying that out without realizing it was in the test code.

We set a particular instance of our application with far less load on the C* driver, but connected to the same C* cluster as other applications instances (that still send much more requests) up. We experienced after some hours, that the C* requests are also getting slower on that particular instance. It think that the test tend to confirm that the issue is indeed about nodes' status handling (and not about requests' capacity of the driver).

Just to make sure I understand, in that case do you observe the same issue ("Cassandra Java Driver worker-x" threads waiting on the LBP while retrying a request)?   Or are you saying that you suspect you may not be getting status events from Cassandra quickly enough in general?

Thanks,

Andy

[Andrew Bryan]

Hi Ankush,

Did you manage to resolve this issue? I think I'm having the exact same problem as you and I believe it to be different to everyone else in this thread. The annoying thing about that error message is that it is returned in a number of other situations where I believe the root cause is very different.

We recently upgrade to the 2.1.4 version of the driver (from 2.1.1) and started seeing the "NoHostAvailableException: All host(s) tried for query failed (no host was tried)" errors--not immediately--maybe eight to 36 hours after application start up:

We have a distributed application that is only inserting into a large and stable cluster (which also has a number of other read/write clients that are not experiencing issues when we are).

Our application will init a cluster connection, create a session, prepare a statement and then a number of threads will be sharing that session and inserting (using the prepared statement) fairly large statements (the insert query might be about 3KB in size; contains some Map columns etc). We don't use batching because of the size.

The application runs continuously with a constant stream of inserts. Nothing ever closes a session or the cluster. All the reads/connect timeouts for our driver are large and larger than their server counterparts.

We never saw any issues (over many months) using the 2.1.1 version of the driver, but at least 2.1.4 and I believe 2.1.5 exhibit the issue described.

Curiously, when the errors start, they will begin simultaneously on several of my inserting nodes, but NOT all. The synchronization can only mean that something network related or triggered by C* is setting it off... regardless, I'm certain there's a bug in the recent versions of the driver.

Unfortunately I can't enabled trace logs because the issue might take a day or two to exhibit and the disks would fill in minutes.

I've dropped back to the 2.1.1 driver and things are stable again, but I would love to have this solved...  

[Ankush Goyal]

Hi Andrew,

We upgraded from driver version 2.0.2 to 2.1.4, and after a few days starting seeing this error. After making a few changes we didn't get this error for a few days, and falsely thought it was fixed. But, again started getting this issue after 3 days. This functionality of marking hosts down on getting timeouts was added in 2.0.6 version of the driver, so we decided to downgrade to 2.0.5 for the time being. There are various other issues with that version, but at least we are not seeing this particular issue.

I tried keeping up with the discussion on this thread until it started going in different directions.
Unfortunately, I haven't heard a good explanation from anyone on the Driver team about this particular issue or any plans to resolve it.

[Ananth kumar]

Hi Ankush,

Are you doing cross DC writes? When i stopped the cross DC writes , the problem stopped occurring for me. I think the timeout is mainly due to driven lag. 

Regards,

Ananthkumar K S

On Friday, February 27, 2015 at 10:56:36 PM UTC+5:30, Ankush Goyal wrote:

Hi Guys,

[Ankush Goyal]

Hi Ananth,

Nope, our cluster is in one DC only.

[Ananth kumar]

You are running your cassandra node in the same LAN of your app server? I doubt if its a WAN that you are using. If thats the case , the problem that i faced due to uplink slowness could be the case of concern. Either way, not sure why this has started off in the recent versions of driver.

[Olivier Michallat]

Hi,

An update on this issue: after much consideration, we've decided to revert JAVA-425. This will be part of our next release, 2.0.10 (due in the upcoming weeks).

In addition, this release will introduce a query logger, that will help you identify requests that cause client-side timeouts.

--

Olivier Michallat

Driver & tools engineer, DataStax

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Ankush Goyal]

This is great, much appreciated Olivier!

[Olivier Michallat]

Hi,

Yes, the changes will be ported to 2.1.6. I'm not sure of the timeframe yet, it will depend on whether 2.1.6 contains additional features apart from those changes, which we haven't decided yet.

--

Olivier Michallat

Driver & tools engineer, DataStax

On Wed, Apr 22, 2015 at 2:29 PM, Cristian Manuel Vertiz Fernandez <vertiz....@gmail.com> wrote:

Oliver

Hi,

How about 2.1.X version, will it be fixed?

[xuexu guo]

Hi Olivier,

I met the same issue when I am using spark-cassandra-connector 1.2.2 which depends on driver 2.1.5. 

If the connector is upgraded to depends on 2.1.6, it will be fixed, right?

Thanks 

[Olivier Michallat]

If your issue was caused by requests timing out, yes that would address the issue.

--

Olivier Michallat

Driver & tools engineer, DataStax

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Gábor Auth]

Hi,

We recently upgrade to the 2.1.4 version of the driver (from 2.1.1) and started seeing the "NoHostAvailableException: All host(s) tried for query failed (no host was tried)" errors--not immediately--maybe eight to 36 hours after application start up:

Same issue here... the last stable driver is the 2.1.3, I've tried all driver from 2.1.4 to 2.2.0-rc3... :(

Bye,

Gábor Auth

[deepak dabi]

HI Ankush,

I have seen here so much research on this topic, i am running in this issue on C* 3.5 (on windows 1 machine node), CQL 3.4 using datastax driver latest. I have below doubt if you could know about it . 

I am trying to write 2GB(which is limit of Cassandra single key/value) data into single(or many) column using Datastax driver,CQL3 on one machine windows node.I am hardly able to write data like 100MB(in single column), that too by facing almost all kind of exceptions and config changes.If i try write 100MB data i have to keep "commitlog_segment_size_in_mb: 200" which works; after that Cassandra killing itself.Is there any way where i can insert upto 2GB data into one(at least) or many column and find out timing ? 1. I am able to write till 100MB (in single column of table) by commitlog_segment_size_in_mb: 200; 2. When i tried to write more ; i am getting nohostavailableexception: all host(s) tried for query failed waiting for server ; operation time out waiting for server response. 3 I have seen system and debug log there in nothing in there. 4. I tried it in both linux and window machine setup in windows error is :- ERROR [HintsWriteExecutor:1] 2016-06-10 12:59:40,239 CassandraDaemon.java:195 - Exception in thread Thread[HintsWriteExecutor:1,5,main] java.lang.OutOfMemoryError: Direct buffer memory Qus is can we write more then 1GB data in one column of table or not ? Any help appreciated on above - Thanks deepal

On Fri, Feb 27, 2015 at 9:26 AM, Ankush Goyal <ank...@gmail.com> wrote:

Hi Guys,

We recently (this week) upgraded our java driver from 2.0.4 to 2.1.4 (we are running Cassandra version 2.0.12). We were using 2.0.4 for a long time (more than an year) in production and did not have this issue.
But, after upgrading we starting seeing intermittent  NoHostAvailableExceptions (that fail all the requests on client side):

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)
	at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108) ~[stormjar.jar:na]
	at com.datastax.driver.core.SessionManager.execute(SessionManager.java:530) ~[stormjar.jar:na]
	at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:566) ~[stormjar.jar:na]
	at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:119) ~[stormjar.jar:na]

We are having this issue in production, so would love to get some advice on how to triage this. What we know for sure:
* When errors start happening Cassandra cluster isn't under unusual load.
* We are able to reach Cassandra hosts (cluster) from client side - Not a connection issue.

Cheers,
-Ankush

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[孙奇]

We recently upgrade from the 2.1.4 to 2.1.6, is seems a little better!

在 2015年10月30日星期五 UTC+8上午12:50:44，Gábor Auth写道：

[Alex Popescu]

Just for reference: the latest version of the Java driver on the 2.1 branch is 2.1.10.2. I'd strongly encourage everyone to try to upgrade to the latest.

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

--

Bests,

Alex Popescu | @al3xandru

Sen. Product Manager @ DataStax

» DataStax Enterprise - the database for cloud applications. «

[Leandro Mendes Ferreira]

Hi guys,

I'm having the same problem.

When I do query in  Cassandra table from Spark(dataframe) the error is shows:

"NoHostAvailableException: All host(s) tried for query failed (no host was tried)"

I'm using Apache Cassandra 3.7 but this same erro ocorred in Apache Cassandra 3.0.8. I'm using Apache Spark 1.6 and Cassandra java drive 3.0.2.

I compiled the spark-cassandra-connector with the following directives:

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector" % "2.0.0-M1"

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector-embedded_2.10" % "2.0.0-M1"

libraryDependencies += "com.datastax.spark" %% "spark-cassandra-connector-java" % "2.0.0-M1"

libraryDependencies += "com.datastax.cassandra" %% "cassandra-driver-core" % "3.0.3"

libraryDependencies += "com.datastax.cassandra" %% "cassandra-driver-mapping" % "3.0.3"

libraryDependencies += "com.datastax.cassandra" %% "cassandra-driver-extras" % "3.0.3"

I saw that the error was with the Cassandra Java Drive but I'm using a new version java drive...

Can someone help me?

Sorry for my bad English.

Regards!

[Kevin Gallardo]

Hi Leandro,

NoHostAvailableException ("NHAE") is the end consequence of previous errors that have caused the driver to close all connections to all hosts.

Initially, the driver connects to a cluster, then an error happens because of "reason X", which causes the driver to defunct the connection that errors happens with, and if such error happens with all hosts on all connections, NHAE is thrown ("No host was tried" == "there isn't any valid connection left that the driver can use to send a request"). Hence what you need, to debug your problem is to find the root cause that causes the driver to close the connections initially (the "reason X" mentioned previously). You can check in your logs, before the time the NHAE is thrown, there should be previous exceptions indicating the root cause of your problem, because the NHAE alone doesn't give much information. Would you mind checking this?

Cheers.

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.

--

Kévin Gallardo.

kgdo.me

Software Engineer in Drivers and Tools Team at DataStax.

 

[uttam anand]

I have configured a cassandra clustter with 3 nodes

Node1(192.168.0.2) , Node2(192.168.0.3), Node3(192.168.0.4)

Created a keyspace 'test' with replication factor as 2

.

Create KEYSPACE test WITH replication = {'class':'SimpleStrategy', 'replication_factor' : 2}

When I stop either Node2 or Node3 (one at a time and both at one time) , I am able to do the CRUD operations on the keyspace.table.

When I stop Node1 and try to update/create a row from Node4 or Node3, getting following error although Node3 and Node4 are up and running-:

All host(s) tried for query failed (tried: /192.168.0.4:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections))) com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /192.168.0.4:9042 (com.datastax.driver.core.exceptions.DriverException: Timeout while trying to acquire available connection (you may want to increase the driver number of per-host connections)))

I am not sure how Cassandra elects a leader if a leader node dies.

Thanks

Uttam Anand