All host(s) tried for query failed (no host was tried) when doing multiple queries

[Tim Rombauts]

When using the Datastax Java Driver for Cassandra, I get the 'All host(s) tried for query failed (no host was tried)' exception after a while when performing lots of queries.

All C* instances are running stable though, and using a 2nd Session or a cqlsh session works perfectly.

After this exception, the Session is no longer usable.

However, doing this with multiple Sessions in parallel is not a problem, so I doesn't look like a problem of C* but more like a problem of the driver.

I've been trying multiple scenarios with different options of the client, like MinConnection, CoreConnection, MaxConnection, SocketTimeout, ... but the result is the same.

I also tried newer versions of the client (current version is 2.0.0, but also tried with 2.0.9 and 2.1), but the result is the same.

I've also done an experiment: when the exception occurs I close the existing cluster/session and open a new cluster/session.

This works, however, it seems the old Cluster/Session retains 10s of open Threads, so after a while we end up with 1000s of Threads.

According to the docs, 1 Session per client (in this case it's even just 1 Thread using it) should be sufficient, so I must be doing something wrong.

Has anyone encountered the same issue before? If so, what other options can I try?

Thanks in advance,

Regards,

   Tim

Please take care of our planet. Do not print this email unnecessarily.

This message is intended for the exclusive use of the person(s) mentioned as recipient(s) and may contain personal and/or confidential information. If you have received this message in error, please notify the sender and delete this message from your system immediately. Directly or indirectly copying, disclosing, distributing, printing, publicizing and/or in any way using this message or any part thereof by any means is strictly prohibited if you are not the intended recipient(s). This message and any associated attachments cannot be deemed as legally binding under any circumstances without the express written consent from REstore NV/SA or its affiliated companies. REstore NV/SA or its affiliated companies are not responsible for any loss and/or damages arising from any errors and/or omissions in its e-mail messages.

[Alex Popescu]

On Mon, Feb 9, 2015 at 1:05 AM, Tim Rombauts <tim.ro...@restore.eu> wrote:

I also tried newer versions of the client (current version is 2.0.0, but also tried with 2.0.9 and 2.1), but the result is the same.

Tim, in order to figure out what's behind this error I'd strongly suggest using the latest version (2.0.9) to eliminate the issues that have been addressed since 2.0.0 was released. Second, we'd need more details about the stack trace you are getting.

thanks

--

[:>-a)

Alex Popescu
Sen. Product Manager @ DataStax
@al3xandru

[Tim Rombauts]

Hey Alex,

Thx for your reply.

I tried with version 2.0.9.1, but the result is the same.

Here the stacktrace:

com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)

at com.datastax.driver.core.exceptions.NoHostAvailableException.copy(NoHostAvailableException.java:84)

at com.datastax.driver.core.DefaultResultSetFuture.extractCauseFromExecutionException(DefaultResultSetFuture.java:263)

at com.datastax.driver.core.DefaultResultSetFuture.getUninterruptibly(DefaultResultSetFuture.java:179)

at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:52)

at com.datastax.driver.core.AbstractSession.execute(AbstractSession.java:36)

at ...

at java.lang.Thread.run(Thread.java:722)

Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)

at com.datastax.driver.core.RequestHandler.sendRequest(RequestHandler.java:108)

at com.datastax.driver.core.SessionManager.execute(SessionManager.java:508)

at com.datastax.driver.core.SessionManager.executeQuery(SessionManager.java:544)

at com.datastax.driver.core.SessionManager.executeAsync(SessionManager.java:119)

Thanks for your help.

Regards,

   Tim

Op maandag 9 februari 2015 22:16:43 UTC+1 schreef Alex Popescu:

[Ajay]

Hi Tim,

Are you using the same Statement in multiple threads? I remember getting a similar when I tried using the same Statement in different threads(by mistake).

Thanks

Ajay

To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.

[Vishy Kasaravalli]

“No host was tried” below could be due to:

1. Your LoadBalancingPolicy newQueryPlan is not returning any hosts for some reasons. What LoadBalancingPolicy are you using?

2. LoadBalancingPolicy is returning the hosts but none of those hosts have connection pool. In such cases, query is not sent to them. 

How many nodes do your cassandra ring have? What is your SocketOptions.readTimeOutMiliis? How are you firing your queries: execute or executeAsync? Does this problem happen right away or after some time? Please set the following on one of your clients, and send the logs from 

log4j.logger.com.datastax.driver.core=TRACE

[Tim Rombauts]

Hey Ajay,

No, we're not using a Statement, we're executing a plain cql string.

Also: this error occurs using the Cluster/Session from within a single Thread.

Regards,

   Tim

Op dinsdag 10 februari 2015 10:59:29 UTC+1 schreef Ajay Garga:

[Tim Rombauts]

Hey Vishy,

Actually enabling the tracing on the driver was an excellent tip!

As expected the problem was on our side: by enabling this tracing we noticed that after a while some code in our application was closing the Cluster/Session (which is shared ofcourse), which resulted in the NoHostAvailable.

It would have been better if the exception would have indicated this (All connections closed or so), but your tip is much appreciated!

Regards,

   Tim

Op dinsdag 10 februari 2015 18:38:35 UTC+1 schreef Vishy Kasar:

[uskratos]

Hi Tim.
What was the solution? I am facing the same issue with driver version 2.1.7 and cassandra 2.2.0

All host(s) tried for query failed (tried: /10.82.51.72:9042 (com.datastax.driver.core.TransportException: [/10.82.51.72:9042] Connection has been closed), /10.82.51.73:9042 (com.datastax.driver.core.TransportException: [/10.82.51.73:9042] Connection has been closed), /10.82.51.71:9042 (com.datastax.driver.core.TransportException: [/10.82.51.71:9042]

I changed the load balancing policy to "new TokenAwarePolicy(new DCAwareRoundRobinPolicy())" from "new RoundRobinPolicy()" and made no difference at all. Interesting to note that this issue only occurs on large batch jobs and usually after 200 or so records (inserts) were processed and since upgrading to 2.1.7 driver.

Thank you for your reply.