NoNodeAvailableException on exceeding max concurrent requests per connection

[Ammar Khaku]

Hiya, I ran into a case recently where I had was trying to submit more requests to a node (really to all nodes) than advanced.connection.max-requests-per-connection. I got a NoNodeAvailableException and spent my time scratching my head before realizing what had happened. Is that by design, it seems a little misleading? BusyConnectionException seems more relevant and I assume it would be one of the errors on an AllNodesFailedException but I got NoNodeAvailableException instead.

In my specific case it was on preparing statements (since I was preparing a ton of statements concurrently) but when testing it appears to happen for executeAsync as well. The code path is similar: we run out of stream IDs so session#getChannel returns null here [1], [2] and so we end up throwing AllNodesFailedException.fromErrors with a null errors object [3], [4] which ends up throwing NoNodeAvailableException [5]

-Ammar

1: https://github.com/datastax/java-driver/blob/4.13.0/core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlPrepareHandler.java#L188

2: https://github.com/datastax/java-driver/blob/4.13.0/core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java#L249

3: https://github.com/datastax/java-driver/blob/4.13.0/core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlPrepareHandler.java#L195

4: https://github.com/datastax/java-driver/blob/4.13.0/core/src/main/java/com/datastax/oss/driver/internal/core/cql/CqlRequestHandler.java#L261

5: https://github.com/datastax/java-driver/blob/4.13.0/core/src/main/java/com/datastax/oss/driver/api/core/AllNodesFailedException.java#L53

[Alexandre Dutra]

Hello Ammar,

I am sorry to hear that you had a hard time understanding why the driver was throwing a NoNodeAvailableException.

One thing that strikes me in your case is that not only one, but all nodes are busy. Even if we change the exception type that is being thrown, this wouldn't solve the root problem: either your application is generating way too many requests, or your cluster is heavily undersized. I suggest that you spend some time tuning both the driver and the cluster accordingly.

I agree however that a NoNodeAvailableException is not very self-explanatory. I think we could improve this by recording an error for the node when session#getChannel returns null, and only then skip to the next one. 

Would you be willing to contribute a PR for this? If so, please start by creating a Jira here [2] and I would provide you with some guidance there.

Thanks,

Alex Dutra

[1] https://github.com/datastax/java-driver/blob/4.13.0/core/src/main/java/com/datastax/oss/driver/internal/core/pool/ChannelSet.java#L91-L119

[2] https://datastax-oss.atlassian.net/browse/JAVA

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-us...@lists.datastax.com.
To view this discussion on the web visit https://groups.google.com/a/lists.datastax.com/d/msgid/java-driver-user/7dbb7cb7-651c-454c-967c-d68efe6eaeean%40lists.datastax.com.

--

Alexandre Dutra
Software Engineer, DataStax

[Ammar Khaku]

Absolutely, yes I'd love to contribute a PR! Created https://datastax-oss.atlassian.net/browse/JAVA-2959 to get started, we can chat there.

Yes agreed, the root problem is I tuned max-requests-per-connection too aggressively - throwing an exception is valid, it's just the specific exception sent me down a rabbit hole of checking if my nodes were up. The Cassandra cluster is running fine, once I figured out what was going on I just had to tweak the client.

-Ammar

You received this message because you are subscribed to a topic in the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/a/lists.datastax.com/d/topic/java-driver-user/koLPCeEP-Dw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-driver-us...@lists.datastax.com.
To view this discussion on the web visit https://groups.google.com/a/lists.datastax.com/d/msgid/java-driver-user/CAKaajdBG3nU9Bwnpi1k93dWN_w1t6ju%2BwrXXL1qm7HYE4B%3Dr8w%40mail.gmail.com.