Node list refreshes in 4.x vs 3.x

[Ammar Khaku]

In Datastax4, the entire Cassandra node list is refreshed via MetadataManager#refreshNodeList. This happens in two cases: 1. when a session is initialized, and 2. when we reconnect to a node that was temporarily unavailable.

Note that a refresh does not happen if a node goes down because (for example) it was terminated ungracefully. In that case the driver holds on to the old node and tries to reconnect periodically, leading to warn-level log messages that look like this:

2022-03-24 16:07:23.846  WARN 11077 --- [Thread-1] c.d.o.d.i.c.p.ChannelPool                : [sessionname|/1.2.3.4:1234]  Error while opening new channel (ConnectionInitException: [sessionname|connecting...] Protocol initialization request, step 1 (STARTUP {CQL_VERSION=3.0.0, DRIVER_NAME=DataStax Java driver for Apache Cassandra(R), DRIVER_VERSION=4.13.0, CLIENT_ID=16f1615a-ef11-4111-a111-c9b01112f453}): failed to send request (java.nio.channels.NotYetConnectedException))

Datastasx3 was more aggressive where it came to refreshing node lists, for instance it would refresh the list when connecting to a new node [1] [2]. This means that when a node came up (with a different IP) to replace any node that was terminated ungracefully, the entire node list would be refreshed and the bad node would be removed.

My question here is: is it worth revisiting the behavior w.r.t node list refresh frequency in Datastax4? If not I can implement a workaround on our end e.g. by implementing a NodeStateListener (should be straightforward, but any advice there would be appreciated too!)

1: https://github.com/datastax/java-driver/blob/3.11.1/driver-core/src/main/java/com/datastax/driver/core/ControlConnection.java#L319

2: https://github.com/datastax/java-driver/blob/3.11.1/driver-core/src/main/java/com/datastax/driver/core/ControlConnection.java#L758

[Erick Ramirez]

This looks very similar to JAVA-3002 where the driver is not aggressive enough at evicting nodes from its nodes list.

Would you please log a ticket so it's on the board for us to look into? Cheers!

[Ammar Khaku]

Thanks, this does look like it causes JAVA-3002! Logged JAVA-3005. Happy to contribute a PR if yall decide this is worth changing.

-Ammar

On Tue, Apr 5, 2022 at 5:01 AM Erick Ramirez <erick....@datastax.com> wrote:

This looks very similar to JAVA-3002 where the driver is not aggressive enough at evicting nodes from its nodes list.

Would you please log a ticket so it's on the board for us to look into? Cheers!

--
You received this message because you are subscribed to a topic in the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/a/lists.datastax.com/d/topic/java-driver-user/yuvxXCpUFmc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to java-driver-us...@lists.datastax.com.
To view this discussion on the web visit https://groups.google.com/a/lists.datastax.com/d/msgid/java-driver-user/68c85cc8-bd26-455c-9835-9f62565385cbn%40lists.datastax.com.

[Ammar Khaku]

Just checking in on https://datastax-oss.atlassian.net/browse/JAVA-3005, I'd love to get some thoughts on it so we can move forwards! Or if you think it's not worth tackling within Datastax itself that's fine too, just let me know.

-Ammar

[Bret McGuire]

   I was just looking at this very ticket today in fact.  I hope to have some follow-up information on this ticket soon.

 - Bret -