Question about contact points going down

[Alejandro Ginés BEEVA]

Hi all,

Some days ago we faced the following situation. We have a 4 node cluster. When starting the Java Driver, we add all 4 nodes as contact points. During some weeks, there has been some activities regarding to the nodes, 3 of them were replaced by other 3 nodes with different IPS. Then, some days ago, the remaining original node was also replaced by another one, and the driver lost contact with the DB. We needed to restart our application to restore the DB access.

So, the question is, how can we avoid this behaviour? I guess the error here is that no original contact point was up. Our nodes are located in AWS and sometimes we are forced to change some servers. Any way that the driver can register the new nodes as contact points?

We are using the java 3.0.0 driver.

Kind regards,

Alex

[Andrew Tolbert]

Hi Alex,

The driver should be able to handle this, but without more information it is hard to know what could have happened.

One thought I had since you mentioned AWS.   Are the contact points you are providing the same as the rpc_address configuration in cassandra.yaml on each of your cassandra nodes?  i.e. if you do a 'select * from system.peers' on a node, does the rpc_address column match the ips you are providing as contact points?   The driver uses the system.peers table and events sent to it from C* (that contain the rpc_address) to discover new nodes.

If the contact points you provide are reachable ip addresses, where the rpc_address in cassandra.yaml cannot be reached, it's possible that the driver is not able to connect to newly discovered nodes since their rpc_address is not reachable by your client.  That could explain that once all 4 nodes have been replaced/are down that the driver is no longer able to connect.


Thanks,
Andy

[Alejandro Ginés BEEVA]

Hi Andrew, thanks for your response.

Is there any method on the driver to check if the DB is reachable? We want to implement a fail proof system that would check if the DB is reachable before sending any query.

Kind regards,

Alex

[Andrew Tolbert]

Hi Alex,

There is not really a direct 'Is up' API call as far as I know, there are a few things you can do that comes to mind:

1. Cluster.getConfiguration().getPolicies.getLoadBalancingPolicy().newQueryPlan().hasNext() if returns false, no hosts are available.
2. Session.getState().getConnectedHosts().isEmpty() if true, no hosts are available

I think a 'let it fail' approach might be better though, execute your queries and if a NoHostAvailableException is returned then behave accordingly.  The reason I think this is better is because it is less complicated and if no hosts are available when you go to make a query it will throw an error pretty fast.  Even if you check either of those two ways I mentioned you could still possibly encounter a NoHostAvailableException (i.e. if hosts go down in process of sending the query).

Thanks!

Andy