Hosts unavailable when used with AWS Keyspaces

[Matt Massicotte]

Hello everyone!

I'm trying out this driver by way of Apple's newly-release Swift wrapper: https://github.com/apple/swift-cassandra-client. The catch is I'm using it with AWS Keyspaces, which is a fully managed Cassandra cluster.

Things are working, but I noticed an interesting failure today. Keyspaces load balances servers, resulting in host IPs changing frequently. It looks like all of the cached hosts were rotated out. I'm including the logging the driver produced, but the relevant line is:

All hosts in current policy attempted and were either unavailable or failed

Restarting the process immediately fixed the issue. Is there a way for me to configure the driver to better handle this kind of situation?

Thanks so much!

Matt

| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.210 during a heartbeat request: Request timed out  |
| 1671711711486 | 1671711711.486 [WARN] (cluster.cpp:774:virtual void datastax::internal::core::Cluster::on_close(datastax::internal::core::ControlConnection *)): Lost control connection to host 3.234.248.210                                                     |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [ERROR] (connection.cpp:352:void datastax::internal::core::Connection::on_terminate(datastax::internal::core::Timer *)): Failed to send a heartbeat within connection idle interval. Terminating connection...                      |
| 1671711711486 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.203 during a heartbeat request: Request timed out  |
| 1671711711487 | START RequestId: 0fad21b7-94ea-4d88-85f6-f1392c6970c2 Version: $LATEST                                                                                                                                                                             |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.198 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.195 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.234 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.210 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.192 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.206 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.486 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.225 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.487 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.248 during a heartbeat request: Request timed out  |
| 1671711711487 | 1671711711.487 [WARN] (connection.cpp:56:virtual void datastax::internal::core::HeartbeatCallback::on_internal_error(CassError, const datastax::String &)): An error occurred on host 3.234.248.255 during a heartbeat request: Request timed out  |

All hosts in current policy attempted and were either unavailable or failed