Connecting to an external cassandra cluster from K8

23 views
Skip to first unread message

CameronG

unread,
Feb 9, 2021, 3:35:00 AMFeb 9
to Stargate Mailing List
Hi All,

    I spent some time late last year evaluating stargate runnng in Kubernetes connected to a locally running cassandra cluster. All ran well.

    I've now tried connecting it to a cassandra cluster running outside of kubernetes and am unable to connect. I'm getting errors such as:

ERROR: Bundle io.stargate.db.cassandra [1] EventDispatcher: Error during dispatch. (io.stargate.core.activator.ServiceStartException: Unable to start persistence-cassandra-3.11)

Caused by: java.lang.RuntimeException: Unable to gossip with any peers

I have some other services that connect to the cassandra cluster using the Cassandra java Driver so the cluster is accessible.

Could anyone suggest what I should look for here? I've confirmed that the DataCenter, Cluster Name, seed and rack are correct in the configuration. We are eventually going to migrate the external Cassandra cluster into K8 at which point this issue goes away but it's not planned to be done for a while.

    Thanks,

    Cameron

Dmitri Bourlatchkov

unread,
Feb 9, 2021, 10:24:22 AMFeb 9
to CameronG, Stargate Mailing List
HI Cameron,

Please double check the IP address Stargate uses internally and the address that C* storage nodes see from their perspective. Those are probably different, I guess.

If that is the case, the --listen-address Stargate start option should be the address as seen by storage nodes and the --bind-to-listen-address option should _not_ be used. This will allow Stargate to accept CQL connections on all network interfaces and use the listen address for communication with storage nodes.

I guess if address translation is in effect that might still not help, but please let us know how it plays out.

Cheers,
Dmitri.

--
You received this message because you are subscribed to the Google Groups "Stargate Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stargate-user...@lists.stargate.io.


--
Dmitri Bourlatchkov

CameronG

unread,
Feb 9, 2021, 3:03:52 PMFeb 9
to Stargate Mailing List, dmitri.bo...@datastax.com, Stargate Mailing List, CameronG
Hi Dmitri,

    Thanks for the information.

     If I have understood correctly:
  1. When stargate is started in k8 I can see from the logs that --listen is set to 192.168.108.3
  2. The ip address of our 2 existing external cassandra nodes are 10.1.2.105 & 10.1.2.107
    I tried setting --listen to 10.1.2.105 which produces a different error:

    Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to bind to address /10.1.2.105:7000. Set listen_address in cassandra.yaml to an interface you can bind to, e.g., your private IP address on EC2

     Our cassandra.yaml has the listen_address blank.

    Regards,

    Cameron

CameronG

unread,
Feb 9, 2021, 4:32:23 PMFeb 9
to Stargate Mailing List, CameronG, dmitri.bo...@datastax.com, Stargate Mailing List
Hi Dmitri,

    Just to clarify from my previous post, by default, the ip address that the --listen is connecting to is just the ip address of the stargate pod itself. 

    Regards,

    Cameron

Doug Wettlaufer

unread,
Feb 9, 2021, 5:47:40 PMFeb 9
to CameronG, Stargate Mailing List, Dmitri Bourlatchkov
Hi Cameron,

Just a heads up, there’s another discussion concerning the same issue going on here https://github.com/stargate/stargate/issues/595.


I think to get this to work how you want stargate will need a service with a clusterIP which stargate will then need to broadcast. Basically you’ll need something that’s externally available which your C* cluster can use in order to gossip with stargate over port 7000.

----
Doug Wettlaufer
e. doug.we...@datastax.com
w. www.datastax.com

CameronG

unread,
Feb 9, 2021, 5:56:56 PMFeb 9
to Stargate Mailing List, doug.we...@datastax.com, Stargate Mailing List, dmitri.bo...@datastax.com, CameronG
Thanks Doug - I'll have a look at setting this up.
Reply all
Reply to author
Forward
0 new messages