Cassandra:Unable to contact seed nodes.

[varun saluja]

Hi Experts,

I am facing issue while adding a new node to multi data center cluster.  

Error: 

INFO  [CompactionExecutor:1] 2017-03-02 15:17:38,769 CompactionTask.java:141 - Compacting [SSTableReader(path='/cassandra/apache-cassandra-2.1.

8/data/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-1-Data.db'), SSTableReader(path='/cassandra/apache-cassandra-2.1.8/da

ta/data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-3-Data.db'), SSTableReader(path='/cassandra/apache-cassandra-2.1.8/data/d

ata/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-2-Data.db'), SSTableReader(path='/cassandra/apache-cassandra-2.1.8/data/data/

system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-4-Data.db')]

INFO  [main] 2017-03-02 15:17:38,776 StorageService.java:1109 - JOINING: sleeping 30000 ms for pending range setup

INFO  [CompactionExecutor:1] 2017-03-02 15:17:38,826 CompactionTask.java:274 - Compacted 4 sstables to [/cassandra/apache-cassandra-2.1.8/data/

data/system/local-7ad54392bcdd35a684174e047860b377/system-local-ka-5,].  5,929 bytes to 5,766 (~97% of original) in 49ms = 0.112222MB/s.  4 tot

al partitions merged to 1.  Partition merge counts were {4:1, }

ERROR [main] 2017-03-02 15:18:08,784 CassandraDaemon.java:541 - Exception encountered during startup

java.lang.IllegalStateException: Unable to contact any seeds!

        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1135) ~[apache-cassandra-2.1.8.jar:2.1.8]

        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:927) ~[apache-cassandra-2.1.8.jar:2.1.8]

        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:723) ~[apache-cassandra-2.1.8.jar:2.1.8]

        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:605) ~[apache-cassandra-2.1.8.jar:2.1.8]

        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:378) [apache-cassandra-2.1.8.jar:2.1.8]

        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:524) [apache-cassandra-2.1.8.jar:2.1.8]

        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:613) [apache-cassandra-2.1.8.jar:2.1.8]

WARN  [StorageServiceShutdownHook] 2017-03-02 15:18:08,787 Gossiper.java:1418 - No local state or state is in silent shutdown, not announcing shutdown

Cluster Info:

DC1

Node A

Node B

Node C

DC2

Node D

Node E

Cassandra.yaml . It has seeds node already defined.  Nodes A and B are from  DC 1 and node  is from DC2.(just mentioned , Node D is not defined as seed in one of the node of DC2.

- seeds: "A,B,D"

-endpoint_snitch: GossipingPropertyFileSnitch

- auto_bootstrap=true

cassandra-rackdc.properties has values:

dc=DC2

rack=RAC1

Issue Background:

We had some issue on Node F and this node was removed successfully from cluster. And later we are trying to add same node F back to cluster and its failing with above mentioned error.

Node status does not show this node F but gossipinfo still reports node F as removed.(Note sure if issue related to this.) We are trying to add same ip back to cluster here.

Can someone please help. Please let me know if any other info needed.

PS: Ports are opened b/w nodes.

Thanks & Regards,

Varun Saluja

[Tzach Livyatan]

Hi Varun

Look like the problem will fit better in the apache Cassandra mailing list[1].

If I can take a wild guess, check the connectivity from the new nodes to the seeds with the internal communication port (7000, 7001). Might be a firewall issue.

Tzach

[1] http://www.mail-archive.com/us...@cassandra.apache.org/

 

--
You received this message because you are subscribed to the Google Groups "DataStax Java Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to java-driver-user+unsubscribe@lists.datastax.com.

[nve...@oath.com]

Hi Arun , Did you got a solution for this issue ?  I am facing the same issue when adding an already removed node

Below is the same error  on the node. (Just an fyi we have all the ports open on the seed nodes and the added node is able to reach all the ports)

ERROR [main] 2018-11-19 19:50:12,105 CassandraDaemon.java:697 - Exception encountered during startup

java.lang.IllegalStateException: Unable to contact any seeds!

    at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1216) ~[apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
    at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:933) ~[apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:707) ~[apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
    at org.apache.cassandra.service.StorageService.initServer(StorageService.java:578) ~[apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:349) [apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:551) [apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:680) [apache-cassandra-3.2.1-SNAPSHOT.jar:3.2.1-SNAPSHOT]
WARN  [StorageServiceShutdownHook] 2018-11-19 19:50:12,107 Gossiper.java:1463 - No local state or state is in silent shutdown, not announcing shutdown