Failed Index shard allocation "shard or keyspace unavailable"

[pr1001]

Hi,

Setup 5 node Cassandra DC and 5 node ELassandra DC, Keyspace replication is 3 per DC. When creating new indexes status is "red" and cluster status is "red"

Elassandra 6.8.4.3

error message

   "unassigned_info" : {

                "reason" : "ALLOCATION_FAILED",

                "at" : "1970-01-01T00:00:00.000Z",

                "failed_attempts" : 1,

                "delayed" : false,

                "details" : "shard or keyspace unavailable",

                "allocation_status" : "deciders_no"

              }

Shard 0 status is "STARTED", Shard 1-4 status "ALLOCATION_FAILED".

Verified KS replication its good, gossip X2 is good, no issue with Cassandra, checked logs.

Old indexes created on the same cluster, status is "green" no issues. Error message indicated Keyspace, so assuming its specific to Elassandra and not elasticsearch.

[Vincent Royer]

Please run node tool gossipinfo and check X1contains your indices with state = 3 (STARTED).

It seems your coordinator node does not see remote shard state through the gossip field X1, so index is red.

--
You received this message because you are subscribed to the Google Groups "Elassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elassandra+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elassandra/4b4138ad-45db-42d5-a8a0-69bd1fe57146%40googlegroups.com.

[Sujeet Sigh]

Hi,

I also getting the same issue. 

We have three node cluster and all indices from all nodes are red.  Below is the output of nodetool gossipinfo.

Anything which we can do to fix this.

===================
  generation:1649500730
  heartbeat:87842
  STATUS:17:NORMAL,-1018529224246886842
  LOAD:87795:1.30088145529E11
  SCHEMA:7956:b908cd30-130a-3a78-b5e1-d7b2b9f9d530
  DC:9:dc1
  RACK:11:rack1
  RELEASE_VERSION:5:3.11.9.1
  INTERNAL_IP:7:xxx.xxx.xxx.xx
  RPC_ADDRESS:4: xxx.xxx.xxx.xx
  NET_VERSION:2:11
  HOST_ID:3:f9ed38c8-b7e0-4944-a0fe-4af6b5c6eb84
  RPC_READY:29:true
  X1:56:{"f_historical_stories_qa_2020_08_16":3,"f_historical_stories_qa_2020_08_17":3,"f_historical_stories_qa_2021_01_31":3,"f_historical_stories_qa_2020_08_18":3,"f_historical_stories_qa_2020_08_19":3,"f_historical_stories_qa_2021_01_30":3,"f_historical_stories_qa_2020_08_23":3,"f_historical_stories_qa_2020_08_24":3,"f_historical_stories_qa_2020_08_25":3,

=================

Thanks