Elassandra version 6.2 - Yellow state on some nodes due to out of order operation on alias create. How to unwind operation in metadata table?
Kathryn Soderstrom
Hello –
We are running Elassandra version 6.2 in a 5 node cluster. We attempted to create an alias with the command below.
We attempted to atomically swap an index with an alias as described in the Elasticsearch documentation: Index Aliases | Elasticsearch Reference [6.2] | Elastic
curl -XPOST http://localhost:9200/_aliases -H "content-type: application/json" -d '{"actions": [{"add": {"index": "xxx_v019", "alias": "xxx"}}, {"remove_index": {"index": "xxx"}}]}'
Inspecting logs on the individual nodes showed that the atomic drop index/create alias operation had completed successfully on one node, but failed on the other 4. Logs seems to indicate that the create alias step executed on the other nodes BEFORE the contact index was actually deleted, see below:
2021-06-29 16:19:57,378 ERROR [MigrationStage:1] CassandraDaemon.java:250 uncaughtException Exception in thread Thread[MigrationStage:1,5,main]
2java.lang.IllegalStateException: index and alias names need to be unique, but the following duplicates were found [contact (alias of [contact_v019/fEaj_OgLSxGzY0kCWeqcsw])]
3 at org.elasticsearch.cluster.metadata.MetaData$Builder.build(MetaData.java:1092)
4 at org.elassandra.cluster.SchemaListener.onEndTransaction(SchemaListener.java:106)
5 at org.apache.cassandra.service.MigrationManager.notifyEndTransaction(MigrationManager.java:221)
6 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1474)
7 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1437)
8 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1409)
9 at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1404)
10 at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51)
11 at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
12 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
13 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
14 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
15 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
16 at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
17 at java.lang.Thread.run(Thread.java:748)
We now have 4 of the nodes where the index we were intending to alias is in Yellow state. The metadata version is advanced on the one node, but the other nodes fail to apply the change on replay because the transaction is played out of order. This seems to be blocking additional changes because it replays and fails the alias swap before attempting the new requests.
When we replicated this issue in a test environment, we were able to update the elastic_admin.metadata_log to the version prior to the aliasing, but could find no way to rollback the state on the node that had advanced a version. We restarted it in an attempt to sync it with the other nodes to the earlier state, but the node came back with 0 indexes and a metadata version of 0 (confirmed by _cluster/state and X2 gossipinfo).
The restarted node was without indices and the only way we could get indices to show up was to post a new “Dummy” index to it, at which point it appeared to restore it’s old indices and add the new one. Having the node come up without indices is undesirable in production, and having to post a dummy index to get it to sync state is not an ideal recovery step. Is there a proper way to handle this rollback and force index sync without having to post an uncessary index operation?
Our question how do we effectively unwind (rollback / refresh) the operation in the meta data table to get the nodes back to green state?