Delete unassigned shards

[Rai Ner]

Hi, 

after successfully upgrading from Wazuh OpenDistro for Elasticsearch to Wazuh OpenSearch/indexer there are 2 unassigned shards left. 

I have tried to delete them (not sure if this is the best solution) but it failed with limited permissions for the admin user. Please have a look. 

# Cluster health

      {
        "cluster_name" : "wazuh-elastic-prod",
        "status" : "red",
        "timed_out" : false,
        "number_of_nodes" : 3,
        "number_of_data_nodes" : 3,
        "discovered_master" : true,
        "active_primary_shards" : 313,
        "active_shards" : 600,
        "relocating_shards" : 0,
        "initializing_shards" : 0,
        "unassigned_shards" : 2,
        "delayed_unassigned_shards" : 0,
        "number_of_pending_tasks" : 0,
        "number_of_in_flight_fetch" : 0,
        "task_max_waiting_in_queue_millis" : 0,
        "active_shards_percent_as_number" : 99.66777408637874
      }

# Unassigned Shards: 
curl -XGET -k 'https://10.xx.127.xxx:9200/_cat/shards?h=index,shards,state,prirep,unassigned.reason' -u admin:xxxxx  | sort -n
.opendistro-alerting-config                             STARTED    p
.opendistro-alerting-config                             STARTED    r
.opendistro-anomaly-detection-state                     UNASSIGNED p NODE_LEFT
.opendistro-anomaly-detection-state                     UNASSIGNED r CLUSTER_RECOVERED

# More details about unassinged Shards: 
curl -XGET -k 'https://10.xx.127.xxx:9200/_cluster/allocation/explain?pretty' -u admin:xxxx
{
  "index" : ".opendistro-anomaly-detection-state",
  "shard" : 0,
  "primary" : true,
  "current_state" : "unassigned",
  "unassigned_info" : {
    "reason" : "NODE_LEFT",
    "at" : "2022-11-09T13:58:29.684Z",
    "details" : "node_left [4xD3yXhnTvm1x1mJbRBorg]",
    "last_allocation_status" : "no_valid_shard_copy"
  },
  "can_allocate" : "no_valid_shard_copy",
  "allocate_explanation" : "cannot allocate because a previous copy of the primary shard existed but can no longer be found on the nodes in the cluster",
  "node_allocation_decisions" : [
    {
      "node_id" : "44xOg6txSIOt89MYSA9qXw",
      "node_name" : "wazuh-el3",
      "transport_address" : "10.xxxxxxxxxxx:9300",
      "node_attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "Knb5hZhoQ8inmuvsBfGAxw",
      "node_name" : "wazuh-el2",
      "transport_address" : "10.xxxxxxxxxxx:9300",
      "node_attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    },
    {
      "node_id" : "Z98KPjvkSt2Y9mcIEAankg",
      "node_name" : "wazuh-el1",
      "transport_address" : "10.xxxxxxxxxxxxxxxx:9300",
      "node_attributes" : {
        "shard_indexing_pressure_enabled" : "true"
      },
      "node_decision" : "no",
      "store" : {
        "found" : false
      }
    }
  ]
}

# Deleting the unassigned shards which fail: 
curl -XGET -k 'https://10.xxxxxxxxx:9200/_cat/shards'  -u admin:xxxxxxxxx  | grep UNASSIGNED | awk {'print $1'}  | xargs -i curl -XDELETE -k -u admin:xxxxxxxxxxxxxx "https://10.xxxxxxxxxx:9200/{}"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 66804  100 66804    0     0   593k      0 --:--:-- --:--:-- --:--:--  593k
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status

Any ideas how to delete the unassigned shards and / or a better approach than deleting the unassigned shards? 

Many thanks

Rainer

[Victor Carlos Erenu]

Hello Rainer

Could you tell me the configuration of the cluster you have and if you have eliminated any node from the cluster in the migration? The most recommended thing would be to relocate those shards to a valid node.

I add a link where this error is referred to with the relocation solution.

https://devcoops.com/how-to-change-the-allocation-of-individual-shards/

[Rai Ner]

Hi Victor, 

thanks for the link, this is exactly about the issue. 

However I have some troubles to move the unassinged shards to one of the active nodes. In the POST command I need to specify the "from_node", if you refer to previous output it says the "node has left" and I don't know how to put this in "from_node"

curl -XPOST 'ES_Endpoint/_cluster/reroute' { "commands": [ { "move": { "index": "index-name", "shard": 0, "from_node": "node1", "to_node": "node2" } } ] }

If there is no easy answer, please let me know how to recreat an index, it is about: ".opendistro-anomaly-detection-state"

Many thanks

Rainer

[Rai Ner]

Hi, 

finally I could fix this issue by using "allocate_empty_primary", refer to https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html

Best regards, 

Rainer