Cluster health turns yellow monthly

[Francesc G]

Hello,

Monthly, the cluster health turns yellow. 

When I list the unassigned shards, I always see "opendistro-alerting-history-yyyy.mm.dd" with one replica in UNASSIGNED status. In my case, I only have one node. 

To remove this replica, I have to edit "opensearch.yml" and set the "plugins.security.system_indices.enabled" parameter to "false" and change the "Number of replicas" setting to "0" in "Indexer Management" → "Index Management" → "Indexes"  →  "opendistro-alerting-history-yyyy.mm.dd"  → "Index Settings". 

How can I set the number of replicas to zero for this index by default so I don't have to do this every month?

Best regards.

[Isaac Yusuf]

Hello Francesc,

What is the command you use to set the replicas to 0? Ensure you add the auto_expand_replicas: false to ensure it does not happen again.

PUT .opendistro-alerting-*/_settings { "index":{ "number_of_replicas":0, "auto_expand_replicas":false } }

Additionally, three categories of settings exist in the cluster settings API: persistent, transient, and default. Persistent settings will persist after a cluster restart. After a restart, OpenSearch clears transient settings.
If you specify the same setting in multiple places, OpenSearch uses the following precedence:

1. Transient settings
2. Persistent settings
3. Settings from opensearch.yml
4. Default settings

PUT /_cluster/settings { "persistent" : { "opendistro" : { "index_state_management" : { "history" : { "number_of_replicas" : "0" } } } } }

​

[Francesc G]

I had to change the plugins.security.system_indices.enabled parameter to false in /etc/wazuh-indexer/opensearch.yml to run:

PUT .opendistro-alerting-*/_settings
{
  "index":{
     "number_of_replicas":0,

     "auto_expand_replicas":false

    }

}

After doing this it worked and the output was:

{
"acknowledged": true
}

After running:

PUT /_cluster/settings
{
  "persistent": {
    "opendistro": {
      "index_state_management": {
        "history": {
          "number_of_replicas": "0" 
        }
      }
    }
  }
}

The output was:

#! Deprecation: [opendistro.index_state_management.history.number_of_replicas] setting was deprecated in OpenSearch and will be removed in a future release! See the breaking changes documentation for the next major version.
{
  "acknowledged": true,

  "persistent": {
    "opendistro": {
      "index_state_management": {
        "history": {
          "number_of_replicas": "0"
        } 
      }
    }

  },
  "transient": {}
}

And this August the problem has been reproduced again: cluster turns yelow because "opendistro-alerting-history-yyyy.mm.dd " index replica is UNASSIGNED.

Any ideas?

Best regards.