ILM policy issue

[CJK]

Hi Team,

I have configured the attached ILM policy on wazuh 4.10.1 . And as per ILM policy Replica count is not getting 0 after one day as conigured on ILM. Can someone please review this and help me fix it?

Thanks

Clint

[CJK]

Hi Stuti,

Thanks for your response.

I didn’t add the replica count twice in any state. I’ve set the replica count to 1 for the Hot state and 0 for the Warm state, as you can see in my attached ILM.

I’ve also configured the index priority values as follows: 100 for Hot, 50 for Warm, and 10 for Cold.

According to the OpenSearch documentation, indexes with higher priority values are recovered first, followed by those with lower priority values:
https://docs.opensearch.org/2.19/im-plugin/ism/policies/#index_priority

I had some issues in my cluster, so my goal is to prioritize Hot indexes during recovery.

So my question is: Does index priority not work for Wazuh indices? Should I remove? That is the reson behind my issue - replica not getting 0 on State Warm ?

On Monday, 27 October 2025 at 09:09:55 UTC+5:30 Stuti Gupta wrote:

Hi CJK 

The ILM policy should be like this :
{
    "policy": {
        "description": " Index lifecycle policy  ",
        "default_state": "hot ",
        "states": [
            {
                "name": "hot ",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "replica_count": {
                            "number_of_replicas": 1
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_index_age": "1d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "replica_count": {
                            "number_of_replicas": 0
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "cold",
                        "conditions": {
                            "min_index_age": "7d"
                        }
                    }
                ]
            },
            {
                "name": "cold",
                "actions": [
                    {
                        "retry": {
                            "count": 10,
                            "backoff": "exponential",
                            "delay": "3m"
                        },
                        "read_only": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "wazuh-alerts-4.x-*"
                ],
                "priority": 1
            }
        ]
    }
}

You don't need to add the "index_priority": { "priority": 5} after every state, and the replica count twice in a state. To know more about ILM policy, please refer to: 
https://www.elastic.co/docs/manage-data/lifecycle/index-lifecycle-management
https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/index-lifecycle-management.html

[CJK]

Hi Team,

Can some one help me on this?

[Stuti Gupta]

Index_priority works for wazuh indices, no need to remove that.

Can you please share the number of nodes you have? 

Please share the output of the following commands to check if the policy is applied to the current indices 

GET _plugins/_ism/explain/wazuh-alerts-4.x-2025.10.27?pretty=true
GET wazuh-alerts-4.x-2025.10.27/_settings?filter_path=**.number_of_replicas
GET _cat/shards | grep wazuh-alerts-4.x-2025.10.27
POST _plugins/_ism/change_policy/wazuh-alerts-4.x-2025.10.27
{
  "policy_id": "wazuh-alert-data-tier",
  "state": "hot"
}


Make sure to run these commands on the Dev tools of Index management 

[CJK]

Number of nodes 3  wazuh 4.10.1 deployed on kubernetees.
Please find the attached results. And you sure about 4? and on 3rd command  "| grep" is not working on dev tools so i have added similar result.

[Stuti Gupta]

It looks like the warm transition is working fine, as after hot stage, the replica is 1, but the warm actions are waiting for the min index age condition to be fully met the condition and for ISM to run

To confirm this, can you please share the output of:

GET _plugins/_ism/explain/wazuh-alerts-4.x-*

You can specify the date of any indices that should have 0 replicas by now. 

We want the values for the latest index (index_creation_date, state.name, step_status).

That will tell us whether the replica_count action simply needs more time to execute or if there is anything else delaying it.

[Stuti Gupta]

Meanwhile, I am also testing this from my end 

[CJK]

Hi Stuti,

Thank you for your support.!!

GET _plugins/_ism/explain/wazuh-alerts-4.x-2025.10.2* - I have attached the results 

But one thing is due to disk watermark issues, I am manually removing replicas using the below request after some days. I have waited for 5 days but all had replica 1 even those moved to warm as per my ILM then disk will get 95%. So I am removing replicas manually now. Today I have removed remaining replicas already.

PUT wazuh-alerts-4.x-2025.10.27/_settings
{
"index.number_of_replicas" : 0,
"index.auto_expand_replicas": false
}

Pfa..

[Stuti Gupta]

Hi CJK,

I’ve tested this as well and got the same results as you.
I’ve opened a GitHub issue to track it: https://github.com/wazuh/wazuh-indexer/issues/1199.

You can follow the issue there and add any additional information or observations from your side.

[CJK]

Hi Stuti,

Thank you so much for your support. Sry for the late response.

So what i found from Issue tracker is - "However, if we remove the "index.auto_expand_replicas": "0-1" setting from the template, or set it to 0, we can achieve the "expected" behavior."

So we have to remove "index.auto_expand_replicas": "0-1" from the template. How to do the same? Is this the default wazuh template we are talking about?

I have checked on the cluster settings using this - "GET /_cluster/settings?include_defaults=true&flat_settings=true" but not able to find any. Can you help me to change the same properly? 

and any suspecetd issues if we remove this from default template?

Also on - "GET _index_template" i have only "wazuh-states-vulnerabilities-wazuh_template"

So workaround in my uderstanding is

------

Remove "index.auto_expand_replicas": "0-1" in template (Replica 1 for HOT index will be allocated by action operation on HOT state of ISM)

Then on state warm - Replica will be 0 based on ISM warm action.

Thanks.

Clint

[Stuti Gupta]

Hi CLK 

Yes, Alex is correct. The real cause was the index template’s index.auto_expand_replicas setting, which was set to "0-1". This setting automatically adjusts the number of replicas whenever cluster nodes can allocate them, effectively overriding the ISM policy’s replica configuration. To change that follow these steps:
You can chnage the setting :
Edit vi /etc/filebeat/wazuh-template.json to set index.auto_expand_replicas" to 0. Change the value from 0-1 to 0

{
  "order": 1,
  "index_patterns": [
    "wazuh-alerts-4.x-*",
    "wazuh-archives-4.x-*"
  ],
  "settings": {
    "index.refresh_interval": "5s",
    "index.number_of_shards": "1",
    "index.number_of_replicas": "0",
    "index.auto_expand_replicas": 0",
    "index.mapping.total_fields.limit": 10000,
    ...

Load the new settings.

curl -X PUT "https://<INDEXER_IP_ADDRESS>:9200/_template/wazuh-custom" -H 'Content-Type: application/json' -d @w-indexer-template.json -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD>

Output
{"acknowledged":true}


Confirm that the configuration was successfully updated.
curl "https://<INDEXER_IP_ADDRESS>:9200/_template/wazuh-custom?pretty&filter_path=wazuh-custom.settings" -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD>
Output
{
  "wazuh-custom" : {
    "settings" : {
      "index" : {
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "5s",
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0",
        "number_of_replicas" : "0",
        ...

If the index has already been created, it must be https://documentation.wazuh.com/current/user-manual/wazuh-indexer/re-indexing.html


The command in the dev tool to verify the index.auto_expand_replicas is: GET /_template

[CJK]

Hi Stuti,

Thanks.