ILM policy update to reduce index volume on nodes

[CJK]

Hi Team,

We have an ILM policy that has been recently updated to reduce the number of days  indices are moved to the cold, frozen and delete phase.

The issue is that existing indices, which are already attached to the updated policy, are not transitioning based on the new, shorter duration. They appear to be sticking to the original, longer timeline.

Our indices average 125 GB in size. We cannot solve this by detaching and re-attaching the ILM policy, as this would force the indices to restart from the beginning. This would be extremely time-consuming and risks breaking nodes disk space limits by moving old one to hot nodes again.

Kindly suggest a solution to get indexes properly mapped to current ILM policy without restarting their entire lifecycle. Or breaking the disk watermarks

Wazuh version: 4.7.5 - Indexer nodes = 3 

Please find the current ILM policy attached.

Note: previous ILM have

hot to warm transition min index age - 7
warm to cold transition min index age - 79
cold to frozen transition min index age - 91

frozen to delete min age - 130. 

Update made to ILM is just changed trasitions dates to - 7, 61, 71 & 110 respectively


Thanks.

Clint

[Stuti Gupta]

Hi CJK

Your updated ILM policy looks correct, and because the structure is the same (only the min_index_age values were changed), ISM will use the new policy.

However, existing indices do not automatically update to the new shorter timings. If an index has already passed the older, longer time, ISM will not move it based on the new age settings.

To bring the existing indices in line with the updated timeline, without restarting their lifecycle and without moving indices back to the hot nodes,

You can use the API command /_plugins/_ism/change_policy.

Go to 
Index Management -> Dev Tools

Run the following command:

POST _plugins/_ism/change_policy/<index name>
{
  "policy_id": "<ILm Policy ID>",
  "state": "delete",
  "include": [
    {
      "state": "searches"
    }
  ]
}

In this example, the policy is applied to the <Index>. The index is changed to <policy_id>, which is an updated version of its existing policy. The process only applies the change if the index is currently in the searches state. After this change in policy takes place, <Index>transitions to the delete state

It simply updates the state pointer.

New indices will follow your updated policy timings (7 → 61 → 71 → 110 days) automatically.

Refer to 
https://docs.opensearch.org/latest/im-plugin/ism/api/#update-managed-index-policy
https://documentation.wazuh.com/current/user-manual/indexer-api/reference.html#tag/Index-state-management-API/operation/updateManagedIndexPolicy