retention policy removes all indexes

[Henry Valero]

I have configured the retention policy following this example:

https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html#index-life-management

but for the practical case I have applied that the indices have a minimum age of 3 days, and I applied the policy to indices from the last week, and this policy has eliminated all the indices that I have selected and now the maximum that remains an index is less than a hour, then deletes them, how can I correctly apply this retention policy so that it deletes indexes greater than 3 days (the 3 days are only for testing).

Atte.:

Henry

[Natalia Castillo]

Hi Henry!

Let's check a few things to identify better what could be the issue
- If the policy is retroactively applied to existing indices, make sure it’s correctly targeting those that are already older than 3 days to avoid immediate deletion.  

- Maybe it could be that the indices that get deleted after an hour, meet the condition and that's why they're getting deleted. Check the indices that are being deleted, to check if the deletion is occurring correctly (with the ones older than 3 days) or if there's a further problem.

You can check it by Navigating to ☰ > Indexer management > Dev Tools and running this command: 
GET /_cat/indices/wazuh-*?v&h=index,creation.date.string

This will show something like this and you could see if the indices met the criteria:
index                                     creation.date.string
wazuh-alerts-4.x-v1-2024.03.19-000032     2024-03-19T17:18:45.501Z
wazuh-alerts-4.x-v1-2024.06.04-000043     2024-06-04T17:36:29.689Z
wazuh-alerts-4.x-v1-2024.06.25-000046     2024-06-25T17:42:52.011Z

Check this few things and let me know how it goes!

[Henry Valero]

Hi Natalia,

The policy is apparently working in reverse, because it has eliminated the indices within 3 days and not the ones greater than three days.

Atte.:

Henry

[moosemaimer]

Check which of your States is the "Initial state"... I found a bit of an oversight in the documentation:

• Click Add state to create a state for index deletion. Enter a name such as delete_alerts.

• Click Add action and select Delete in the Action type. Click Add action. Then click Save state.

• Click Add state again to create an initial state. Enter a name, such as initial.

• Choose Add before from the Order tab and select delete_alerts.

• Click Add transition and select delete_alerts as the Destination state.

• Select Minimum Index Age in Condition. Input the retention value, for example, 90d for 90 days, in the Minimum Index Age.

• Click Add transition. Click Save state. Click Create.

The first state to be created is automatically set as the default, which happens to be the "delete" state. Make sure it's the other one, in the dropdown on the Edit page in the States section, labeled "Initial state."

[Henry Valero]

Hi moosemaimer:

I think there is also an error in the documentation, but I am not very clear about it, I attach how I have configured the states according to the official documentation.

The events keep being deleted constantly, how do I solve this issue?

Atte.:

Henry

[Natalia Castillo]

Hi!

You're absolutely right about the issue with the Visual editor steps in the Wazuh documentation. The main problem is that the deletion state is incorrectly presented as the initial state, which causes the indices to be deleted constantly. As moosemaimer mentioned, the initial state should not be set to deletion right away.

I strongly recommend using the JSON editor instead, as the steps are clearer and function correctly. The key difference is that the JSON guide correctly sets up a retention state first. This retention state ensures that the indices remain until they meet the specified conditions (like reaching a certain age), and only then do they transition to the deletion state.

To resolve this issue, you should delete the current ISM policy and reconfigure it using the JSON editor. Start by creating a retention state where the policy waits for the indices to meet the age condition. Once that condition is met, the policy can then move to the deletion state.

If you need any assistance in configuring the policy this way, feel free to reach out—I'm here to help!

[moosemaimer]