wazuh alert retention policy not working

[Nadita Candra]

Hello Wazuh Team,

i'm trying to apply log retention policy to my wazuh alert indices, here the JSON format of my conf :

{ "id": "wazuh-alert-retention-policy", "seqNo": 1849, "primaryTerm": 3, "policy": { "policy_id": "wazuh-alert-retention-policy", "description": "wazuh-alert-retention-policy (HOT, WARM, COLD and DEL)", "last_updated_time": 1702270397379, "schema_version": 17, "error_notification": null, "default_state": "initial", "states": [ { "name": "initial", "actions": [], "transitions": [ { "state_name": "delete_alerts", "conditions": { "min_index_age": "60d" } } ] }, { "name": "delete_alerts", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "wazuh-alerts-*" ], "priority": 1, "last_updated_time": 1702262445726 } ] } }

and here is my last and past wazuh alert indices whic already passes 60 Days :
But the old indices not yet delete like the Transitions condition i made. On the Policy Managed Indices, all indices from old to the new have the same status :

Managed Index Info

{ "message": "Evaluating transition conditions [index=wazuh-alerts-4.x-2023.10.21]" }

how can i fix this so i don't need to delet the indices manually.

{ "id": "wazuh-alert-retention-policy", "seqNo": 1849, "primaryTerm": 3, "policy": { "policy_id": "wazuh-alert-retention-policy", "description": "wazuh-alert-retention-policy (HOT, WARM, COLD and DEL)", "last_updated_time": 1702270397379, "schema_version": 17, "error_notification": null, "default_state": "initial", "states": [ { "name": "initial", "actions": [], "transitions": [ { "state_name": "delete_alerts", "conditions": { "min_index_age": "60d" } } ] }, { "name": "delete_alerts", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "wazuh-alerts-*" ], "priority": 1, "last_updated_time": 1702262445726 } ] } }

[Stuti Gupta]

Hi team!
Thank you for using wazuh.

Please allow me some time. I'm looking into this query and will update you with an appropriate answer.

Regards,

[Nadita Candra]

no worries, i'm already deleted some indices manually :D, because my avaiable disk space is below 5%

[Stuti Gupta]

Hi Nadita,
Hope you are doing well and thank you for using wazuh.

I have tested your retention policy  and is working fine.
I followed the following steps:
1. Copied the retention policy that you have created. at Json editor, The policy is: { "policy": { "policy_id": "delete retention policy", "description": "Wazuh alerts retention policy", "schema_version": 18, "error_notification": null, "default_state": "initial", "states": [ { "name": "initial", "actions": [], "transitions": [ { "state_name": "delete_alerts", "conditions": { "min_index_age": "60d" } } ] }, { "name": "delete_alerts", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "wazuh-alerts-*" ], "priority": 1, } ] } }
2. I applied this policy to indexes  that are 30d old (for testing purposes).  For that, i choose Indices in Index Management. Select the index or indices to attach the policy. Click Actions > Apply policy.

3. After refreshing the Policy managed indices after 30 min it started deleting the indices. 

4. Then you can check the indices that is deleted.

As you can see the in the image the indices have been deleted.

Please verify that the indices you applied to the policy are 60 days old for the deleting stage as you mention 60 say minimum indexes age.

Reference: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html

Hope this helps.
Regards,

[Nadita Candra]

Dear Stuti Gupta,
thx for assist me before, after waiting the 60d indices come and i can observe that the policy now is working, i dpn't know what the differences steps between the last any from your, it's looks all the same. :D, once again i grateful, hope you always in good shape.

[Nadita Candra]

Hi Stuti Gupta, sorry to have to reply to this message again because after a long holiday, this issue has not been resolved. wazuh-alert-retention-policy won't work again. I checked that the hard disk capacity on the Wazuh server had reached 95% and any changes to the Wazuh dashboard resulted in the error "can't change because the space is in read-only mode", so I tried to check the cause of the previous wazuh-alert-retention-policy worked removing wazuh alert indices but now it doesn't work. It seems that wazuh-alert-retention-policy only deletes logs in /var/ossec/logs/archives, but not the /var/ossec/logs/alerts directory. In this directory the logs take very large amount of hard disk space, causing the capacity to be full before the policy can run automatically. i can say this because i still see logs from September 2023 where the wazuh alert indices have long been deleted.
My question is, does wazuh-alert-retention-policy not delete logs in the /var/ossec/logs/alerts directory? How can I make sure that this directory is also deleted automatically?

NB:after back from holiday i can't analyze alerts from 28th Dec, i lost many days :-(

[Stuti Gupta]

Hi Nadita,
By default, the Wazuh server retains logs and does not delete them automatically. However, you can choose when to manually or automatically delete these logs according to your legal and regulatory requirements. 
These files were stored on the path /var/ossec/logs/archives/ and they rotate and compress in the same way that I explained before.
You can add a command to the crontab in order to schedule the deletion according to your criteria.
i.e.
 Cronjob to remove older alerts
0 0 * * * find /var/ossec/logs/alerts/ -type f -mtime +365 -exec rm -f {} \;
0 0 * * * find /var/ossec/logs/archives/ -type f -mtime +365 -exec rm -f {} \;
Cronjob to move older alerts to another location
0 0 * * * find /var/ossec/logs/alerts/ -type d -mtime +90 -regex '\/var/ossec/logs/alerts\/[0-9]+' -exec cp -rp {} </path/to/NAS/> \; -exec rm -rf {} \;
0 0 * * * find /var/ossec/logs/arvhives/ -type d -mtime +90 -regex '\/var/ossec/logs/alerts\/[0-9]+' -exec cp -rp {} </path/to/NAS/> \; -exec rm -rf {} \;
Cronjob to compress uncompressed JSON alerts
0 8,20 * * * find /var/ossec/logs/archives/ -type f -regex '\S+ossec-archive-\S+-\S+.json' -exec gzip {} \;
0 8,20 * * * find /var/ossec/logs/alerts/ -type f -regex '\S+ossec-alerts-\S+-\S+.json' -exec gzip {} \;
You can take this as a reference and prepare a cronjob based on your required use case. You can also review Crontab – Quick Reference to have a brief idea about how crontab works.
Note: If you have archives enabled, the rotated archives logs will be stored in the /var/ossec/logs/archives/ directory similarly. Therefore, you can follow the same process to move the archives log as well.
Please review this to understand how Wazuh log data collection works: https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/how-it-works.html

Hope this helps,
Regards,