Wazuh-indexer Indices taking too much space

[Haziq Mt Roslan]

Hi team,

/var/lib/wazuh-indexer/nodes/0/indices/

I have a question regarding wazuh-indexer indeces, it taking so much space about 80% of the storage is consume by the indeces. I try delete it previously and i found out it also remove the alert in the wazuh dashboard. In my current system, we already transfer the alert logs to backup server for our compliance. 

So my question is, if the alert log already transfer is it ok to delete the indices. From my understanding if we delete the indeces before transfer the alert log it will delete it also. Am i correct? By deleting the indeces, it also delete the alert we can view on dashboard and also alert.log/alert.json correct?

So meaning if we want to housekeep  we need to transfer the .log/.json and also remove the indeces that already remove? And can you provide a way to housekeep efficiently? Thats all from me. Thanks in advance.

[Stuti Gupta]

Hi team!
Please allow me some time. I'm looking into this query and will update you with an appropriate answer.

[Haziq Mt Roslan]

ok thank you

[Stuti Gupta]

Hi Haziq Mt Roslan.

Yes if the alert.log is transferred it is okay to delete the indices. However, deleting the indices does not delete the alert.log or alert.json from the server,  you need to delete them manually.

The Wazuh server analyzes events received from monitored endpoints and generates alerts when the events match a detection rule. These alerts are saved using the wazuh-alerts-* indices.The Wazuh server logs the alert data into the /var/ossec/logs/alerts/alerts.json and /var/ossec/logs/alerts/alerts.log files by default. Once saved in the /var/ossec/logs/alerts/alerts.json file, it forwards the JSON alert document to the /var/lib/wazuh-indexer/ directory of the Wazuh indexer for indexing. When forwarding alerts to the Wazuh indexer, the Wazuh server formats the current date into an index name.

For you next question:  also remove the indeces that already remove?
You dont need to remove the indexes again. It will be removed at once. For that, you can either delete it manually or you can create ILM policy. To retrieve the list of indices, you can use the following command:
curl -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> https://<INDEXER_IP_ADDRESS>:9200/_cat/indices/wazuh-*?v
This command will provide a detailed view of all indices related to "wazuh," facilitating decision-making regarding their management. To delete the indices manually you can run the following command:
curl -X DELETE -k -u <INDEXER_USERNAME>:<INDEXER_PASSWORD> https://<INDEXER_IP_ADDRESS>:9200/<INDEX_NAME>

 It is necessary to check what the indices stored in the environment, the following API call can help:
GET _cat/indices
Then, it is necessary to delete indices that are not needed or older indices. Bear in mind that this cannot be retrieved unless there are backups of the data either using snapshots or Wazuh alerts backups.
The API call to delete indices is:
DELETE <index_name>

You can create policies that govern the lifecycle of the indices based on different phases.
Four phases can be defined in a Lifecycle Policy:
Hot phase. For recent data that is actively accessed.
Warm phase. Data that you may wish to access, but less often.
Cold phase. Similar to the warm phase you may also freeze indices to reduce overhead.
Delete phase. Data that reaches this phase is deleted.
You can follow the steps mentioned in this document https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html.

By default, the Wazuh server retains logs and does not delete them automatically. You need to delete the log entry from file /var/ossec/logs/alerts/ or d /var/ossec/logs/archives/. for thatYou can also create cronjob, In other words, in Wazuh servers, you can create a cronjob that will act as a retention policy like this:
crontab -e
Add these two new lines at the end:
1 0 * * * find /var/ossec/logs/alerts/ -type f -mtime +360 -exec rm -f {} \;
1 0 * * * find /var/ossec/logs/archives/ -type f -mtime +180 -exec rm -f {} \;
This will execute the tasks every day at 00:01 am for Crontab to delete files in alerts/archives older than 360/180 days. Bear in mind that archive files could be really big in size.
For complete information: Documentation and if you are using linux you can use the following Linux command with >
> /var/ossec/logs/alerts/alerts.json

 You can also take snapshots of the indices that automatically back up your Wazuh indices in local or Cloud-based storage and restore them at any given time. To do so please refer to https://wazuh.com/blog/index-backup-managementhttps://wazuh.com/blog/wazuh-index-management/

Hope this helps