Lost access to Wazuh

[Bobby Garraway]

Hi, 

Wondering if I can get some assistance here. 

I currently cannot log into the Wazuh GUI. This is due to getting the below message:

 

This is caused by not having Wazuh-Indexer.service running our on our Indexer. 

I cannot run it due to the storage space on the machine exceeding 95%. 

So, my question is, how or what can I clear off of the indexer to allow me to run the service again. I have scoured the internet for an answer and everything points me to using the Dev Tools which I don't have access too or using an API Curl which is troublesome without specific names. 

The Wazuh setup we have is not all in one as it is distributed. It was setup a while ago but never fully finalised as we got busy dealing with other issues. This has led to the server pulling down indices and collecting logs and is now full. 

After clearing enough space to finally start the service to bring the GUI back up, I then need to reset the admin password to login to the GUI. The password tool is also failing due to storage space, so that is also not working currently either. 

Wazuh-Server and Wazuh-Dashboard on their own servers are working as intended currently. Services are able to start and working as intended. 

TL:DR - What, how and where can I delete from the Indexer via Linux CLI to create enough storage space to 1. Bring the service back online then 2. Reset the passwords to log back into the admin GUI to setup rules to prevent this happening in the future. 

[Stuti Gupta]

Hi Bobby,

If your indexer is down, you won’t be able to run any curl commands from the Linux CLI either. In this situation, the first priority is to free enough disk space so the indexer service can start again.

If this is an all-in-one deployment

You can either increase the storage or delete old logs, as well as remove agents that are no longer in use. This will immediately free some disk space and allow the indexer to start.
Once the indexer is running again, you can delete old indices through the API, adjust shard/replica settings, and configure ILM policies.

If this is a multi-node deployment

You can increase storage or delete old logs on the indexer node. In case this didn't resolve the issue,I recommend deleting old indices that are consuming most of the space.

To identify which indices are safe to delete, run this command on the indexer node:
sudo du -sh /var/lib/wazuh-indexer/nodes/0/indices/* | sort -hr | head -20

This shows the largest index directories.Then identify the index name inside each folder:
strings /var/lib/wazuh-indexer/nodes/0/indices/<hash>/_state/state* | grep -i index
(Replace <hash> with the directory name you found from the previous command.)

You will see output similar to:
index.creation_date: 1769144792736
index.number_of_replicas: 0 index.number_of_shards: 1
index.provided_name: <index-name>

If the index is old and safe to remove, delete it:
sudo rm -rf /var/lib/wazuh-indexer/nodes/0/indices/<hash>

Note: Be careful while performing these steps. Make sure you do not delete any important or actively used indices, as this can affect the indexer node.

After freeing space, try starting the indexer again. Once it comes up, to prevent the issue from happening again:
Review and adjust replicas and shard configurations: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html
You can also follow this guide to recover old logs once the issue is resolved: https://wazuh.com/blog/recover-your-data-using-wazuh-alert-backups/
Set up Index Lifecycle Management (ILM) policies: https://wazuh.com/blog/wazuh-index-management
You can also configure snapshots to back up your indices to local or cloud storage: https://wazuh.com/blog/index-backup-management
If you plan to scale the cluster in the future, add indexer nodes: https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/add-wazuh-indexer-nodes.html