Wazuh Dashboards Failed to Start Due to Shard Limit Reached (1000/1000)

[Le Sok]

Dear Wazuh Support Team,

I am encountering a critical issue with Wazuh Dashboards on our server. The service fails to start with the following fatal error: FATAL  {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"},"status":400}

As a result, I am unable to access the Wazuh Dashboards, and running:  
curl -X GET "localhost:9200/_cluster/health?pretty"
Returns:
curl: (52) Empty reply from server
and I can't login to dashboard so I need to delete indexces from command line 
Wazuh install from OVA follow this link https://documentation.wazuh.com/current/deployment-options/virtual-machine/virtual-machine.html

Thank you in advance for your support.

Best regards,

[Bony V John]

Hi,

As you mentioned, the issue you're facing is related to the shard limit being reached, and to resolve it, you'll need to delete old shards from the Wazuh indexer. Since your Wazuh dashboard is currently down, you won't be able to delete old indices through the dashboard. Therefore, you'll need to run the necessary commands from the backend.

Regarding the curl command you executed, there is a mistake in your command, which is why the output appears as it does.

First, ensure that your server has sufficient resources. You can check this using the following commands:

Disk usage: df -h

Memory usage: free -h

CPU usage: top

If the server lacks resources, you’ll need to allocate more to the Wazuh server accordingly.  

To check the health of your Wazuh Indexer cluster, run the following command:

curl -k -u <user>:<password> -X GET https://localhost:9200/_cluster/health?pretty

Replace <user> and <password> with your admin credentials.
If you're running the command from a different server, replace localhost with the Wazuh indexer's IP address.  

To identify the old indices that can be deleted, use the command:

curl -k -u <User>:<Password> -X GET https://localhost:9200/_cat/indices/wazuh-alerts-4.x-*?v

This will display a list of all relevant Wazuh indices as shown below:

Once you've identified the indices to be deleted, you can remove them one by one using:  

curl -k -u <User>:<Password> -X DELETE https://localhost:9200/<index_name>

For example, to delete the index for February 11, 2025:  

curl -k -u admin:admin -X DELETE https://localhost:9200/wazuh-alerts-4.x-2025.02.11

You should receive a response like:

{"acknowledged":true}

If you want to delete all indices from February 2025, run:  

curl -k -u admin:admin -X DELETE https://localhost:9200/wazuh-alerts-4.x-2025.02*

After deleting the necessary indices, restart the Wazuh dashboard service:

systemctl restart wazuh-dashboard

Check its status:

systemctl status wazuh dashboard

Note: - Deleting indices from the Wazuh indexer will permanently remove old data. You won’t be able to search for that data in the Wazuh dashboard afterward. For example, if you delete wazuh-alerts-4.x-2025.02.11, you won’t be able to retrieve alerts from February 11, 2025.

Once the Wazuh dashboard is back online, consider applying an ILM (Index Lifecycle Management) policy to automatically delete old indices. This will help prevent similar issues in the future.

You can refer to the Wazuh documentation on ILM policies for detailed configuration guidance.

[Le Sok]

Dear Team,

I would like to ask for guidance on the following:

1. Is it possible to download the Wazuh indices (log data) from the server and store them locally on my machine?
   If yes, what is the recommended method or tool to perform this export securely and efficiently?

2. Once the indices are downloaded, how can I view or analyze those logs locally using Wazuh tools or alternatives?
   For example, can I import them into Wazuh visualize the data?