Logs gets stopped reflecting in Wazuh dashboard

[Herish R Chovatiya]

Hi Team,

I want to understand why logs get stops appearing in wazuh dashboard alerts even though the disk space is available?

The logs starts appearing as soon as I delete the alerts indexes from index management as shown in attached snapshot.

Could you please help with what exactly I have to do and what causing the log stoppage issue in dashboard?

Thank You!

[hasitha.u...@wazuh.com]

Hi Herish,

I believe your indexer cluster reached the max shard limit. A single indexer node can have a maximum of 1000 shards by default.
Please confirm if that's the case by navigating to Indexer management -> Dev Tools
GET _cluster/health

If yes, then follow the instructions below to resolve your issue.
A quick fix is increasing the shards limit per node.
It is possible to change the setting using the WI API. You can either use the Dev tools option within the indexer management section in the Wazuh Dashboard: 
Indexer management -> Dev Tools

1. PUT _cluster/settings
2. {
3.   "persistent" : {
4.     "cluster.routing.allocation.total_shards_per_node" : 1100
5.   }
6. }

or curl the API directly from a terminal:
curl -X PUT https://<elasticsearchip>:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "1100" } }' -k -u admin:pass
But it is not recommended as increasing it too high can cause for the long run as it will bring more problems in the future. However, this guide will explain how to do it in case it is needed.

But as you have more left in your server, I suggest you tune your primary and replica shard numbers.
I suggest you change the number of primary shards to 1, following this document.
https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#setting-the-number-of-shards
The drawback of this solution is that you need to reindex all the existing indices one by one to apply the changes to the old indices.
https://documentation.wazuh.com/current/user-manual/wazuh-indexer/re-indexing.html

You can also add another indexer node or delete some old indices
from your server to free up some shard space.
To add more Wazuh indexer nodes, follow this document:
https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/add-wazuh-indexer-nodes.html
To delete old indices, go to
Index Management > Indices
Search with Wazuh-alerts
Select the indices you want to delete
Click on Action and select Delete from the drop-down.
Check the screenshot for reference.

Additionally, you can create an ILM policy to delete old alerts and keep the Wazuh Indexer running smoothly. You can refer to the Wazuh ILM policy documentation.
Also consider taking a snapshot before deleting indices, as this will allow you to restore data later if needed. Refer to Wazuh documentation for snapshot procedures.

Let us know the update on this issue.