Wazuh Dashboard not showing events

[K&K Gmail]

Hello!

For a couple of months already, my wazuh instance doesn't show any events properly.

I can only see events from the last 10 to 15 minutes.

What i already tried:

# 01) pasted a command to change max-shards-per-node to 3000

curl -X PUT localhost:9200/_cluster/settings -H "Content-Type: application/json" -d '{ "persistent": { "cluster.max_shards_per_node": "3000" } }'

-> this command somehow made the event show up again, but it only lasted for 9 days.

# 02) deleted all Indices, started from the oldest and checked for any changes

wazuh GUI > Index Management > Indice

-> didnt changed anything useful. It just deleted all the old events.

I did a bit of research and found 3 commands that could be useful:

 

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

[2024-07-01T09:31:07,482][ERROR][o.o.s.i.DetectorIndexManagementService] [node-1] info deleteOldIndices

[2024-07-01T09:31:07,482][ERROR][o.o.s.i.DetectorIndexManagementService] [node-1] info deleteOldIndices

[2024-07-01T09:31:07,505][ERROR][o.o.a.a.AlertIndices     ] [node-1] info deleteOldIndices

[2024-07-01T09:31:07,505][ERROR][o.o.a.a.AlertIndices     ] [node-1] info deleteOldIndices

 

cat /var/log/filebeat/filebeat* | grep -i -E "error|warn" cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

/var/ossec/logs/ossec.log:2024/07/01 10:40:34 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.

/var/ossec/logs/ossec.log:2024/07/01 10:40:34 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.

/var/ossec/logs/ossec.log:2024/07/01 10:40:36 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.

/var/ossec/logs/ossec.log:2024/07/01 10:40:36 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.

/var/ossec/logs/ossec.log:2024/07/01 10:40:38 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.

<…snipped…>

 

 

 

journalctl -xeu wazuh-dashboard --no-pager | grep -iE "error|warn" cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

/usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log:{"data":{"message":"validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;","stack":"ResponseError: validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1358:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)"},"date":"2024-04-05T08:15:00.944Z","level":"info","location":"Cron-scheduler"}

/usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log:{"date":"2024-04-05T07:45:00.493Z","level":"error","location":"monitoring:createIndex","message":"Could not create wazuh-monitoring-2024.14w index on elasticsearch due to validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

/usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log:{"date":"2024-04-05T07:45:00.494Z","level":"error","location":"monitoring:insertMonitoringDataElasticsearch","message":"index_not_found_exception: [index_not_found_exception] Reason: no such index [wazuh-monitoring-2024.14w]"}

Any idea whats wrong with the current setup?

Thank you in advance

[Luis Daniel Avendaño Larios]

Hi,

The error you came across indicates that the cluster has hit the maximum shard limit of 1,000. When this limit is reached, new data, like security events, cannot be indexed. You noted that increasing the max_shards_per_node temporarily resolved the problem, but this fix only lasted for 9 days. Consider implementing a strategy for index management to automatically deal with older indexes.

Please follow this guide that details how to create a retention policy, with this retention policy you will ensure that the indexes will rotate periodically therefore you won't be hitting the max shards again:
https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html

Regards, 

Luis Avendaño