Wazuh Dashboard Not Showing Security Events

[John]

Greetings - My Wazuh dashboard suddenly stopped showing security related events on 3/27/23, although Wazuh agents are active/running on endpoints, logs are still being generated, and the dashboard seems functional -- just no data.

- I started by running systemctl restart wazuh-dashboard, then systemctl status wazuh-dashboard:

● wazuh-dashboard.service - wazuh-dashboard
   Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2023-03-29 12:18:15 UTC; 29s ago
 Main PID: 29661 (node)
   CGroup: /system.slice/wazuh-dashboard.service
   └─29661 /usr/share/wazuh-dashboard/bin/../node/bin/node --no-warnings --max-http-head...

Mar 29 12:18:38 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...}
Mar 29 12:18:38 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...p
Mar 29 12:18:39 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...}
Mar 29 12:18:39 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...}
Mar 29 12:18:39 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...e
Mar 29 12:18:41 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...}
Mar 29 12:18:41 wazuh-server opensearch-dashboards[29661]: {"type":"log","@timestamp":"2023-03-...}
Hint: Some lines were ellipsized, use -l to show in full.

- I do not have any filters applied in my searches

- Next, I ran /var/ossec/bin/wazuh-control status:

wazuh-clusterd not running...
wazuh-modulesd is running...
wazuh-monitord is running...
wazuh-logcollector is running...
wazuh-remoted is running...
wazuh-syscheckd is running...
wazuh-analysisd is running...
wazuh-maild is running...
wazuh-execd is running...
wazuh-db is running...
wazuh-authd is running...
wazuh-agentlessd not running...
wazuh-integratord is running...
wazuh-dbd not running...
wazuh-csyslogd not running...
wazuh-apid is running...

- Next, I ran curl -k -X GET "https://localhost:55000/" -H "Authorization: Bearer $(curl -u wazuh-user:wazuh -k -X POST 'https://https://localhost:55000/security/user/authenticate?raw=true')" :

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: https; Unknown error
{"title": "Unauthorized", "detail": "No authorization token provided"}

- Then I ran cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn" which results in many errors like:

{"date":"2023-03-29T11:49:15.566Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED x.x.x.x:55000"}


- Curiously, from the wazuh -> Tools -> API Console, I can run requests fine. For example, these requests return valid data:

GET /agents?status=active
GET /manager/info
GET /syscollector/000/packages?search=ssh&limit=1
PUT /logtest

I cannot figure out why my data dropped off the cliff on 3/27/23. Have I hit some kind of index or shards limit?

Thank you!

[Bin Do Tuan Anh]

Hi, 

Please let me know if your agents are still online from the Wazuh Manager perspective. 

In case they are displayed as online, we will need to check if the alerts.json file getting alerts from your agents. You will need to check that file by running following command:

tail -f /var/ossec/logs/alerts/alerts.json

The data should keep coming to your output. 

Then, we can check if the filebeat read the file alerts.json. To do it please run the command: 

lsof /var/ossec/logs/alerts/alerts.json

You should be able to see filebeat there.

Also, I would check the communication between Filebeat and Wazuh Indexer with the command: 

filebeat test output

After that I would check the status of the environment. You will need to go to Dev Tools. In the Wazuh Dashboard main menu please go to Dev Tools (under Management section) and run following command: 

GET _cluster/health?pretty

Please check the status and active_shards. In case you have amount of shards close to 1000 - the issue may related to the maximum amount of shards per node (that is by default is 1000). 

It is recommended to set the Index Lifecycle Management. This way you will be able to remove the data that you do not need anymore according to your retention policy.
You can check more about it here: https://wazuh.com/blog/wazuh-index-management/
At the same time, please let me know how many Wazuh Indexer nodes you have. In case you have only 1 node, you would need to change the index per shard. By default you have the value 3, for a single Wazuh Indexer node you only need to have 1. To change that you need to go to the file: /etc/filebeat/wazuh-template.json and change the value of the "index.number_of_shards" to 1.

And additionally you can increase the maximum amount of shards. To do it you need to go to Dev Tools and run the command
PUT _cluster/settings
{
  "persistent": {
    "cluster.max_shards_per_node": 2000
  }
}

Please check the information above and let me know if you have any questions.

Best regards,

Bin.