Wazuh dashboard stopped showing alerts

[J.Carlos A]

Hi, 

I have  an all-in-one wazuh manager 4.7 working for several months now and suddenly no more alerts are displayed in the dashboard.

However, activating logall_json I checked that all alerts are received correctly.

I run "filebeat test output" and it looks good.

Restarted the server but still no more alerts displayed for the last 10 hours.

Thanks 

[J.Carlos A]

Not sure if this helps but I noticed this error 

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

[2024-08-16T09:52:52,996][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for wazuh from 10.10.10.1:42940
[2024-08-16T12:07:31,465][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 10.10.10.1:57174
[2024-08-16T12:08:01,472][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 10.10.10.1:35924

However admin password hasn't been changed since first installation.

I have tested the admin password and it works, as in the following check, which provides some cluster error message but from an older date (2024.07.23 ,  a that time the dashboard seem to be working fine).

curl -k -u admin:<admin passwd> https://10.10.10.1:9200/_cluster/allocation/explain

{"index":".opendistro-alerting-alert-history-2024.07.23-000004","shard":0,"primary":false,"current_state":"unassigned","unassigned_info":{"reason":"CLUSTER_RECOVERED","at":"2024-08-16T06:59:10.443Z","last_allocation_status":"no_attempt"},"can_allocate":"no","allocate_explanation":"cannot allocate because allocation is not permitted to any of the nodes","node_allocation_decisions":[{"node_id":"q6mXFNLobSSZW5pUGtC4Ag","node_name":"node-1","transport_address":"10.10.10.1:9300","node_attributes":{"shard_indexing_pressure_enabled":"true"},"node_decision":"no","deciders":[{"decider":"same_shard","decision":"NO","explanation":"a copy of this shard is already allocated to this node [[.opendistro-alerting-alert-history-2024.07.23-000004][0], node[q6mXFNLobSSZW5pUGtC4Ag], [P], s[STARTED], a[id=GbzgqtWvRjW7m9e-XQd-9w]]"}]}]}

please help

[Olusegun Adenrele Oyebo]

Hello J.Carlos,

As a workaround, can you try to restart the Wazuh indexer service and let's see if that helps using command systemctl restart wazuh-indexer.

If issue still persists, kindly assist with logs for further review. Revert with the full output of the below commands:

• Wazuh dashboard: 

• • cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

• • journalctl -u wazuh-dashboard

• Wazuh indexder: 
  • cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

Also, check the disk space on the Wazuh indexer server using command df -H

Will be expecting your feedback on the outcome.

Best regards.

[J.Carlos A]

THanks Olusegun,

Just sent you the files.

please let me know your findings.

Best regards,

[Olusegun Adenrele Oyebo]

Hello J.Carlos,,

I can't see any of the files you sent. Can you help to verify this and resend them? Thank you.

Best regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/3d1ba183-a91e-452a-aea3-505035da4b59n%40googlegroups.com.

[J.Carlos A]

Hi Olusegun,

Please find attached compressed file.

Thanks,

[J.Carlos A]

[Olusegun Adenrele Oyebo]

Hello J.Carlos,

Reviewd your logs and saw multiple error entries like below:

[ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)

[validation_exception]: Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1039559848/991.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1039559848/991.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]"}

Based on the first error entry above, can you try to run the security admin script on your server using the below command:

• /usr/share/wazuh-indexer/bin/indexer-security-init.sh

If issue still persists, next, review your indexes as it seems your shard limits has been exceeded as indicated in the second entry. With this, there are multiple ways to resolve this issue.

Option 1:

To resolve this issue, you can go through the below procedures in order to delete old indices. To delete old indices that are not needed, you can first check the indices stored by running the below command:

• curl -k -u admin:<admin_password> -XGET "https://<ip_address>:9200/_cat/indices?v" replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer.

You can now use the below command to delete the old indexes:

• curl -k -u admin:<admin_password> -XDELETE "https://<ip_address>:9200/<index_to_delete>". Replace <admin_password> with the password of the user admin and also <ip_address> with the IP address of the wazuh indexer and <index_to_delete> with the index to delete e.g. curl -k -u admin:Test123 -XDELETE "https://192.168.227.139:9200/wazuh-alerts-4.x-2023.08.20"

If you want to delete indices for a whole month, you can use the wildcard (*). For example, you want to delete for the whole of September:

• curl -k -u admin:Test123 -XDELETE "https://192.168.227.139:9200/wazuh-alerts-4.x-2023.09*"

Option 2:
Another way which we can recommend to resolve this issue from happening later in the future is adding more nodes to your Wazuh indexer cluster. You can check the below link for more information on how to configure wazuh indexer cluster.

• https://documentation.wazuh.com/current/user-manual/upscaling/index.html

Next, based on the "circuit breakage exception" error, will recommend you increase the JVM heap size to the recommended value which is half of the system RAM. The essence is to increase and improve the performance of your Wazuh indexer. Edit the /etc/wazuh-indexer/jvm.options file .For example, set the size as follows for a system with 8 GB of RAM:

-Xms4g
-Xmx4g

Where the total heap space:
-Xms4g - initial size is set to 4Gb of RAM.
-Xmx4g - maximum size is to 4Gb of RAM

Restart the Wazuh indexer and dashboard services after performing the changes:

• systemctl daemon-reload
• systemctl restart wazuh-indexer
• systemctl restart wazuh-dashboard

Let me know if this resolves your issue.

Best regards.

[J.Carlos A]

Hi Olusegun,

I went with option 1.

I had also to delete manually index of August 15th as it appear in red status in the dashboard and was not possible to reindex.

It is fixed now.

thanks for your support