Wazuh alerts are not coming

[Kerim Karataş]

Hello Everyone

I had no problems with Wazuh until a few weeks ago, but now the alerts that should come are not coming. I did not see any problems in the logs. Can you suggest places to check and solutions?

Best regards.

[Javier Medeot]

Hi Kerim.

If no new alerts show up in your Wazuh dashboard, you can first check the services are up and running. For example, systemctl status filebeat. Same for wazuh-indexer, wazuh-manager, and wazuh-dashboard services.You can look for errors and warnings in the logs. For example, by running cat /var/ossec/logs/ossec.log | grep -i -E "fatal|critical|error|warn" in the Wazuh server. Other log files to look at are /var/log/filebeat/filebeat and under /var/log/wazuh-indexer/.

You need to identify where the problem lies. If the alerts do get generated and indexed but do not get displayed in the Wazuh dashboard it could be some filters are applied in your visualization. If, for instance, alerts don't get indexed, it could be some issue in the forwarding or with the Wazuh indexer. If events do get archived but no alerts are created it might be some issue with your rules. If no events get archived at all it might be something with your endpoints or the Wazuh server. You can refer to Enable archiving for checking whether events reach the archive.

Let me know what you find and how else can I help. Thank you

[Kerim Karataş]

Hello Javier Medeot

I am sharing the logs in the attachment, for example I am making a commit in Palo Alto and sometimes this log and alert comes as an e-mail and sometimes it does not go to the dashboard, and since it does not go to the dashboard, the alert does not come either.

Since it was a long message, the system did not allow it, so I uploaded the logs as a pdf.

Best regards.

[Kerim Karataş]

Wazuh log

[Kerim Karataş]

Hi Javier

Is there any progress?

Best regards.

5 Aralık 2024 Perşembe tarihinde saat 15:19:47 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

Wazuh log

[Javier Medeot]

Hi Kerim.

Connection refused indicates the service is unavailable. Check if the Elasticsearch service is running. It could be that it is down and you need to start it. For example, systemctl status elasticsearch and systemctl start elasticsearch. Share the details of your deployment. Is this an "all-in-one" deployment? What Wazuh version? If this was a distributed deployment you would need to check connectivity for the Elasticsearch node and your Elasticsearch configuration. Check the output of curl -k -u <user>:<password> https://<IP_ADDRESS>:9200

Share your findings. Thank you.

[Kerim Karataş]

Hi Javier,

Yes this is a Quick Start distribution.

When I run the systemctl status elasticsearch command I get the response that there is no such service.

Version 4.9.2

Best regards.

10 Aralık 2024 Salı tarihinde saat 15:16:32 UTC+3 itibarıyla Javier Medeot şunları yazdı:

[Javier Medeot]

Ok, I see. If this is standard quickstart installation, make sure wazuh-indexer service is up and running as recommended previously.

From the JVM errors you shared earlier, it seems you need to configure Memory locking. Please follow the steps detailed in this link to accomplish the configuration:

• https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking

And since there's an error about deleteIndices, maybe you would benefit from implementing a retention policy as explained here:

• https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html

Let me know how it goes with these configurations. It seems JVM memory usage issues might be the cause of failed connection with the Wazuh indexer.

Thank you.

[Kerim Karataş]

Hello Javier,

I did the memory locking, but wazuh still misses the alert. The systemctl log is like this:

Best regards.

11 Aralık 2024 Çarşamba tarihinde saat 17:16:23 UTC+3 itibarıyla Javier Medeot şunları yazdı:

[Kerim Karataş]

Hi Javier,

cat /var/log/wazuh-indexer/wazuh-cluster.log

16 Aralık 2024 Pazartesi tarihinde saat 17:21:43 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

[Kerim Karataş]

 cat wazuh-cluster_deprecation.log

17 Aralık 2024 Salı tarihinde saat 13:33:54 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

[Javier Medeot]

Hi Kerim.

"CacheMaxSize is null" might be related  to the Performance analyzer plugin. Try disabling it by following this guide and restarting the Wazuh indexer:

• https://opensearch.org/docs/2.13/monitoring-your-cluster/pa/index/#disable-performance-analyzer

What about resources? What about disk free space? Check hardware requirements here:

• https://documentation.wazuh.com/current/quickstart.html#hardware

Check the indexer cluster health by running the following command:

curl -k -u admin:<YOUR_ADMIN_PASSWORD> https://localhost:9200/_cluster/health?pretty

Look for the "status" value which should be "green". Check you have no "unassigned_shards".

Let me know what you find. Thanks.

[Kerim Karataş]

Hi Javier,

My System Status:
24 GB RAM

12 CPU 
1.5 TB SSD (free 340GB)
5 Agent

1 Firewall

wazuh my machine doesn't have Performance Analyzer. Should I install it?

Best regards.

17 Aralık 2024 Salı tarihinde saat 15:37:37 UTC+3 itibarıyla Javier Medeot şunları yazdı:

[Kerim Karataş]

17 Aralık 2024 Salı tarihinde saat 16:01:16 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

[Javier Medeot]

Hi Kerim.

Yellow status means the indexer cluster (single node in your case) is not fully healthy which could lead to failures. There are unassigned shards. Check the number of replicas for your indices (rep column) by running:

curl -k -u admin:<YOUR_ADMIN_PASSWORD> https://localhost:9200/_cat/indices?v

To set replicas count to zero you can run the following command:

curl -k -u "admin:<YOUR_ADMIN_PASSWORD>" -XPUT "https://localhost:9200/wazuh-alerts-*/_settings" -H 'Content-Type: application/json' -d'
{
  "index": {
    "number_of_replicas": 0,
    "auto_expand_replicas": "false"
  }
}'

Then check the cluster health again. It should turn status to green and get all shards assigned. Let me know what you find. Thank you.

Javier

[Kerim Karataş]

Hello Javier,

I checked the rep column, but everything shows 0.

Then I tried to do what you said, I guess I didn't succeed.

Best regards.

17 Aralık 2024 Salı tarihinde saat 20:02:13 UTC+3 itibarıyla Javier Medeot şunları yazdı:

[Kerim Karataş]

Hi Javier,

I think the problem is that the alerts are stuck in the queue and that's why they don't come.

So I did the following work. I stopped the service of 5 agents and tested only the alerts coming from the firewall. When I do this, the alerts come with a little less delay.

Best regards.

18 Aralık 2024 Çarşamba tarihinde saat 10:56:51 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

[Kerim Karataş]

Hi Javier,

I can't find exactly where the problem is.

Can you please tell me step by step where to check and what to do?

Best regards.

18 Aralık 2024 Çarşamba tarihinde saat 11:40:30 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

[Kerim Karataş]

Hi javier,

I don't know what I did but I think I solved the problem with the Filebeat output :D

but the warnings are still displayed in the wazuh interface sometimes and sometimes not.

Best regards.

18 Aralık 2024 Çarşamba tarihinde saat 18:34:42 UTC+3 itibarıyla Kerim Karataş şunları yazdı:

[Javier Medeot]

Hi Kerim.

What problem with the Filebeat output? Do you mean connection with indexer refused? What "warnings are still displayed in the wazuh interface sometimes and sometimes not"?

Let's identify the problem so we can look for solutions.

So far you have deployed Wazuh using the Quickstart guide on a system with enough resources for your use case. You've even configured memory locking and index retention policies to prevent issues with memory and disk space by following our guides. The Wazuh services are up and running and your indexer cluster is in green status and all shards get assigned. Your indexes have no replicas as expected.

You are saying that alerts that "should come are not coming". What do you mean? Have you enabled events archiving as suggested earlier to identify where the problem lies here? Do events get archived in archives.json? Are alerts generated and stored in alerts.json? Can you tell where alerts are missing other than from the dashboard? Please share any relevant information so we can pinpoint the problem here.

Thank you.

[Kerim Karataş]

Hi Javier,

I am interested in the subject. I will inform you later.

Thank you.

Best regards.

18 Aralık 2024 Çarşamba tarihinde saat 21:29:58 UTC+3 itibarıyla Javier Medeot şunları yazdı: