Wazuh dashboard logs

[Jan Paračka]

Hello,

We are running Wazuh version 4.12.0. After upgrading on May 12th, 2025 around 09:00 CEST, we encountered the following issue:

- Since that moment, no new "Events" are showing in the Wazuh dashboard.
- Agents are communicating correctly and logs are being written into `/var/ossec/logs/alerts.log`, but no new data appears in the `wazuh-alerts-*` indices.
- The latest timestamp in the `wazuh-alerts-*` index is `2025-05-12T08:19:36.202Z`.
- The configuration in `elastic-output.yml` is correct and the `wazuh-modulesd` service is running.
- OpenSearch is up and running, cluster status is `yellow`, with all 153 shards active and just a few unassigned.
- Restarting the service and manually refreshing the index (`_refresh`) does not help.

I am attaching:

- `/var/ossec/logs/ossec.log`
- `/var/ossec/etc/modules.d/elastic-output.yml`
- Outputs of:
  - `curl -ku *****:***** 'https://localhost:9200/_cat/indices?v'`
  - `curl -ku *****:***** 'https://localhost:9200/_cluster/health?pretty'`
  - `curl -ku *****:***** 'https://localhost:9200/wazuh-alerts-*/_search?pretty' -H 'Content-Type: application/json' -d'{ "size": 1, "sort": [ { "@timestamp": "desc" } ] }'`

Please help us investigate why no new alerts are being written to the index even though logs are being generated and all services appear to be running.

Thank you,  
Jan Paračka

[Damian Nicastro]

Hello Jan:

I hope you are fine.

Since you mentioned that /var/ossec/logs/alerts/alerts.json are being generated, the problem must be in filebeat or in the wazuh-indexer

Please, include the following information:

- From which version did you make the upgrade

- The content of the  /var/ossec/etc/modules.d/elastic-output.yml file (it was not included previously)

- Filebeat:

# systemctl status filebeat -l

# filebeat test output

# lsof   /var/ossec/logs/alerts/alerts.json

# less /var/log/filebeat/filebeat* | grep -iE 'WARN|ERROR'

- wazuh-indexer:

# df -h

# cat /etc/wazuh-indexer/opensearch.yml
# less   /var/log/wazuh-indexer/<indexer_cluster_name>.log | grep -iE 'WARN|ERROR'

Note: If you have more than one filebeat service running. Please, provide the requested info for all of them. The same applies if you have more than one wazuh-indexer node.

Thanks

[Jan Paračka]

  I’m using a single-node deployment.

https://documentation.wazuh.com/current/deployment-options/virtual-machine/virtual-machine.html
Upgrade was from 4.11.3 to 4.12.0  

Dne úterý 13. května 2025 v 21:41:36 UTC+2 uživatel Damian Nicastro napsal:

[Jan Paračka]

Hi Damian,

Please, include the following information:

- From which version did you make the upgrade - from 4.11

- The content of the  /var/ossec/etc/modules.d/elastic-output.yml file (it was not included previously) - indexer-output.yml - it was a chatgpt advice

- Filebeat:

# systemctl status filebeat -l

[root@wazuh-server /]# systemctl status filebeat -l
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: disabled)
     Active: active (running) since Tue 2025-05-13 18:14:22 CEST; 3h 30min ago
       Docs: https://www.elastic.co/products/beats/filebeat
   Main PID: 26350 (filebeat)
      Tasks: 10 (limit: 9467)
     Memory: 31.2M
        CPU: 4.156s
     CGroup: /system.slice/filebeat.service
             └─26350 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home />
...skipping...
● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; preset: disabled)
     Active: active (running) since Tue 2025-05-13 18:14:22 CEST; 3h 30min ago
       Docs: https://www.elastic.co/products/beats/filebeat
   Main PID: 26350 (filebeat)
      Tasks: 10 (limit: 9467)
     Memory: 31.2M
        CPU: 4.156s
     CGroup: /system.slice/filebeat.service
             └─26350 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home />

# filebeat test output

[root@wazuh-server /]# filebeat test output
elasticsearch: https://172.16.1.12:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.16.1.12
    dial up... ERROR dial tcp 172.16.1.12:9200: connect: connection refused
[root@wazuh-server /]#

# lsof   /var/ossec/logs/alerts/alerts.json

[root@wazuh-server /]# lsof   /var/ossec/logs/alerts/alerts.json
COMMAND     PID  USER   FD   TYPE DEVICE   SIZE/OFF      NODE NAME
filebeat  26350  root    9r   REG    8,1 1579186548 100788558 /var/ossec/logs/alerts/alerts.json
wazuh-ana 75584 wazuh   13w   REG    8,1 1579190530 100788558 /var/ossec/logs/alerts/alerts.json

# less /var/log/filebeat/filebeat* | grep -iE 'WARN|ERROR'

-file

- wazuh-indexer:

# df -h

[root@wazuh-server /]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           3.9G  2.7M  3.9G   1% /dev/shm
tmpfs           1.6G  8.7M  1.6G   1% /run
/dev/sda1        50G   38G   13G  76% /
tmpfs           3.9G   36K  3.9G   1% /tmp
/dev/sda128      10M  1.3M  8.7M  13% /boot/efi
tmpfs           794M     0  794M   0% /run/user/1000

# cat /etc/wazuh-indexer/opensearch.yml

[root@wazuh-server /]# cat /etc/wazuh-indexer/opensearch.yml
network.host: ["127.0.0.1", "172.16.1.12"]
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
cluster.name: "wazuh-cluster"

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer
path.repo: /mnt/snapshots

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=indexer,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

# less   /var/log/wazuh-indexer/<indexer_cluster_name>.log | grep -iE 'WARN|ERROR'

[root@wazuh-server /]# less   /var/log/wazuh-indexer/<indexer_cluster_name>.log | grep -iE 'WARN|ERROR'
bash: indexer_cluster_name: No such file or directory

Dne úterý 13. května 2025 v 21:41:36 UTC+2 uživatel Damian Nicastro napsal:

Hello Jan:

[Damian Nicastro]

Hello Jan:

I think is pretty clear that there is no indexation because filebeat cannot connect to the wazuh-indexer:

# filebeat test output

# filebeat test output
elasticsearch: https://172.16.1.12:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 172.16.1.12
    dial up... ERROR dial tcp 172.16.1.12:9200: connect: connection refused

#

Please, verify the IP of wazuh-indexer, check connectivity in the port, send the config of filebeat and ensure that the credentials are properly saved in the filebeat keystore:

# telnet 172.16.1.1 9200

# cat /etc/filebeat/filebeat.yml

# echo <admin_user> | filebeat keystore add username --stdin --force (by default, these credentials are admin:admin)
# echo <admin_passwd> | filebeat keystore add password --stdin --force

Ensure also that you have the proper keystores configured in the wazuh-manager for vulnerabilities indexation:

# echo '<admin_user> ' | /var/ossec/bin/wazuh-keystore -f indexer -k username
# echo '<admin_passwd> ' | /var/ossec/bin/wazuh-keystore -f indexer -k password

After this try again:

# filebeat test output

If this did not solve the issue, it could be also that the filebeat cert is invalid or expired:

# ls -l /etc/filebeat/certs/
# cd /etc/filebeat/certs/

# openssl x509 -noout -subject -in wazuh-server.pem

# openssl x509 -enddate -noout -in wazuh-server.pem

# openssl verify -CAfile root-ca.pem wazuh-server.pem

Important Note: the <> means that you have to replace what is inside with you proper name or user

Considering this, and checking your indexer config, the cluster name is wazuh-cluster and the log file will be:

# less   /var/log/wazuh-indexer/wazuh-cluster.log | grep -iE 'WARN|ERROR'

And, please also send the filebeat log in case there is any error:

# less /var/log/filebeat/filebeat* | grep -iE 'WARN|ERROR'

I hope this helps.

Thanks

[Jan Paračka]

Hi Damian,

The issue has been resolved by updating the IP address in the filebeat.yml file — changed from our internal address (172.16.1.12) to 127.0.0.1.
I also added additional disk space as part of the fix.

Thank you very much for your help!

Dne středa 14. května 2025 v 14:53:44 UTC+2 uživatel Damian Nicastro napsal:

[Damian Nicastro]

I am glad to hear that

Thanks for the update.