Issues with Wazuh Log Storage and Forwarding to Elasticsearch

[Shawn Mac Gavin]

Hello Wazuh Community,

I'm experiencing issues with my Wazuh setup, where logs are not being forwarded to my  indexer servers and are instead being stored on the Wazuh manager server. I found this out this morning when I came in and my manager server's disk was full. Below are the details of my setup and the troubleshooting steps I have taken so far:

Wazuh Configuration:

• Manager Server: WAZUH-SERVER (10.111.10.69)
• Dashboard Server: WAZUH-DASHBOARD (10.111.10.70)
• Indexer Servers:
  • WAZUH-INDEXER01 (10.111.10.65)
  • WAZUH-INDEXER02 (10.111.10.68)
  • WAZUH-INDEXER03 (10.111.10.64)

ossec.conf Configuration:

<indexer>
    <enabled>yes</enabled>
    <hosts>
        <host>https://10.111.10.64:9200</host>
        <host>https://10.111.10.68:9200</host>
        <host>https://10.111.10.65:9200</host>
    </hosts>
    <ssl>
        <certificate_authorities>
            <ca>/etc/filebeat/certs/root-ca.pem</ca>
        </certificate_authorities>
        <certificate>/etc/filebeat/certs/cor-wazuh-server.pem</certificate>
        <key>/etc/filebeat/certs/cor-wazuh-server-key.pem</key>
    </ssl>
</indexer>

Troubleshooting Steps Taken:

1. Expanded Disk Space on Wazuh Manager:

   • Increased the disk size from 100GB to 200GB.
   • Resized the filesystem and logical volume to utilize the new space.
   • Verified the space allocation with df -h.

2. Verified Connectivity:

   • Ran the following curl commands to check connectivity:

curl -X GET "https://10.111.10.64:9200" --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/cor-wazuh-server.pem --key /etc/filebeat/certs/cor-wazuh-server-key.pem
curl -X GET "https://10.111.10.68:9200" --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/cor-wazuh-server.pem --key /etc/filebeat/certs/cor-wazuh-server-key.pem
curl -X GET "https://10.111.10.65:9200" --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/cor-wazuh-server.pem --key /etc/filebeat/certs/cor-wazuh-server-key.pem

 

1. Checked Wazuh Logs:

   • Reviewed /var/ossec/logs/ossec.log for any errors or warnings .
   • Found the following relevant log entries:

wazuh-analysisd: ERROR: dbsync: Bad response from database: Cannot save Syscheck
wazuh-analysisd: WARNING: (7612): Rule ID '100001' is duplicated. Only the first occurrence will be considered.
wazuh-authd: WARNING: Duplicate name 'VEEAM-DC-Repo', rejecting enrollment. Agent '036' can't be replaced since it is not disconnected.
wazuh-logcollector: ERROR: (1103): Could not open file '/var/log/auth.log' due to [(2)-(No such file or directory)].
wazuh-analysisd: ERROR: Read error on queue/diff/cor-files-public/535/last-entry: file empty.
wazuh-modulesd:vulnerability-scanner: ERROR: Error updating feed: [json.exception.out_of_range.401] array index 3 is out of range, trying to re-download the feed.

1. Checked Logs:

   • Verified logs on each indexer server for any connection issues or errors.

2. Verified Indexing Status:

   • Checked Wazuh status using wazuh-control info.

3. Restarted Services:

   • Restarted Wazuh manager and Elasticsearch services after making changes.

Despite these steps, the logs still appear to be stored on the Wazuh manager server rather than being forwarded to the Elasticsearch indexer servers. I would appreciate any assistance or guidance on resolving this issue.

Thank you!

[Himanshu Sharma]

Hi Team,

As you updated the logs are stored in the Wazuh -manager but you are not seeing them in the dashboard. To find out the root cause can you please share the output of the below commands?

Wazuh indexer:
cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

Wazuh manager:
cat /var/log/filebeat/filebeat | grep -i -E "error|warn"
filebeat test output

systemctl status filebeat

tail -n 5 /var/ossec/logs/alerts/alerts.jso

Also, you can check out our official document to learn more. https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html#troubleshooting

Waiting for your response soon.