Security Events not showing and Agents disappearing

[Patrick Cunningham]

Hello,
We are currently setting up a new Wazuh server after having issues with events not showing up in our old server, even though we were getting email alerts.

With this new installation we decided to use the OVA installation method as it was the easiest. We set a static IP address for it (same as the old server which has been shut down) and all 30 agents connected.

We were having an issue with LDAP integration which we believe has been resolved (after using the securityadmin.sh command), however now we are having an issue where any account (LDAP or 'Admin') will sometimes show 0 agents on the server (connected, disconnected, etc.). They seem to come back after I log back in with the Admin account and I refresh the page a few times.

The other issue we are having is that even though when all 30 agents are connected, we are not getting any logs showing up in the web interface. We had some come in last Friday (7/22) for a little bit before LDAP was setup, but nothing after that.

I can see logs in /var/ossec/logs/alerts/alerts.log but nothing in the web interface.

Some background on our setup:

16% disk utilization
16GB of ram in the VM, 8GB allocated to the JVM
Number of Agents: 30
Agent versions: 4.2.4, 4.2.5, 4.3.0, 4.3.6
Agent OS's: Ubuntu 16, 18, 20, Windows Server 2012R2, 2016, 2019, and Windows 10 Pro


The only logs we have from /var/log/wazuh-indexer/wazuh-cluster.log are the following:

(This repeats a lot, here is just a sample)

[2022-07-25T18:00:00,305][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No index-level perm match for User [name=kibanaserver, backend_roles=[], requestedTenant=null] Resolved [aliases=[], allIndices=[wazuh-mo$[2022-07-25T18:00:00,305][INFO ][o.o.s.p.PrivilegesEvaluator] [node-1] No permissions for [indices:admin/get]
[2022-07-25T18:04:23,160][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep

I apologize for the long post, but I look forward to hearing back

Thank You,
Patrick C.

[Jose Camargo]

Hi Patrick, thanks for using Wazuh

First, Wazuh OVA is for testing purposes, it is not recommended for your production environment. If you want to do a simple installation, you can follow this guide to have an "All-In-One" deployment: https://documentation.wazuh.com/current/installation-guide/index.html . And I'd also recommend to re-enroll all agents once the deployment is done.

Second, if you verified that you're getting logs in /var/ossec/logs/alerts/alerts.log, then, in case no new alerts are appearing on the WUI, please check the manager's, Filebeat, Elastic or Kibana logs for any trace of errors.

• cat /var/ossec/logs/ossec.log | grep -i -E "err|warn"

Manager version < 4.3:

• systemctl status filebeat -l | grep -i -E "err|warn"
• systemctl status kibana -l | grep -i -E "err|warn"
• cat /var/log/elasticsearch or systemctl status elasticsearch -l | grep -i -E "err|warn"

Manager version >= 4.3:

• cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"
  
• cat /var/log/filebeat/filebeat | grep -i -E "error|warn"
  
• cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"
  
• journalctl -u wazuh-dashboard
  
• cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"
  

Please also verify that all services are running correctly:

• /var/ossec/bin/ossec-control status

If any of the key services (like analisysd) are not active, try restarting the manager and paste a longer trace of the ossec.log here right after the restart is complete:

• tail -n200 /var/ossec/logs/ossec.log

Third, please verify that there are alerts in the Wazuh indexer:

• curl https://<WAZUH_INDEXER_IP>:9200/_cat/indices/wazuh-alerts-* -u <wazuh_indexer_user>:<wazuh_indexer_password> -k

Output should be like this:

• green open wazuh-alerts-4.x-2021.03.03 xwFPX7nFQxGy-O5aBA3LFQ 3 0 340 0 672.6kb 672.6kb

If you do not see any Wazuh related index, it means you have no alerts stored in Wazuh indexer.

To ensure that Filebeat is correctly configured, run the following command:

• filebeat test output

You can check more troubleshooting steps in these documents:

• https://documentation.wazuh.com/current/user-manual/agent-enrollment/troubleshooting.html
  
• https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html#none-of-the-above-solutions-are-fixing-my-problem
  

I'll be awaiting your feedback.