Wazuh Vulnerability Scanner suddenly stopped reporting results

[Raphael Pepi]

recently upgraded from Wazuh Server 4.3 to 4.4.

After the update, many of my agents have mysteriously stopped reporting vulnerabilities.

I'ts confusing  and unclear what changed or why.  hoping someone might have some thoughts on where to begin.

heres the ossec.conf from the server followed by the global agents defaults conf

#### Server Config

cat /var/ossec/etc/ossec.conf

<!--

  Wazuh - Manager - Default configuration for ubuntu 20.04

  More info at: https://documentation.wazuh.com

  Mailing list: https://groups.google.com/forum/#!forum/wazuh

-->

<ossec_config>

  <global>

    <jsonout_output>yes</jsonout_output>

    <alerts_log>yes</alerts_log>

    <logall>no</logall>

    <logall_json>no</logall_json>

    <email_notification>no</email_notification>

    <smtp_server>smtp.example.wazuh.com</smtp_server>

    <email_from>wa...@example.wazuh.com</email_from>

    <email_to>reci...@example.wazuh.com</email_to>

    <email_maxperhour>12</email_maxperhour>

    <email_log_source>alerts.log</email_log_source>

    <agents_disconnection_time>10m</agents_disconnection_time>

    <agents_disconnection_alert_time>0</agents_disconnection_alert_time>

  </global>

  <alerts>

    <log_alert_level>3</log_alert_level>

    <email_alert_level>12</email_alert_level>

  </alerts>

  <!-- Choose between "plain", "json", or "plain,json" for the format of internal logs -->

  <logging>

    <log_format>plain</log_format>

  </logging>

  <remote>

    <connection>secure</connection>

    <port>1514</port>

    <protocol>tcp</protocol>

    <queue_size>131072</queue_size>

  </remote>

  <!-- Policy monitoring -->

  <rootcheck>

    <disabled>no</disabled>

    <check_files>yes</check_files>

    <check_trojans>yes</check_trojans>

    <check_dev>yes</check_dev>

    <check_sys>yes</check_sys>

    <check_pids>yes</check_pids>

    <check_ports>yes</check_ports>

    <check_if>yes</check_if>

    <!-- Frequency that rootcheck is executed - every 12 hours -->

    <frequency>43200</frequency>

    <rootkit_files>etc/rootcheck/rootkit_files.txt</rootkit_files>

    <rootkit_trojans>etc/rootcheck/rootkit_trojans.txt</rootkit_trojans>

    <skip_nfs>yes</skip_nfs>

  </rootcheck>

  <wodle name="cis-cat">

    <disabled>yes</disabled>

    <timeout>1800</timeout>

    <interval>1d</interval>

    <scan-on-start>yes</scan-on-start>

    <java_path>wodles/java</java_path>

    <ciscat_path>wodles/ciscat</ciscat_path>

  </wodle>

  <!-- Osquery integration -->

  <wodle name="osquery">

    <disabled>yes</disabled>

    <run_daemon>yes</run_daemon>

    <log_path>/var/log/osquery/osqueryd.results.log</log_path>

    <config_path>/etc/osquery/osquery.conf</config_path>

    <add_labels>yes</add_labels>

  </wodle>

  <!-- System inventory -->

  <wodle name="syscollector">

    <disabled>no</disabled>

    <interval>1h</interval>

    <scan_on_start>yes</scan_on_start>

    <hardware>yes</hardware>

    <os>yes</os>

    <network>yes</network>

    <packages>yes</packages>

    <ports all="no">yes</ports>

    <processes>yes</processes>

    <!-- Database synchronization settings -->

    <synchronization>

      <max_eps>10</max_eps>

    </synchronization>

  </wodle>

  <sca>

    <enabled>yes</enabled>

    <scan_on_start>yes</scan_on_start>

    <interval>12h</interval>

    <skip_nfs>yes</skip_nfs>

  </sca>

  <vulnerability-detector>

    <enabled>yes</enabled>

    <interval>2h</interval>

    <min_full_scan_interval>3d</min_full_scan_interval>

    <run_on_start>yes</run_on_start>

    <!-- Ubuntu OS vulnerabilities -->

    <provider name="canonical">

      <enabled>yes</enabled>

      <os>trusty</os>

      <os>xenial</os>

      <os>bionic</os>

      <os>focal</os>

      <os>jammy</os>

      <update_interval>1h</update_interval>

    </provider>

    <!-- Debian OS vulnerabilities -->

    <provider name="debian">

      <enabled>no</enabled>

      <os>buster</os>

      <os>bullseye</os>

      <update_interval>1h</update_interval>

    </provider>

    <!-- RedHat OS vulnerabilities -->

    <provider name="redhat">

      <enabled>no</enabled>

      <os>5</os>

      <os>6</os>

      <os>7</os>

      <os>8</os>

      <os>9</os>

      <update_interval>1h</update_interval>

    </provider>

    <!-- Amazon Linux OS vulnerabilities -->

    <provider name="alas">

      <enabled>no</enabled>

      <os>amazon-linux</os>

      <os>amazon-linux-2</os>

      <update_interval>1h</update_interval>

    </provider>

    <!-- SUSE OS vulnerabilities -->

    <provider name="suse">

      <enabled>no</enabled>

      <os>11-server</os>

      <os>11-desktop</os>

      <os>12-server</os>

      <os>12-desktop</os>

      <os>15-server</os>

      <os>15-desktop</os>

      <update_interval>1h</update_interval>

    </provider>

    <!-- Arch OS vulnerabilities -->

    <provider name="arch">

      <enabled>no</enabled>

      <update_interval>1h</update_interval>

    </provider>

    <!-- Windows OS vulnerabilities -->

    <provider name="msu">

      <enabled>yes</enabled>

      <update_interval>1h</update_interval>

    </provider>

    <!-- Aggregate vulnerabilities -->

    <provider name="nvd">

      <enabled>yes</enabled>

      <update_from_year>2010</update_from_year>

      <update_interval>1h</update_interval>

    </provider>

  </vulnerability-detector>

  <!-- File integrity monitoring -->

  <syscheck>

    <disabled>no</disabled>

    <!-- Frequency that syscheck is executed default every 12 hours -->

    <frequency>43200</frequency>

    <scan_on_start>yes</scan_on_start>

    <!-- Generate alert when new file detected -->

    <alert_new_files>yes</alert_new_files>

    <!-- Don't ignore files that change more than 'frequency' times -->

    <auto_ignore frequency="10" timeframe="3600">no</auto_ignore>

    <!-- Directories to check  (perform all possible verifications) -->

    <directories>/etc,/usr/bin,/usr/sbin</directories>

    <directories>/bin,/sbin,/boot</directories>

    <!-- Files/directories to ignore -->

    <ignore>/etc/mtab</ignore>

    <ignore>/etc/hosts.deny</ignore>

    <ignore>/etc/mail/statistics</ignore>

    <ignore>/etc/random-seed</ignore>

    <ignore>/etc/random.seed</ignore>

    <ignore>/etc/adjtime</ignore>

    <ignore>/etc/httpd/logs</ignore>

    <ignore>/etc/utmpx</ignore>

    <ignore>/etc/wtmpx</ignore>

    <ignore>/etc/cups/certs</ignore>

    <ignore>/etc/dumpdates</ignore>

    <ignore>/etc/svc/volatile</ignore>

    <!-- File types to ignore -->

    <ignore type="sregex">.log$|.swp$</ignore>

    <!-- Check the file, but never compute the diff -->

    <nodiff>/etc/ssl/private.key</nodiff>

    <skip_nfs>yes</skip_nfs>

    <skip_dev>yes</skip_dev>

    <skip_proc>yes</skip_proc>

    <skip_sys>yes</skip_sys>

    <!-- Nice value for Syscheck process -->

    <process_priority>10</process_priority>

    <!-- Maximum output throughput -->

    <max_eps>100</max_eps>

    <!-- Database synchronization settings -->

    <synchronization>

      <enabled>yes</enabled>

      <interval>5m</interval>

      <max_interval>1h</max_interval>

      <max_eps>10</max_eps>

    </synchronization>

  </syscheck>

  <!-- Active response -->

  <global>

    <white_list>127.0.0.1</white_list>

    <white_list>^localhost.localdomain$</white_list>

    <white_list>127.0.0.53</white_list>

  </global>

  <command>

    <name>disable-account</name>

    <executable>disable-account</executable>

    <timeout_allowed>yes</timeout_allowed>

  </command>

  <command>

    <name>restart-wazuh</name>

    <executable>restart-wazuh</executable>

  </command>

  <command>

    <name>firewall-drop</name>

    <executable>firewall-drop</executable>

    <timeout_allowed>yes</timeout_allowed>

  </command>

  <command>

    <name>host-deny</name>

    <executable>host-deny</executable>

    <timeout_allowed>yes</timeout_allowed>

  </command>

  <command>

    <name>route-null</name>

    <executable>route-null</executable>

    <timeout_allowed>yes</timeout_allowed>

  </command>

  <command>

    <name>win_route-null</name>

    <executable>route-null.exe</executable>

    <timeout_allowed>yes</timeout_allowed>

  </command>

  <command>

    <name>netsh</name>

    <executable>netsh.exe</executable>

    <timeout_allowed>yes</timeout_allowed>

  </command>

  <!--

  <active-response>

    active-response options here

  </active-response>

  -->

  <!-- Log analysis -->

  <localfile>

    <log_format>command</log_format>

    <command>df -P</command>

    <frequency>360</frequency>

  </localfile>

  <localfile>

    <log_format>full_command</log_format>

    <command>netstat -tulpn | sed 's/\([[:alnum:]]\+\)\ \+[[:digit:]]\+\ \+[[:digit:]]\+\ \+\(.*\):\([[:digit:]]*\)\ \+\([0-9\.\:\*]\+\).\+\ \([[:digit:]]*\/[[:alnum:]\-]*\).*/\1 \2 == \3 == \4 \5/' | sort -k 4 -g | sed 's/ == \(.*\) ==/:\1/' | sed 1,2d</command>

    <alias>netstat listening ports</alias>

    <frequency>360</frequency>

  </localfile>

  <localfile>

    <log_format>full_command</log_format>

    <command>last -n 20</command>

    <frequency>360</frequency>

  </localfile>

  <ruleset>

    <!-- Default ruleset -->

    <decoder_dir>ruleset/decoders</decoder_dir>

    <rule_dir>ruleset/rules</rule_dir>

    <rule_exclude>0215-policy_rules.xml</rule_exclude>

    <list>etc/lists/audit-keys</list>

    <list>etc/lists/amazon/aws-eventnames</list>

    <list>etc/lists/security-eventchannel</list>

    <!-- User-defined ruleset -->

    <decoder_dir>etc/decoders</decoder_dir>

    <rule_dir>etc/rules</rule_dir>

  </ruleset>

  <rule_test>

    <enabled>yes</enabled>

    <threads>1</threads>

    <max_sessions>64</max_sessions>

    <session_timeout>15m</session_timeout>

  </rule_test>

  <!-- Configuration for wazuh-authd -->

  <auth>

    <disabled>no</disabled>

    <port>1515</port>

    <use_source_ip>no</use_source_ip>

    <purge>yes</purge>

    <use_password>no</use_password>

    <ciphers>HIGH:!ADH:!EXP:!MD5:!RC4:!3DES:!CAMELLIA:@STRENGTH</ciphers>

    <!-- <ssl_agent_ca></ssl_agent_ca> -->

    <ssl_verify_host>no</ssl_verify_host>

    <ssl_manager_cert>etc/sslmanager.cert</ssl_manager_cert>

    <ssl_manager_key>etc/sslmanager.key</ssl_manager_key>

    <ssl_auto_negotiate>no</ssl_auto_negotiate>

  </auth>

  <cluster>

    <name>wazuh</name>

    <node_name>node01</node_name>

    <node_type>master</node_type>

    <key></key>

    <port>1516</port>

    <bind_addr>0.0.0.0</bind_addr>

    <nodes>

        <node>NODE_IP</node>

    </nodes>

    <hidden>no</hidden>

    <disabled>yes</disabled>

  </cluster>

</ossec_config>

<ossec_config>

  <localfile>

    <log_format>syslog</log_format>

    <location>/var/ossec/logs/active-responses.log</location>

  </localfile>

  <localfile>

    <log_format>syslog</log_format>

    <location>/var/log/auth.log</location>

  </localfile>

  <localfile>

    <log_format>syslog</log_format>

    <location>/var/log/syslog</location>

  </localfile>

  <localfile>

    <log_format>syslog</log_format>

    <location>/var/log/dpkg.log</location>

  </localfile>

  <localfile>

    <log_format>syslog</log_format>

    <location>/var/log/kern.log</location>

  </localfile>

</ossec_config>

#### Default Agent Config

cat /var/ossec/etc/shared/default/agent.conf

<agent_config>

  <wodle name="syscollector">

    <disabled>no</disabled>

    <interval>1h</interval>

    <os>yes</os>

    <packages>yes</packages>

    <hotfixes>yes</hotfixes>

  </wodle>

</agent_config>

whats odd is that about 5 of my 17 agents are reporting vulnerabilities normally, but the other 12 just suddenly stopped.

[Fabian Ruiz]

Hi  Raphael Pepi

Can you send me the logs of the manager and agents to check if there is an error that can tell us what is happening?

Thanks for using Wazuh,

Regards.

[Raphael Pepi]

Heres my ossec-log data from the wazuh-server, and 2 agents (one working and 1 not)  as well as the screen shots of what the results look like inside the wazuh app

If theres a different log that you need let me know the name.

[Fabian Ruiz]

I have checked the logs and it seems that the vunerability detector is working fine, it is possible that it has not detected vunerabilities for these agents, because they use different operating systems:

- Ubuntu 18.04.6 LTS (Bionic Beaver)
- Ubuntu: 20.04.6 LTS (Focal Fossa)

If the Wazuh server is on the internet it should have the CVEs updated, in case it should not be updated manually, you can check this documentation: https://documentation.wazuh.com/current/user-manual/capabilities/vulnerability-detection/offline-update.html 

[Raphael Pepi]

after looking a bit more i've noticed specifically that all the servers that are reporting 0 results are ubuntu 18.04.6 (bionic) servers and the ones reporting correctly are all ubuntu 20+ (focal)

i checked ossec.conf to verify correctness of this block:

    <!-- Ubuntu OS vulnerabilities -->

    <provider name="canonical">

      <enabled>yes</enabled>

      <os>trusty</os>

      <os>xenial</os>

      <os>bionic</os>

      <os>focal</os>

      <os>jammy</os>

      <update_interval>1h</update_interval>

    </provider>

and it does look correct.  Beyond that Im not sure where to look to dive deeper into whats causing no results for bionic servers, but this is at least the right direction.

[Raphael Pepi]

Where/How can i clear the database for this (Bionic)? The Machine is connected to the internet, and does appear to be updating automatically, but somehow results for bionic servers arnt working corretkly. i would like to restore it to working automatically (as it ad been till 06-03-2023) vs manually updating it. 

[Fabian Ruiz]

Hi Rapahel,

If your agents are connected to the Internet, it is not necessary to update the CVE manually.

Make sure that the Wazuh server version is updated to the version that matches the version of the agents, in this case if I am not mistaken it is 4.3 or if not use the latest version 4.4.

[Raphael Pepi]

All machines are connected to the Internet, none are throwing any type of logging or connection errors, in fact all seem to be regularly (hourly) polling for CVE updates,  the wazuh server is running 4.4.4, all agents are 4.4.4

All machines have received Regular and frequent security patch apt updates. 

ALL Ubuntu 20.x or Later Servers are reporting Vulnerabilities

NONE of the Ubuntu 18.X Servers are reporting Vulnerabilities anymore (as of 06/03/2022 when they last correctly reported vulnerabilities)

Im kind of as my wits end as to why this is occurring.  as I've mentioned previously it just suddenly stopped for no apparent reason, around the time that i updated machines to 4.4.x however not necessarily related. 

 

i do occasionally see this in the logs on a 18.x server:

sca: INFO: Integration checksum failed for policy '/var/ossec/ruleset/sca/cis_ubuntu18-04.yml'. Resending scan results in 185 seconds.

is there a way to fix the checksum or re-get the file so this fail does not occur? 

[Raphael Pepi]

This issue is still ongoing. The Ui Shows that scans are running but the results its showing for ubuntu 18 are 0/0/0/0  i can see from searching the event log that the last date it actually reported anything was on June 4th. From My end nothing changed on that day, but that is when Ubunutu 18 results stopped showing up completely.  Heres an example of what my Ui (Event Log) looks like. 

At this point Im wondering if theres any way to remove one of my agents completely and reset the wazuh manager database, then reconnect/rescan so that the agent will start showing results again. Anything would be better than its current state of showing that its scanning, but not showing any results  at all. Again this Issue is relegated to specifically our Ubuntu 18 Clients, all Ubuntu 20+ and Other Os are working normally.

[Fabian Ruiz]

Hi Raphael,

I would recommend you to open your case in a wazuh issue, apparently Wazuh is not working as it should in that version of ubuntu, I leave you the links to open an issue and comment everything you have to do with this.

https://github.com/wazuh/wazuh/issues/new/choose

Regards.