Vulnerability Detection not working for all agents

[Matthew Hemker]

I have around 90 or so total active agents, but for some reason am only getting vulnerability data for around 50 of them. 

I've looked at agents that I'm not getting vuln data for and confirmed that I am successfully getting inventory data, validating my syscollector config is applied. 

We're running Wazuh Manager 3.13.2, with agents varying between 3.9.x and 3.13.2 and OS's varying between WIN10, Server 12, Server 16, and CentOS 6/7. 

Oddly, the 50 agents that I am getting vulnerability data for don't seem to correlate with a particular agent version or OS.. I attempted to upgrade a 3.9.x agent that I wasn't getting vuln data for to the 3.12.2 version but that didnt seem to fix anything: still getting inventory data but no vulnerability data. 

Using the Splunk app 3.13.2 as well. 

Thanks for anything help anyone is able to provide!

-Matt 

[Alvaro Romero Sepulveda]

Hi Matt,

Thank you for posting in our group!

I'll be working on an answer but in order to help your better, I'd need more information regarding your Wazuh's manager and connected agents, mainly, your vulnerability-related information in ossec.log (you can simply run cat /var/ossec/logs/ossec.log | grep -i -E "vulnerability").

Regarding your problem, It's also recommended to upgrade all agents at least to the manager's version (3.13.2). There could be some incompatibility issues regarding the OS and the agent version, whether or not they are being triggered in all of your agents. For example, the vulnerabilities scan in your v3.9 Windows agents is probably not working because the Syscollector module is not meant to get Window's hotfixes before v3.11. Something similar can happen for CentOS depending on the package.

Lastly, we would recommend you to upgrade both your manager and agents to 4.0.x. You can find all the information about release notes here, but the agent enrollment process has been simplified and some missing vulnerabilities are now being reported, along with general improvements in the vulnerability detector module.

I hope this helps! I'll be looking for your answer so I can keep helping you.

Regards,

Álvaro Romero.

[Matthew Hemker]

Thank you for you response!

Attached are ossec logs from today related to vulnerability detection. 

Some new findings: after upgrading agent 005 yesterday from 3.9.x to 3.13.2 I initially did not see any vulnerability findings. But it seems after the manager ran its assessment this morning, I am now getting vulnerability data for it. 

Interestingly though, logs show the manager doing assessments for just about all agents, though not all are actually being populated in the Splunk app like previously described. For instance, we can see agent 052 having successfully been assessed in the attached logs but no data shows up in the Splunk app. 

Due to finding success in upgrading agent 005 and now getting vulnerability data, that seems to be the route to take in resolving this issue. I've upgraded agent 052 to v3.13.2 and will see if data starts populating tomorrow morning as well. 

You mentioned your recommendation to upgrade to Wazuh version 4.0.x. Unfortunately the documentation doesn't explicitly list compatibility for Wazuh 4.0.x and any particular versions of Splunk. Is Splunk supported in Wazuh 4.0.x yet, or has the documentation not been updated yet?

Thanks again!

[Alvaro Romero Sepulveda]

Hi again, Matt!

I'm glad you get said agent to work. Your log looks completely fine, so as you've stated, please continue upgrading all of your agents and tell me if you end up having another problem.

If you want to check how many vulnerabilities are being reported in each agent in real-time, you can activate the debug 1 mode for the vulnerability detector module with # echo "wazuh_modules.debug=1" >> /var/ossec/etc/local_internal_options.conf (debug 2 may be too verbose). With this option, you can hopefully do real-time checking of the vulnerabilities and find why some of your agents are being ignored. That being said, upgrading should be enough to fix your problem.

Regarding your last question, you are right, our team is working hard to complete the development of Wazuh's Splunk plugin for v4.0, but we have yet to finish it. We will update Wazuh with said plugin and documentation as soon as possible, so keep notified!

I hope this helps. Don't hesitate to ask again if you end up having problems with the vulnerability scan even after the upgrades.

Regards,
Álvaro Romero.

[Giorgio Solari]

Hi Alvaro,

I’m having a similar issue. But I’m using the latest version of Wazuh virtual appliance (4.1) in a small environment, and I have two windows 2016 different releases, for the first one, agent id 001 the vulnerability report works ok, but for the second agent id 003 I'm only getting "There are no results for selected time range. Try another one."

Here is the output of ossec.log with debug=1:

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:2078 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5450): Analyzing agent '003' vulnerabilities.

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:4352 at wm_vuldet_get_software_info(): DEBUG: (5437): Collecting agent '003' software.

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:4370 at wm_vuldet_get_software_info(): DEBUG: (5439): A partial scan will be run on agent '003'

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:4583 at wm_vuldet_get_software_info(): DEBUG: (5445): No changes have been found with respect to the last package inventory or no packages have been indexed for agent '003'

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:2119 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5471): Finished vulnerability assessment for agent '003'

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:2120 at wm_vuldet_check_agent_vulnerabilities(): DEBUG: (5470): It took '0' seconds to 'scan' vulnerabilities in agent '003'

2021/04/08 21:07:19 wazuh-modulesd:vulnerability-detector[8710] wm_vuln_detector.c:7038 at wm_vuldet_run_scan(): INFO: (5472): Vulnerability scan finished.

The Inventory data, Security Events, SCA, integrity monitoring , etc looks good for agent 003. Both windows 2016 has the same agent configuration. 

Please your help to solve this issue.

Thanks a lot for your help, I think Wazuh is awesome!

Regards,

Giorgio Solari

[Alvaro Romero Sepulveda]

Hi, Giorgio Solari

Thank you for posting in our community! It's always rewarding to hear from users who enjoy using Wazuh.

The first thing to do in order to identify if you are experiencing a bug in your Vulnerability Detector is to check both your agent's hotfixes and installed packages inventory. You can get said information from the GUI, inside the inventory data section of your agent. 

There, you should find a list of installed packages that you can download as a CSV, and a list of your system's updates. Sharing this information from both your 001 and 003 agents would be incredibly helpful.

 If you prefer to get said information without accessing the user interface, you can run these next commands:

# sqlite3 /var/ossec/queue/db/00X.db "select * from sys_programs;"

# sqlite3 /var/ossec/queue/db/00X.db "select * from sys_hotfixes;"

Being 'X' the ID number of your agent. Mind that, though highly improbable, this option could introduce unexpected information inside your database, so it's always more secure to get this data from the GUI. Once we have this information, we can compare both agent's packages and updates to check either if there's a bug in your vulnerability detector, or if your agent 001 just has vulnerable packages or programs that 003 doesn't have.

One last thing! Our Window's MSU feed has been updated just some hours ago, which should considerably improve the vulnerability detection accuracy in Windows agents. By default, the Wazuh manager is configured to download and use the new feed automatically, but in case you are using the offline update functionality, you can get the feed from this link.

I hope this helps!

[Giorgio Solari]

Hi Alvaro,

Thanks for your quick answer. At the GUI inventory, the data looks normal of both windows 2016 servers (not windows 2012 as I told earlier, sorry) Due to that I decided to run the commands You ask.

The output of sys_programs for agent 003 has some special characters on it. 

You can find attached the gathered information from the GUI and the output of the commands.

I hope this information helps to detect the cause of the issue.

Regards,

Giorgio Solari

[Alvaro Romero Sepulveda]

Hi Giorgio Solari,

Thank you very much for sharing this information, it's very much appreciated.

As I suspected, agents 001 and 003 differ drastically in both installed packages and hotfixes. Normally, that should justify the existence of differences in their corresponding vulnerability diagnosis. However, it's certainly strange that with so many installed packages and programs, agent 003 is still not reporting any vulnerability. That being said, I'll keep studying your case in order to identify if a bug is causing false negatives in your vulnerability scan.

One more thing! It would be incredibly helpful if you could you share your agent 001 vulnerabilities. Again, thank you for sharing this information. I'll reach to you in this thread when I find something new.

Best regards,
Álvaro Romero.

[Giorgio Solari]

Hi Álvaro,

Thanks a lot for your answer. Attached is the requested information.

Best regards,

Giorgio Solari

[Alvaro Romero Sepulveda]

Hi again, Giorgio

First of all, thank you for sharing your agent 001 vulnerabilities! I'm still working on this issue, but I'm having some difficulties trying to replicate your lack of vulnerabilities. I've been scanning a Windows Server 2016 in a testing environment with the same hotfixes as your agent, and my vulnerability detector is reporting several vulnerability-related alerts, especially ones related to the patch KB5000803 and the other patches it supersedes, which your agent 003 seems to lack. 

Have you checked if your manager has downloaded our latest MSU feed? To check this, search in your ossec.log a line like this next one:

2021/04/15 17:15:38 wazuh-modulesd:vulnerability-detector: INFO: (5430): The update of the 'Microsoft Security Update' feed finished successfully.

Mind that our MSU feed was last updated two days ago, so the lack of a log like this in the last few days would mean that your manager is not updated with the latest vulnerability information. As I said, this feed is automatically downloaded by default. Another question, has your agent been scanned since you reported your lack of vulnerabilities? If it's possible, you can manually trigger a vulnerability scan by restarting your manager with systemctl restart wazuh-manager if you have the <run_on_start> section in your ossec.conf enabled.

I hope this helps! We are sorry for the inconvenience and as I said, I'll reach to you whenever I find something relevant. Any additional information that you could share (ossec.log most recent vulnerability-related logs, your vulnerability detector configuration in your ossec.conf, etc.) could be helpful.

Best regards,

Alvaro Romero.

[Giorgio Solari]

Hi Álvaro,

Thanks for your help. Looks that the manager is updating normaly the MSU.  (log of the manager) :

2021/04/15 19:19:55 wazuh-modulesd:vulnerability-detector[32666] wm_vuln_detector.c:4017 at wm_vuldet_check_feed(): INFO: (5400): Starting 'Microsoft Security Update' database update.

2021/04/15 19:19:55 wazuh-modulesd:vulnerability-detector[32666] wm_vuln_detector.c:5565 at wm_vuldet_update_MSU(): DEBUG: (5406): The feed 'Microsoft Security Update' is in its latest version.

2021/04/15 19:19:55 wazuh-modulesd:vulnerability-detector[32666] wm_vuln_detector.c:4040 at wm_vuldet_check_feed(): INFO: (5430): The update of the 'Microsoft Security Update' feed finished successfully.

[root@manager ~]# date

Thu Apr 15 19:58:57 -04 2021 

And restarting the manager, the vulnerability detector check all the agents, here the log for 003:

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:2078 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5450): Analyzing agent '003' vulnerabilities.

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:4352 at wm_vuldet_get_software_info(): DEBUG: (5437): Collecting agent '003' software.

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:4370 at wm_vuldet_get_software_info(): DEBUG: (5439): A partial scan will be run on agent '003'

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:4583 at wm_vuldet_get_software_info(): DEBUG: (5445): No changes have been found with respect to the last package inventory or no packages have been indexed for agent '003'

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:2119 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5471): Finished vulnerability assessment for agent '003'

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:2120 at wm_vuldet_check_agent_vulnerabilities(): DEBUG: (5470): It took '0' seconds to 'scan' vulnerabilities in agent '003'

I'm goig to attach ossec.log ossec.conf files.

Thanks for your time to check this issue.

Best Regards

Gracias!

Giorgio Solari

[Alvaro Romero Sepulveda]

Hi Giorgio Solari,

Thank you very much for both your log and configuration! Both ossec.conf and ossec.log files look perfectly normal. However, I noticed that your manager is only performing partial scans to most of your agents:

2021/04/15 20:02:51 wazuh-modulesd:vulnerability-detector[1559] wm_vuln_detector.c:2078 at wm_vuldet_check_agent_vulnerabilities(): INFO: (5450): Analyzing agent '003' vulnerabilities.
2021/04/15 20:02:51 wazuh-modulesd:vulnerability-detector[1559] wm_vuln_detector.c:4352 at wm_vuldet_get_software_info(): DEBUG: (5437): Collecting agent '003' software.
2021/04/15 20:02:51 wazuh-modulesd:vulnerability-detector[1559] wm_vuln_detector.c:4370 at wm_vuldet_get_software_info(): DEBUG: (5439): A partial scan will be run on agent '003'

This is completely normal considering your configuration, as it happens when the <ignore_time> value is activated. Said value starts counting from zero each time the wazuh-modulesd daemon is restarted. However, it seems that systemctl restart wazuh-manager doesn't fully restart the said daemon (my fault!),  so your log only contains partial scans for agent 003. If possible, I'd still like to see logs from a full agent 003 scan, as with your agent '1823'. You can trigger them either by waiting for the ignore time to restart. One example I found in your log:

2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:4352 at wm_vuldet_get_software_info(): DEBUG: (5437): Collecting agent '1823' software.
2021/04/15 20:18:31 wazuh-modulesd:vulnerability-detector[3546] wm_vuln_detector.c:4363 at wm_vuldet_get_software_info(): DEBUG: (5438): A full scan will be run on agent '1823

Lastly, If we want to fully confirm the presence of a bug in your vulnerability detector, you could maybe try to install a explicitly vulnerable program in your agent 003 (like Mozilla Firefox 61.0) to check if the vulnerability detector detects it. I completely understand the risks involved in this and I'll find it completely understandable if you don't want to install vulnerable programs in your agents.

I hope this helps and thank you very much for your patience, I'll keep you updated whenever I find something relevant. This next Monday I'll probably be able to get help from more people from our team, so hopefully we'll made some progress regarding your issue. Also, mind that any additional information about your agent could be helpful (vulnerability diagnosis, alerts, etc.).

Best regards,

Álvaro Romero.

[Giorgio Solari]

Hi Álvaro,

Following your recommendations, I made the installation of Firefox 61.0. After that I decided to reboot the manager, now the vulnerability report has information, but all about Firefox.

Please let me know any additional file or screenshot that you need.

Attached is a new ossec.log file the report and a screenshot of the vulnerabilities.  

Thanks for your Help!

Best regards,

Giorgio Solari

[Alvaro Romero Sepulveda]

Hello Giorgio,

Thank you very much for providing this information about your agent 003. It has been very helpful while studying your issue.

I think I've finally found the reason why your agent is reporting zero vulnerabilities, in contrast to both your agent 001 and my own testing environment. As I told you, all the testing I've done so far has been made on a Windows Server 2016 environment. However, after checking your agent 003's report, It seems that said agent operating system is Windows Server 2012 R2. Considering that I was testing your hotfixes in a newer operating system, that should explain the incredibly high amount of vulnerabilities that were being reported in my environment.

With this in mind, I've tested from scratch in a Windows server 2012 R2 testing environment and I've ended up with drastically different vulnerability diagnoses: There are hardly any vulnerabilities with the hotfixes you've shared! The only vulnerabilities (apart from the Firefox 61.0 that I installed for testing purposes) I've found are those related to the patch KB5001382

2021/04/19 18:21:06 wazuh-modulesd:vulnerability-detector[64494] wm_vuln_detector.c:1525 at wm_vuldet_send_cve_report() : DEBUG: (5467): Agent '142' is vulnerable to 'CVE-2021-26415'. Condition: 'KB5001382 patch is not installed'.   

The information for this patch may have probably been included in our latest MSU feed, meaning that your scan from last week was probably accurate.

That being said, if you want to do a final check to discard any possibility of an error, you can enable increase the debug mode of wazuh-modulesd to 2 in local_internal_options.conf (as I explained in my second message) and wait for the full scan of agent 003 to happen (you can also disable the ignore_time value). In this way, you should find in your logs highly detailed information about both your agent vulnerability status and fixes. 

If your agent still lacks the KB5001382 patch, your MSU is updated to the latest version, and your agent still does not report the mentioned vulnerability, we could be talking about a false negative related to the mentioned patch. You can find more information about this patch here (it seems that it only affects x64 systems). That being said, your vulnerability detector seems to be working perfectly fine and It's very likely that your manager is reporting the right amount of vulnerabilities, as your agents 001 and 003 differ quite drastically in both hotfixes and packages (and even operating system!).

Thank you very much for your patience and I hope I helped you!

Best regards,

Álvaro Romero.

[Giorgio Solari]

Hi Álvaro,

Thanks a lot for all your time and effort to answer my question. I will rise up the debug level to check the logs.

Best regards,

Giorgio Solari