Vulnerability detection time

[aaa bbb]

Hi,

Is Wazuh capable of quickly checking vulnerabilities of about 1500 agents?

It seems that vulnerability scans are performed sequentially in single thread.

So is it true that to fit scanning of 1500 agents in 24 hours, scan of single agent can not be longer than ~57 seconds?  86400 seconds (24 hours) divide by 1500 agents equals 57,6s.

What is or should be average time of vuln. scanning per one Centos7 agent?

In our environment it is about 400 seconds, which seems way too much.

About 80% of our agents are Centos7.

What can be done to make it faster?

Is it possible to parallelize vulnerability scans?

What is the reason of doing scans in single thread, problems with SQLite concurrent access maybe (like i.e. lock contention)?

[Chema Martinez]

Hi,

We are aware that Vulnerability Detector may take more time than the desired to scan agents. In addition, CentOS and RHEL agents are the ones heaviest due to the number of vulnerabilities to be scanned.

From the Wazuh side, as you discovered this is mainly caused by the lack of multithreading when running the scan. The reason is that the product architecture where Vulnerability Detector runs inside the Wazuh modules daemon which provides one thread per component. In addition, originally it was OK for the scanner load. Currently, we know this is not enough so it will be solved in the future of course.

You have several options to improve the vulnerability detection performance in your environment:

• Having more than 1K agents you should deploy a Wazuh cluster of two or more nodes. That way the agents' load is spread between the nodes and scanning times are decreased. I paste here the link to the cluster documentation: https://documentation.wazuh.com/current/user-manual/configuring-cluster/index.html
• You can also increase the hardware resources of the manager host if possible. Here you have some guidance data for this purpose: https://documentation.wazuh.com/current/quickstart.html#hardware

I hope it helps.

Regards,

Chema.

[Chema Martinez]

Hi again,

I forgot to tell you a very important thing related to this.

We analyzed this issue a few months ago, here you can see all the analyses on the Vulnerability Detector performance: https://github.com/wazuh/wazuh/issues/9188

After a deeper analysis, we determined that the main waste of time was in the DB queries due to the SQLite used by Wazuh. Therefore, for Wazuh 4.3.0 we updated the SQLite version used and the results were significantly better. Here you can see the pull request where we included the changes: https://github.com/wazuh/wazuh/pull/10247

I strongly recommend that you upgrade your manager to 4.3.1 and repeat the test to see if the scan times decrease.

Best regards,

Chema.

[aaa bbb]

Hi,

We have upgraded to 4.3.1 version of the manager. Would you kindly answer couple of our questions to clarify how vulnerabiliy detector works in 4.3?

1) We need current state of vulnerabilities for given agent. Is this information available on API endpoint /vulnerability/{agent_id} ?

2) What could be the reason that for some active agents VULN_CVES table is empty (and /vulnerability andpoint returns no vulnerabilities) ?

3) It is not possible at the moment to manually execute vulnerabiity scans? There is open issue: Improve the Vulnerability Detector and Syscollector modules to allow a manual scan executions https://github.com/wazuh/wazuh/issues/9220

Best regards,

Przemek.

[Chema Martinez]

Hi Przemek,

Of course, regarding your questions:

1. That information is available in the UI, in the Vulnerabilities section -> Inventory since 4.3.0. This is retrieved through the API so you also have an endpoint available to request the vulnerabilities of each agent. See: https://documentation.wazuh.com/current/user-manual/api/reference.html#operation/api.controllers.vulnerability_controller.get_vulnerability_agent That information is up-to-date to the latest performed scan for each agent (full or partial).
2. Probably the reason is a failure when evaluating that agent. Please review the file /var/ossec/logs/ossec.log in the manager to look for any error related to that agents when running the vulnerability scanner.
3. That issue is in progress right now. It will be available in a future release. We hope soon.

Finally, I encourage to upgrade your agents at least to 4.x to ensure proper detection of vulnerabilities for those agents.

Best regards,

Chema.

[aaa bbb]

Thank you, this is good news!

In 2. it seems that no baseline scans performed yet for agent, empty VULN_CVES table is populated with data after baseline scan

Best regards,

Przemek.

[Chema Martinez]

Hi Przemek,

How many agents are facing that issue? Have they anything in common such as the Wazuh version or the OS?

Please, identify one of them and run the following command on the manager side, if, for example, its ID is 007:

# cat /var/ossec/logs/ossec.log | grep "007"

Best regards,

Chema.