Vulnerability Detector - No Events

[Steve]

Hi All,

   I've been trying to get the Vulnerability Detector to work, but so far nothing I've done worked. No events are ever generated and the dashboard always says " No results match your search criteria ". I've triple-check everything in the troubleshooting guide.

Output from cat /var/ossec/logs/ossec.log | grep -i -E "warn|error|vuln"

2024/09/06 16:54:41 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process
2024/09/06 17:07:29 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update
2024/09/06 17:07:29 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed

Then this error repeated over and over:

2024/09/06 17:07:30 wazuh-modulesd:vulnerability-scanner: ERROR: Error processing delayed event: Error deleting data: Invalid argument: Invalid column family specified in write batch

I suspect part of the problem may be that I can't get the cluster out of yellow status. I can this command (with the correct user:pass) to deleted the unassigned shards, but it still shows yellow and 1 unassigned shared afterwards.

curl -k -XGET -u user:pass "https://<elasticsearxch>:9200/_cat/shards" | grep UNASSIGNED | awk '{print $1}' | xargs -i curl -k -XDELETE -u user:pass "https://<indexer_ip>:9200/{}"

GET _cat/shards?h=index,shard,prirep,state,unassigned.reason

.opendistro-ism-managed-index-history-2024.09.06-1 0 p STARTED    
.opendistro-ism-managed-index-history-2024.09.06-1 0 r UNASSIGNED INDEX_CREATED

Thanks in advance for any assistance.

Steve

[Federico Ramos]

Hi Steve,

Can you pass the Indexer logs? Could you please check if the index pattern for the wazuh-states-vulnerabilities-*? If the index pattern is not created, there will be no information for this. You can find this in the top left menu ☰ > Dashboard Management > Dashboard Management, and then click on Index Patterns.

Another question, was your Wazuh environment working correctly before using the VD? To validate if it is a problem with the Indexer itself or the manager. Would you be so kind as to pass your ossec.conf, especially the VD and cluster configurations.

Thanks

[Steve]

Hi Federico,

 Current log for the indexer and ossec.conf attached. Yes, wazuh-states-vulnerabilities-* does exist.

  The vulnerability detector has not ever worked unfortunately.

  I have confirmed that the certificates are good using this command:

curl -u admin:pass --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/wazmaster.pem --key /etc/filebeat/certs/wazmaster-key.pem -X GET "https://192.168.117.142:9200/_cluster/health?pretty"

Thanks,

Steve

[Steve]

Any thoughts? I'm not sure what else to try.

Thanks,

Steve

[Federico Ramos]

Hi, sorry for the delay in the responses, Steve.

I asked the team for some advice on what could be the possible solution for your Issue. One of the possibilities is to clean and re-start the module, as suggested in this comment https://github.com/wazuh/wazuh/issues/24410#issuecomment-2209130497.

Let us know if this helps.

[Steve]

Hi Federico,

   Thanks for your response. I have disabling/re-enabling the vulnerability detector before and have done so again today. I updated ossec.conf on both my Master node and my Worker node.

Logs from my Master node:

cat /var/ossec/logs/ossec.log | grep -i -E "vuln"
2024/09/13 15:43:18 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module.
2024/09/13 15:43:31 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/09/13 15:43:32 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module is disabled
2024/09/13 15:47:48 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module.
2024/09/13 15:48:02 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/09/13 15:48:02 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilis-wazuh_cluster.
2024/09/13 15:48:02 wazuh-modulesd:vulnerability-scanner: INFO: Policy changed. Re-scanning all agents
2024/09/13 15:48:06 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module started

Logs from my worker node:

cat /var/ossec/logs/ossec.log | grep -i -E "vuln"
2024/09/13 15:42:02 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module.
2024/09/13 15:42:14 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/09/13 15:42:14 wazuh-modulesd:vulnerability-scanner: INFO: Vulnerability scanner module is disabled
2024/09/13 15:50:02 wazuh-modulesd:vulnerability-scanner: INFO: Stopping vulnerability_scanner module.
2024/09/13 15:50:15 wazuh-modulesd:vulnerability-scanner: INFO: Starting vulnerability_scanner module.
2024/09/13 15:50:16 indexer-connector: INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-wazuh_cluster.
2024/09/13 15:50:16 wazuh-modulesd:vulnerability-scanner: ERROR: VulnerabilityScannerFacade::start: Failed to open RocksDB database. Reason: While opening a file for sequentially reading: queue/vd/event/MANIFEST-000005: No such file or directory

It appears the RockDB database problem has re-surfaced on the worker node. I had previously resolved that by stopping the manager, removing the folder /var/ossec/queue/indexer/wazuh-states-vulnerabilities-siem/, and restarting the manager as described here: https://github.com/wazuh/wazuh/issues/24151#issuecomment-2179825266

It has been over 2 hours and the Dashboard is still empty with no events.

Thanks,

Steve

[Steve]

I posted this on Friday, but it doesn't seem to have gone through. There are still no events and the vulnerability detector dashboard is still empty.

[Steve]

Hello,

  I added "wazuh_modules.debug=2" to /var/ossec/etc/local_internal_options.conf and restarted the manager. The logs now show these messages for all the agents which appear to explain the lack of events. What can be done to fix this?

2024/09/17 13:49:47 wazuh-modulesd:content-updater[32281] action.hpp:221 at runAction(): DEBUG: Action for 'vulnerability_feed_manager' finished
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] vulnerabilityScannerFacade.cpp:450 at start(): INFO: Vulnerability scanner module started
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] scanAgentList.hpp:71 at scanAgentOs(): DEBUG: Empty response for agent '001' in Wazuh-DB 'sys_' query
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] scanAgentList.hpp:247 at handleRequest(): DEBUG: Error executing query to fetch agent data for agents. Reason: DB query not synced.
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] scanOrchestrator.hpp:143 at operator()(): DEBUG: AgentReScanListException. Reason: Error executing rescan for multiple agents.
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] scanAgentList.hpp:71 at scanAgentOs(): DEBUG: Empty response for agent '002' in Wazuh-DB 'sys_' query
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] scanAgentList.hpp:247 at handleRequest(): DEBUG: Error executing query to fetch agent data for agents. Reason: DB query not synced.
2024/09/17 13:49:48 wazuh-modulesd:vulnerability-scanner[32281] scanOrchestrator.hpp:143 at operator()(): DEBUG: AgentReScanListException. Reason: Error executing rescan for multiple agents.

Thanks,

Steve

[Steve]

Good news. I've been digging into this further and now have detections on the dashboard and events in that tab. The logs posted last time were from our master node. The agents connect to a worker node. I found that the agent databases on the master node (/var/ossec/queue/db/xxx.db) do not have any syscollector data in them. The tables are all empty, which explains the error messages. The agent databases on the worker node does contain the syscollector data.

Looking closer at the prior logs, I realized the RocksDB issue that I said came back was actually a different database than the one that was broken initially. In this case it was in /var/ossec/queue/vd/event/. Initially, it was in /var/ossec/queue/indexer/wazuh-states-vulnerabilities-siem/.

I stopped wazuh-manager.

Removed /var/ossec/queue/vd/event/  folder

and restarted wazuh-manager to have the system recreate the event folder.

After that I, it started populating the VD dashboard. Now on to make sure that the detections are valid....

Steve

[Commercial League]

Thanks, this helped me also.