indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK.

[Zero Two]

Greetings:

Since the 4.8, I have not been able to get the Vulnerability Detection to work with our Active Directory PKI certs (using the Wazuh self-signed certs was hit or miss).  Every other part of Wazuh works fine (or apparently so), its just the Vulnerability Detection module.  I have set wazuh_modules.debug = 2 and this is what the various output, is:

  cat /var/ossec/logs/ossec.log |grep vuln:

2024/09/17 13:10:13 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 16 seconds.
2024/09/17 13:10:29 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 32 seconds.

2024/09/17 13:13:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:14:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:15:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:16:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:17:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:18:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:19:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:20:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:21:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:22:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:23:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index '

cat /var/ossec/logs/ossec.log |grep index:

2024/09/17 13:23:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/17 13:24:01 indexer-connector[1212314] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.

Further Testing:

root@wazuhserver:~# filebeat test output

elasticsearch: https://192.168.128.108:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.128.108
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.2
    dial up... OK
  talk to server... OK
  version: 7.10.2



root@wazuhserver:~# curl -u admin:<redacted> --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/wazuhserver.foo.bar-fullchain.pem --key /etc/filebeat/certs/wazuhserver.foo.bar.key -X GET "https://192.168.128.108:9200/_cluster/health?pretty=true"
{
  "cluster_name" : "wazuh-indexer-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 823,
  "active_shards" : 823,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.75757575757575
}
root@wazuhserver:~#

Some additional error output:

2024/09/17 17:27:12 wazuh-modulesd:vulnerability-scanner[1232413] wm_vulnerability_scanner.c:52 at wm_vulnerability_scanner_main(): INFO: Starting vulnerability_scanner module.
2024/09/17 17:27:12 wazuh-modulesd[1232413] main.c:95 at main(): DEBUG: Created new thread for the 'vulnerability_scanner' module.
2024/09/17 17:27:12 wazuh-modulesd:vulnerability-scanner[1232413] wm_vulnerability_scanner.c:45 at wm_vulnerability_scanner_log_config(): DEBUG: {"vulnerability-detection":{"enabled":"yes","index-status":"yes","feed-update-interval":"60m","cti-url":"https://cti.wazuh.com/api/v1/catalog/contexts/vd_1.0.0/consumers/vd_4.8.0"},"wmMaxEps":100,"translationLRUSize":2048,"osdataLRUSize":1000,"remediationLRUSize":2048,"managerDisabledScan":1,"indexer":{"enabled":"yes","hosts":["https://192.168.128.108:9200"],"ssl":{"certificate_authorities":["/etc/filebeat/certs/root-ca.pem"],"certificate":"/etc/filebeat/certs/filebeat.pem","key":"/etc/filebeat/certs/filebeat-key.pem"}},"clusterEnabled":false,"clusterName":"wazuhserver","clusterNodeName":"undefined"}
2024/09/17 17:27:13 indexer-connector[1232413] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 2 seconds.
2024/09/17 17:27:13 indexer-connector[1232413] indexerConnector.cpp:482 at operator()(): WARNING: IndexerConnector initialization failed for index 'wazuh-states-vulnerabilities-wazuhserver', retrying until the connection is successful.
2024/09/17 17:27:13 wazuh-modulesd:vulnerability-scanner[1232413] vulnerabilityScannerFacade.cpp:457 at start(): ERROR: VulnerabilityScannerFacade::start: Failed to open RocksDB database. Reason: While opening a file for sequentially reading: queue/vd/event/MANIFEST-000005: No such file or directory.
2024/09/17 17:27:15 indexer-connector[1232413] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 4 seconds.

Any help is greatly appreciated!

[Obinna Uchubilo]

Hi,

Can you please verify that have you followed the vulnerability detection module https://documentation.wazuh.com/current/user-manual/capabilities/vulnerability-detection/configuring-scans.html.

Please make sure to update the <vulnerability-detection> and <indexer> block in /var/ossec/etc/ossec.conf in version 4.8.0. 

Replace `0.0.0.0` with  the indexer IP in the Filebeat config file, For example:
output.elasticsearch.hosts:
  - 127.0.0.1:9200

Wazuh indexer node's IP address or hostname. If you have a Wazuh indexer cluster, add a `<host>` entry for each one of your nodes. For example, in a two-node configuration:
<hosts>
  <host>https://10.0.0.1:9200</host>
  <host>https://10.0.0.2:9200</host>
</hosts>

Check the certificate name:
ll /etc/filebeat/certs
Verify the Filebeat certificate name and path are correct and update the `<indexer>` block in `/var/ossec/etc/ossec.conf` accordingly.
In case the certs is missing you can extract that from wazuh-certificates.tar which is created at the time of generating certs 

Save the Wazuh indexer username and password into the Wazuh manager keystore using the Wazuh-keystore tool:
/var/ossec/bin/wazuh-keystore -f indexer -k username -v <INDEXER_USERNAME>
/var/ossec/bin/wazuh-keystore -f indexer -k password -v <INDEXER_PASSWORD>

After that, save the configuration and restart the manager/cluster using the command:
systemctl restart wazuh-manager

If this didnt resolve the issue then please share the output of the following command again

cat /var/ossec/logs/ossec.log | grep vul

Refer: https://documentation.wazuh.com/current/upgrade-guide/troubleshooting.html.

Regards.

[Zero Two]

Obinna:

Thank you for your prompt response.  Here are the relevant portions of the ossec.conf file on the Wazuh-Manager node:

  <vulnerability-detection>
    <enabled>yes</enabled>
    <index-status>yes</index-status>
    <feed-update-interval>60m</feed-update-interval>
  </vulnerability-detection>

<ossec_config>
  <indexer>
    <enabled>yes</enabled>
    <hosts>
      <host>https://192.168.128.108:9200</host>
    </hosts>
    <ssl>
      <certificate_authorities>
        <ca>/etc/filebeat/certs/root-ca.pem</ca>
      </certificate_authorities>
      <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
      <key>/etc/filebeat/certs/filebeat-key.pem</key>
    </ssl>
  </indexer>

</ossec_config>

With respect to certs:

root@wazuhserver:~# readlink -f /etc/filebeat/certs/root-ca.pem

/etc/filebeat/certs/root-ca.pem

root@wazuhserver:~# readlink -f /etc/filebeat/certs/filebeat.pem
/etc/filebeat/certs/filebeat.pem
root@wazuhserver:~# readlink -f /etc/filebeat/certs/filebeat-key.pem
/etc/filebeat/certs/filebeat-key.pem

Other Info:

root@wazuhserver:~# systemctl restart wazuh-manager && tail -f /var/ossec/logs/ossec.log | grep vul
2024/09/18 14:45:14 indexer-connector[1307283] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 4 seconds.
2024/09/18 14:45:18 indexer-connector[1307283] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 8 seconds.
2024/09/18 14:45:26 indexer-connector[1307283] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 16 seconds.
2024/09/18 14:45:42 indexer-connector[1307283] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 32 seconds.
2024/09/18 14:45:43 wazuh-modulesd:vulnerability-scanner[1307283] osScanner.hpp:346 at handleRequest(): DEBUG: Vulnerability scan for OS 'Debian GNU/Linux' on Agent '034' has completed.
2024/09/18 14:45:43 wazuh-modulesd:vulnerability-scanner[1307283] eventDetailsBuilder.hpp:101 at handleRequest(): DEBUG: Building event details for component type: 2
2024/09/18 14:45:43 wazuh-modulesd:vulnerability-scanner[1307283] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 3 processed
2024/09/18 14:45:45 wazuh-modulesd:vulnerability-scanner[1307283] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/09/18 14:45:46 wazuh-modulesd:vulnerability-scanner[1307283] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/09/18 14:46:14 indexer-connector[1307283] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.

root@wazuhserver:~# ping 192.168.128.108
PING 192.168.128.108 (192.168.128.108) 56(84) bytes of data.
64 bytes from 192.168.128.108: icmp_seq=1 ttl=64 time=0.057 ms
64 bytes from 192.168.128.108: icmp_seq=2 ttl=64 time=0.079 ms
64 bytes from 192.168.128.108: icmp_seq=3 ttl=64 time=0.052 ms

root@wazuhserver:~# curl -u admin:<redacted> --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/filebeat.pem --key /etc/filebeat/certs/filebeat-key.pem -X GET "https://192.168.128.108:9200/_cluster/health?pretty=true"

{
  "cluster_name" : "wazuh-indexer-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,

  "active_primary_shards" : 827,
  "active_shards" : 827,

  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 2,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,

  "active_shards_percent_as_number" : 99.75874547647769
}

[Obinna Uchubilo]

Hi,

Please go to Dashboard Management > Index Pattern > wazuh-states-vulnerabilties-* > press "refresh" field list icon

Let me know if it resolves the issue.

Regards

[Zero Two]

Obinna:

Unfortunately, it did not; I am getting the same errors:

root@wazuhserver:~# systemctl restart wazuh-manager && tail -f /var/ossec/logs/ossec.log | grep vul

2024/09/19 12:11:38 indexer-connector[1386585] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 4 seconds.
2024/09/19 12:11:42 indexer-connector[1386585] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 8 seconds.
2024/09/19 12:11:50 indexer-connector[1386585] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 16 seconds.
2024/09/19 12:12:06 indexer-connector[1386585] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': SSL peer certificate or SSH remote key was not OK. Retrying in 32 seconds.
2024/09/19 12:12:38 indexer-connector[1386585] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.

[Obinna Uchubilo]

Hello,

Please confirm the state of the cluster health once again. Going through what was shared initially it was yellow. You can check it using the command below.
curl -u <user>:<pass> --cacert <path.pem> --cert <path-client.pem> --key <path-client-key.pem> -X GET "https://<IP>:9200/_cluster/health" 

A yellow or red status for indexer cluster health will prevent the Vulnerability module from indexing. Check if you have unassigned shards. To fix the cluster health issue, please follow the steps below

You need to change the plugins.security.system_indices.enabled: true to false in /etc/wazuh-indexer/opensearch.yml
Restart the wazuh-indexer using the command: systemctl restart wazuh-indexer.
Then from the wazuh interface UI. You need to change the replica no. For that Go to Index management >> Index Management > Indexes > click on the unassigned indices. 
Finally, change the Number of replicas to 0 and click on save.

[Zero Two]

I was able to remove the 1 unassigned shard however the other is proving problematic:

.opendistro-ism-config                   0     p      STARTED                   192.168.128.108 wazuhindexer.foo.bar
.opendistro-ism-config                   0     r      UNASSIGNED 

Not how I would go about getting rid of the replica                              

[Zero Two]

I was able to get rid of the the unassigned .opendistro-ism-config (had to be done via cURL).  The cluster health is now green.  I can see the vulnerability scan appears to be running but I am not sure if I can trust it.  For  instance, I have a number machines running Debian 12. All are patched/upgraded via APT and, in that sense, up to date (per APT).  Here are the scan results of three endpoints:

A:

B:

C:

Further, most (about 90%) of the agents don't have Dashboards or Inventory that reflect the Events.

Thank you

[Zero Two]

For comparison, here is a SentinelOne vulnerability scan on DD-0121-001L (Endpoint A, above):


Bit of a difference

[Obinna Uchubilo]

Hi,

It may require some time to update the vulnerability dashboard for the agents. You can get the comprehensive list of all vulnerable packages detected on the endpoint from the inventory. 

Regards

[Zero Two]

I mean, how long is long?  The main Dashboard hasn't changed in Vulnerability Count since the initial upgrade to 4.8 (which has been over a month).  

And despite clearing up the yellow shard issue, I am still seeing this (red):
2024/09/21 02:36:25 wazuh-modulesd:vulnerability-scanner[1459007] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/09/21 02:36:29 indexer-connector[1459007] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/21 02:37:29 indexer-connector[1459007] indexerConnector.cpp:474 at operator()(): DEBUG: Unable to initialize IndexerConnector for index 'wazuh-states-vulnerabilities-wazuhserver': No available server. Retrying in 60 seconds.
2024/09/21 02:37:32 wazuh-modulesd:vulnerability-scanner[1459007] osScanner.hpp:346 at handleRequest(): DEBUG: Vulnerability scan for OS 'Debian GNU/Linux' on Agent '040' has completed.
2024/09/21 02:37:32 wazuh-modulesd:vulnerability-scanner[1459007] eventDetailsBuilder.hpp:101 at handleRequest(): DEBUG: Building event details for component type: 2
2024/09/21 02:37:32 wazuh-modulesd:vulnerability-scanner[1459007] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 3 processed
2024/09/21 02:37:44 wazuh-modulesd:vulnerability-scanner[1459007] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/09/21 02:37:45 wazuh-modulesd:vulnerability-scanner[1459007] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/09/21 02:37:59 wazuh-modulesd:vulnerability-scanner[1459007] osScanner.hpp:346 at handleRequest(): DEBUG: Vulnerability scan for OS 'Debian GNU/Linux' on Agent '073' has completed.
2024/09/21 02:37:59 wazuh-modulesd:vulnerability-scanner[1459007] eventDetailsBuilder.hpp:101 at handleRequest(): DEBUG: Building event details for component type: 2
2024/09/21 02:37:59 wazuh-modulesd:vulnerability-scanner[1459007] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 3 processed 

Despite it reporting that the Vulnerabilty scan has completed for the Agents (green)

[esnak62]

Gracias a todos tenia el mismo problema, elimine los indices no asignados desde el dev tools del indexer y al reiniciar el manager se solvento,
Thanks to everyone, I had the same problem, I deleted the unassigned indexes from the indexer's dev tools and when I restarted the manager it was solved.

[Gary C. Hernandez]

Hello,

I have a similar problem. I run a Wazuh cluster consisting of 4 servers (1 Wazuh dashboard incl. Nginx load balancer, 1 Wazuh master and 2 Wazuh workers, each containing the wazuh-indexer and the wazuh-manager.

Since the update to v 4.9.1 I have a certification problem with the Vulnerability Scanner:
2024/10/23 15:03:29 wazuh-modulesd:content-updater: ERROR: Action for 'vulnerability_feed_manager' failed: orchestration run failed: error -1 from server: SSL peer certificate or SSH remote key was not OK.

I have already checked that the certificate paths and the host addresses in ossec.conf match the correct values (in filebeat.yml) as well as the correct credentials in the wazuh-keystore.

Is it possible that I have to distribute the filbeat certificates to the cluster hosts and specify them correctly in the respective ossec.conf?

Here is the cluster configuration with regard to the certificates: https://pastebin.com/QCRhv62s