Failed to sync agent with indexer 4.8

[Alan Baltic]

  Hello,

So the problem is I see a lot of messages in manager like this bellow  log after 4.7.2 -> 4.8.0 upgrade:

WARNING: Failed to sync agent '1086' with the indexer. 

Also Vulnerability Detector Dashboard is not updated.

I have red some links and tried to figure it out by myself but with no luck :(

similar problem 1
similar problem 2
git issue closed

My setup is:

OS: RHEL 9 (for Wazuh central components)

Wazuh cluster (1 master, 1 worker)

Wazuh indexer (3 nodes - all masters)

53 Agents (all RHEL  -> 7 - 9)

VM's (IP's not real only for this post)

1. Wazuh Manager, Wazuh indexer node 1, Dashboard, Filebeat (10.10.10.11)
2. Wazuh indexer node 2 (10.10.10.12)
3. Wazuh indexer node 3 (10.10.10.13)
4. Wazuh Worker (10.10.10.14)
5. HaProxy which is load balancing traffic between agents and Wazuh(I also tried with and without HaProxy - issue is the same)
6. Keycloak used for authenticating - following this guide

Using certificates with full chain ( intermediate and root )

Certificates have SAN.
CN's are names (example: indexer01) and in SAN is IP.
Filebeat have configured same certificate as Indexer node 1 (they are on same server) and it is only sending events to Indexer node 1

Check:

ossec.conf - VD (on both managers) (indexer is in the and of config with separated ossec_config block - also tried with sam block where is VD)

INDEXER in ossec.conf:

VD in ossec.conf:

Offline feeds gets updated successfully and indexer is connected while also vulnerability index is created:

I also red the indexerConnector code to see if I can find something, and as far as I can understand is that "No available server" is response from Wazuh indexer (I am not a programmer but I gave it a shot)

In dashboard I can see to much vulnerabilities but not for all agents. 

Every day I can see messages like these every 60min (when VD sync is supposed to happen):

I restarted everything multiple times, redirected all agents only to wazuh master or only to wazuh slave, connected agents with or without proxy...

Then I restarted Wazuh indexer node 1, and shortly after I saw a lot of these "Failed to sync agent" messages so my main suspect is that Wazuh indexer is somehow overwhelmed with requests (I did not see nothing special in indexer debug log - enabled it thru log4j -> root logger on DEBUG ). 

Here are my other configs:

filebeat:

Indexer ( 2 pics):

Dashboard:

Everything else is working as expected.

I hope that you will find time to point me in right direction.

Thank you

Alan

[Alan Baltic]

Forgot to mention that all agents are 4.8.0 also

[Alan Baltic]

Does anyone else have the same problem?
I configured new certificates for every component to see if its problem there maybe, and also deleted vd and vd_updater folders alongside with the vulnerability index to try everything from the scratch. At the moment I can see "Failed to sync agent" warning on the worker node, while vulnerability scanner is working on the master node.

Also I noticed that vd and vd_updater folders in /var/ossec/queue/ are both owned by root (root:root). Is that like its supposed to be? Because every other file/folder is either root:wazuh or wazuh:wazuh

Thank you

Alan

[Alan Baltic]

UPDATE:
"failed to sync agent" messages now started on master also

indexer-connector[2184605] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1109' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
wazuh-modulesd:vulnerability-scanner[2184605] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
indexer-connector[2184605] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1109' sync omitted due to abuse control.
wazuh-modulesd:vulnerability-scanner[2184605] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
indexer-connector[2184605] indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '1086' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1086' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
wazuh-modulesd:vulnerability-scanner[2184605] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
indexer-connector[2184605] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1086' sync omitted due to abuse control.

[Alessio L]

Hi Alan,

I noticed the exact same behaviour on our wazuh. The Vuln Alerts come in/updates at a very slow rate, like 3-4 agent @ day.

I suspect that the "abuse-control" is the culprit, could be that the indexer detects all the incoming vulns json as flood?
I had no luck in finding a solution yet and I don't even know where else to look

[Javier Sanchez Gil]

Hi Alan Baltic,

I have been reviewing all your contributions to help resolve the situation.

I was reviewing the log output and the problem when attempting to sync a specific agent with the indexer:

indexer-connector[2184605] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1109' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server

I saw that you already updated both the <vulnerability-detection> and <indexer> blocks in /var/ossec/etc/ossec.conf for version 4.8.0.

Wazuh indexer node IP address or hostname. If you have a Wazuh indexer cluster, add a <host> entry for each of your nodes. For example, in a three-node configuration:

<hosts>
  <host>https://10.10.10.11:9200/</host>
  <host>https://10...</host>
  <host>https://10...</host>
</hosts>

Check the certificate name: ll /etc/filebeat/certs. Verify the Filebeat certificate name and path are correct and update the <indexer> block in /var/ossec/etc/ossec.conf accordingly.
Save the Wazuh indexer username and password into the Wazuh manager keystore using the Wazuh-keystore tool:

/var/ossec/bin/wazuh-keystore -f indexer -k username -v <INDEXER_USERNAME>
/var/ossec/bin/wazuh-keystore -f indexer -k password -v <INDEXER_PASSWORD>

Then, save the configuration and restart the manager/cluster using the command: systemctl restart wazuh-manager.

Refer to:
https://documentation.wazuh.com/current/user-manual/capabilities/vulnerability-detection/configuring-scans.html
https://documentation.wazuh.com/current/upgrade-guide/troubleshooting.html

[Alan Baltic]

Hi Javier,

thanks for having time to review.

From previous posts you can see that I deleted vd and vd_updater folders to see if recreation would take effect in resolving this. Also yesterday I deleted again vd, vd_updater and indexer folder. All these folders are in /var/ossec/queue alongside with vulnerability index. After that action I was only able to see empty vulnerability index ~200 bytes with only one Wazuh indexer host configured in indexer block in ossec.conf.

After making suggested changes in Wazuh master and Wazuh worker it seems that I can see inventory only for one agent and events are empty.



ossec.log
2024/07/02 11:41:31 indexer-connector[2220215] indexerConnector.cpp:319 at initialize(): INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-mycluster.
---
2024/07/02 12:14:36 indexer-connector[2220215] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1086' with the indexer.
2024/07/02 12:14:36 indexer-connector[2220215] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
2024/07/02 12:14:36 wazuh-modulesd:vulnerability-scanner[2220215] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/07/02 12:14:36 indexer-connector[2220215] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1086' sync omitted due to abuse control.
2024/07/02 12:14:46 wazuh-modulesd:vulnerability-scanner[2220215] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/07/02 12:14:46 indexer-connector[2220215] indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '1103' with the indexer.
2024/07/02 12:14:46 indexer-connector[2220215] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1103' with the indexer.
2024/07/02 12:14:46 indexer-connector[2220215] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
2024/07/02 12:14:47 wazuh-modulesd:vulnerability-scanner[2220215] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/07/02 12:14:47 indexer-connector[2220215] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1103' sync omitted due to abuse control.

Differences I made:

1. Changes in ossec.conf  (indexer block) are on both master and worker:
   
   OLD:
   <indexer>
       <enabled>yes</enabled>
       <hosts>
         <host>https://10.10.10.11:9200</host>
       </hosts>
       <ssl>
         <certificate_authorities>
           <ca>/etc/filebeat/certs/intermed-ca.pem</ca>
           <ca>/etc/filebeat/certs/root-ca.pem</ca>
         </certificate_authorities>
         <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
         <key>/etc/filebeat/certs/filebeat-key.pem</key>
       </ssl>
     </indexer>
   
   NEW:
   <indexer>
       <enabled>yes</enabled>
       <hosts>
         <host>https://10.10.10.11:9200</host>
         <host>https://10.10.10.12:9200</host>
         <host>https://10.10.10.13:9200</host>
       </hosts>
       <ssl>
         <certificate_authorities>
           <ca>/etc/filebeat/certs/intermed-ca.pem</ca>
           <ca>/etc/filebeat/certs/root-ca.pem</ca>
         </certificate_authorities>
         <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
         <key>/etc/filebeat/certs/filebeat-key.pem</key>
       </ssl>
     </indexer>
   
   
2. Check filebeat cert files on both master and worker:
    Wazuh - Filebeat configuration file
   output.elasticsearch:
     hosts: ["10.10.10.11 :9200","10.10.10.12:9200","10.10.10.13 :9200"]
   #  hosts: ["10.10.10.11:9200"] #Tried with on or all hosts in cluster
     protocol: https
     username: ${username}
     password: ${password}
     ssl.certificate_authorities: ["/etc/filebeat/certs/intermed-ca.pem","/etc/filebeat/certs/root-ca.pem"]
     ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
     ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
   setup.template.json.enabled: true
   setup.template.json.path: '/etc/filebeat/wazuh-template.json'
   setup.template.json.name: 'wazuh'
   setup.ilm.overwrite: true
   setup.ilm.enabled: false
   
   
3. Wazuh-keystore (I did it again to be sure):
   
   /var/ossec/bin/wazuh-keystore -f indexer -k username -v {same_user_for_login_to_wazuh_or_curl_wazuh_indexer} (default: admin)
   /var/ossec/bin/wazuh-keystore -f indexer -k password -v {same_password_for_login_to_wazuh_or_curl_wazuh_indexer} (default: admin)
   
   
4.  Chuck cluster health (did curl on all three nodes to be sure every node have expected response):
   
   curl -u admin:admin --cacert /etc/filebeat/certs/intermed-ca.pem --cert /etc/filebeat/certs/filebeat.pem --key /etc/filebeat/certs/filebeat-key.pem -X GET "https://10.10.10.11:9200/_cluster/health?pretty"
   {
     "cluster_name" : "MyCluster",
     "status" : "green",
     "timed_out" : false,
     "number_of_nodes" : 3,
     "number_of_data_nodes" : 3,
     "discovered_master" : true,
     "discovered_cluster_manager" : true,
     "active_primary_shards" : 236,
     "active_shards" : 600,
     "relocating_shards" : 0,
     "initializing_shards" : 0,
     "unassigned_shards" : 0,
     "delayed_unassigned_shards" : 0,
     "number_of_pending_tasks" : 0,
     "number_of_in_flight_fetch" : 0,
     "task_max_waiting_in_queue_millis" : 0,
     "active_shards_percent_as_number" : 100.0
   }
   
   

One last thing regarding certificates:

Every certificate is signed by intermed-ca in my case. (intermed-ca is signed by root-ca)

Ceritifcates for every component have CN hostname and SAN as IP address of the host where this component should be.

So filebeat certs (/etc/filebeat/certs/filebeat.yml) in:
wazuh master is:
CN: master01
SAN: 10.10.10.11

wazuh worker is:
CN: worker01
SAN: 10.10.10.14

Wazuh indexer cert for indexer01:
CN: indexer01
SAN: 10.10.10.11 (on same server as wazuh master)

Wazuh indexer cert for indexer02:
CN: indexer01
SAN: 10.10.10.12 

and so on..

I apologize for longer message but I am trying to specify as much details as possible for someone who will see these afterwards

Thank you

[Alan Baltic]

UPDATE:

I can still see inventory packages only for one agent in vulnerability detector, but relatively good news is that I am able to see Events for 3 agents for now (31 hits).

Inventory shows only for this one agent 7138 hits  which is quite a lot (OS updated), but in Events I am able to see only 9,10 vulnerabilities per agent for those 3 agents.

Hopefully it takes time to scan all of them. Failed to sync messages are still present though.

Thank you