Lots of "Failed to sync agent 'XXX' with the indexer." after 4.8 upgrade

[Alessio L]

Hi all,

I've upgraded our Wazuh to 4.8. All seems to works fine but I've some warnings on logs that worries me.
We have 2 server for wazuh: indexed + dashboard/manager, the password in the keystore are correct and the ossec.conf has been updated for the new vulnerabilities scanner. I've also checked that it contains only one <indexer> <\indexer> paragraph with the correct indexer ip.

Any suggestion on how to get rid of all those failed sync? Could they break our wazuh in  the future if ignored?

Here the logs redacted for readability, it's just a small part, the same warnings goes on for every agent

indexerConnector.cpp:366 at operator()(): DEBUG: Waiting for initialization thread to process events.
indexerConnector.cpp:319 at initialize(): INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-siem-server.
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '044' with the indexer.
indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '044' sync omitted due to abuse control.
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '025' with the indexer.
indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '025' sync omitted due to abuse control.
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:414 at operator()(): REDACTED very long JSON of CVE
indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '097' with the indexer.
indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '097' with the indexer.
indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '097' sync omitted due to abuse control.
indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '052' with the indexer.
indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '052' with the indexer.
indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '052' sync omitted due to abuse control.
indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '069' with the indexer.
indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '069' with the indexer.
indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server

[Octavio Valle López]

Hi, Could you check the health status?

This message indicates that your indexer is down or is in red or yellow status.

indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server

what is the output of this command ? please replace the <IP> with the indexer IP.

curl -XGET "http://<IP>:9200/_cat/health?v"

[Alessio L]

Sorry Octavio,

I've answered to you in private. I cannot retrieve the message.

You could be on the right path to troubleshoot my problem

With your command it responds:

curl: (52) Empty reply from server

with

curl -u <USER>:<PASSWORD> --cacert /etc/filebeat/certs/root-ca.pem --cert /etc/filebeat/certs/filebeat.pem --key /etc/filebeat/certs/filebeat-key.pem -X GET "https://<INDEXER IP>:9200/_cat/health?v"

it responds

epoch      timestamp cluster       status node.total node.data discovered_cluster_manager shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1718637837 15:23:57  wazuh-cluster yellow          1         1                       true    842 842    0    0       37             0                  -                 95.8%

I'm clueless, from the dashboard all seems fine: Threat alerts are generates and also Vulnerabilities alerts

Every help are appreciated

[Alessio L]

The issue is still present, but since the 

IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-siem-server.

I feel confident that is not a configuration issue on the manager.

I noticed that is like the indexer is flooded and stops to ingest vulnerabilities from the manager. The clue of this is that every now and then I got mail notification of new vulnerabilites from agents that weren't shown in the dashboard before (like 2-3 @ day)
This theory COULD be compatible with:

indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '044' sync omitted due to abuse control.

I'm still searching info about that (with no luck atm)

[Alberto Garro]

Hello,
You have to make the cluster status change to green since in any other state it gives DEBUG: Error: Not available server

If you have a single-node cluster you can try adding in /etc/wazuh-indexer/opensearch.yml:

discovery.type: single-node

And commenting:
#cluster.initial_master_nodes:
#- "windexer01"

In my case with a single indexer I had to configure the template indexes to 1 shard and 0 replicas so that the cluster status turned green.

Status control is in:
monitoring.hpp
143: fields.at(HealthCheckColumns::STATUS).compare("green") == 0)

[Alessio L]

@Alberto: the yellow status is due to another reason that I carry on since last year (.opendistro-alerting-alerts is set to 1 replica and  won't allow me to change to 0). It's totally unrelated to vulnerabilities. 

By looking here and on wazuh reddit I'm not the only one to have this issue, I'm starting to think that is a bug.
If my system was misconfigured, my Vulnerabilities Dashboard would show ZERO agents. Instead it's just delayed, as I said seems that it triggers some sort of abuse control that discards many (but NOT all) connection.

Il giorno mercoledì 26 giugno 2024 alle 15:16:07 UTC+2 Alan Baltic ha scritto:

Hi,

I will continue on this thread because i have same problem.
On debug I can also see No available server

2024/06/26 15:06:06 indexer-connector[2067106] indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '1112' with the indexer.
2024/06/26 15:06:06 indexer-connector[2067106] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1112' with the indexer.
2024/06/26 15:06:06 indexer-connector[2067106] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server


And the cluster is in green status:
{
  "cluster_name" : "Wazuh Cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 209,
  "active_shards" : 527,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

[Alessio L]

Sorry for the double post but I noticed something really peculiar.

Our wazuh runs on a cluster of 2 ubuntu 22.04 LTSVMs , I've just updated the kernel from 5.15.0-112 to 5.15.0-113, rebooted the system and now all sync errors are gone.
I still don't have any clue or explanation

[Alessio L]

Update: over the weekend the sync errors showed again

[Octavio Valle López]

Hi Alessio,

What API are you using to view the indexer status? It is this?

curl -XGET "http://localhost:9200/_cat/health?v"

https://opensearch.org/docs/latest/api-reference/cat/cat-health/

[Alessio L]

Hi Octavio,

yes, I use that endpoint to check the health. Now is green since I've fixed some old stuck indexes.

epoch      timestamp cluster       status node.total node.data discovered_cluster_manager shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent

1719996875 08:54:35  wazuh-cluster green           1         1                       true    841 841    0    0        0             0                  -                100.0%

The problem about failed sync agents is still present though

Since I'm not the only one facing this issue I don't think that is related to my specific configuration, is there any way to temporary disable indexer-connector abusecontrol()? Or at least make it more permissive?

[Octavio Valle López]

Hi Alessio

So not having it in yellow state caused synchronization problems.

Errors will probably be purged over time.


Regarding modifying the abuse-control, this is not configurable since it is to avoid flooding the opensearch server in case you have a bad configuration in the syscollector agent.