Failed to sync agent with indexer 4.8

[Alan Baltic]

  Hello,

So the problem is I see a lot of messages in manager like this bellow  log after 4.7.2 -> 4.8.0 upgrade:

WARNING: Failed to sync agent '1086' with the indexer. 

Also Vulnerability Detector Dashboard is not updated.

I have red some links and tried to figure it out by myself but with no luck :(

similar problem 1
similar problem 2
git issue closed

My setup is:

OS: RHEL 9 (for Wazuh central components)

Wazuh cluster (1 master, 1 worker)

Wazuh indexer (3 nodes - all masters)

53 Agents (all RHEL  -> 7 - 9)

VM's (IP's not real only for this post)

1. Wazuh Manager, Wazuh indexer node 1, Dashboard, Filebeat (10.10.10.11)
2. Wazuh indexer node 2 (10.10.10.12)
3. Wazuh indexer node 3 (10.10.10.13)
4. Wazuh Worker (10.10.10.14)
5. HaProxy which is load balancing traffic between agents and Wazuh(I also tried with and without HaProxy - issue is the same)
6. Keycloak used for authenticating - following this guide

Using certificates with full chain ( intermediate and root )

Certificates have SAN.
CN's are names (example: indexer01) and in SAN is IP.
Filebeat have configured same certificate as Indexer node 1 (they are on same server) and it is only sending events to Indexer node 1

Check:

ossec.conf - VD (on both managers) (indexer is in the and of config with separated ossec_config block - also tried with sam block where is VD)

INDEXER in ossec.conf:

VD in ossec.conf:

Offline feeds gets updated successfully and indexer is connected while also vulnerability index is created:

I also red the indexerConnector code to see if I can find something, and as far as I can understand is that "No available server" is response from Wazuh indexer (I am not a programmer but I gave it a shot)

In dashboard I can see to much vulnerabilities but not for all agents. 

Every day I can see messages like these every 60min (when VD sync is supposed to happen):

I restarted everything multiple times, redirected all agents only to wazuh master or only to wazuh slave, connected agents with or without proxy...

Then I restarted Wazuh indexer node 1, and shortly after I saw a lot of these "Failed to sync agent" messages so my main suspect is that Wazuh indexer is somehow overwhelmed with requests (I did not see nothing special in indexer debug log - enabled it thru log4j -> root logger on DEBUG ). 

Here are my other configs:

filebeat:

Indexer ( 2 pics):

Dashboard:

Everything else is working as expected.

I hope that you will find time to point me in right direction.

Thank you

Alan

[Alan Baltic]

Forgot to mention that all agents are 4.8.0 also

[Alan Baltic]

Does anyone else have the same problem?
I configured new certificates for every component to see if its problem there maybe, and also deleted vd and vd_updater folders alongside with the vulnerability index to try everything from the scratch. At the moment I can see "Failed to sync agent" warning on the worker node, while vulnerability scanner is working on the master node.

Also I noticed that vd and vd_updater folders in /var/ossec/queue/ are both owned by root (root:root). Is that like its supposed to be? Because every other file/folder is either root:wazuh or wazuh:wazuh

Thank you

Alan

[Alan Baltic]

UPDATE:
"failed to sync agent" messages now started on master also

indexer-connector[2184605] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1109' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
wazuh-modulesd:vulnerability-scanner[2184605] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
indexer-connector[2184605] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1109' sync omitted due to abuse control.
wazuh-modulesd:vulnerability-scanner[2184605] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
indexer-connector[2184605] indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '1086' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1086' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
wazuh-modulesd:vulnerability-scanner[2184605] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
indexer-connector[2184605] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1086' sync omitted due to abuse control.

[Alessio L]

Hi Alan,

I noticed the exact same behaviour on our wazuh. The Vuln Alerts come in/updates at a very slow rate, like 3-4 agent @ day.

I suspect that the "abuse-control" is the culprit, could be that the indexer detects all the incoming vulns json as flood?
I had no luck in finding a solution yet and I don't even know where else to look

[Javier Sanchez Gil]

Hi Alan Baltic,

I have been reviewing all your contributions to help resolve the situation.

I was reviewing the log output and the problem when attempting to sync a specific agent with the indexer:

indexer-connector[2184605] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1109' with the indexer.
indexer-connector[2184605] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server

I saw that you already updated both the <vulnerability-detection> and <indexer> blocks in /var/ossec/etc/ossec.conf for version 4.8.0.

Wazuh indexer node IP address or hostname. If you have a Wazuh indexer cluster, add a <host> entry for each of your nodes. For example, in a three-node configuration:

<hosts>
  <host>https://10.10.10.11:9200/</host>
  <host>https://10...</host>
  <host>https://10...</host>
</hosts>

Check the certificate name: ll /etc/filebeat/certs. Verify the Filebeat certificate name and path are correct and update the <indexer> block in /var/ossec/etc/ossec.conf accordingly.
Save the Wazuh indexer username and password into the Wazuh manager keystore using the Wazuh-keystore tool:

/var/ossec/bin/wazuh-keystore -f indexer -k username -v <INDEXER_USERNAME>
/var/ossec/bin/wazuh-keystore -f indexer -k password -v <INDEXER_PASSWORD>

Then, save the configuration and restart the manager/cluster using the command: systemctl restart wazuh-manager.

Refer to:
https://documentation.wazuh.com/current/user-manual/capabilities/vulnerability-detection/configuring-scans.html
https://documentation.wazuh.com/current/upgrade-guide/troubleshooting.html

[Alan Baltic]

Hi Javier,

thanks for having time to review.

From previous posts you can see that I deleted vd and vd_updater folders to see if recreation would take effect in resolving this. Also yesterday I deleted again vd, vd_updater and indexer folder. All these folders are in /var/ossec/queue alongside with vulnerability index. After that action I was only able to see empty vulnerability index ~200 bytes with only one Wazuh indexer host configured in indexer block in ossec.conf.

After making suggested changes in Wazuh master and Wazuh worker it seems that I can see inventory only for one agent and events are empty.



ossec.log
2024/07/02 11:41:31 indexer-connector[2220215] indexerConnector.cpp:319 at initialize(): INFO: IndexerConnector initialized successfully for index: wazuh-states-vulnerabilities-mycluster.
---
2024/07/02 12:14:36 indexer-connector[2220215] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1086' with the indexer.
2024/07/02 12:14:36 indexer-connector[2220215] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
2024/07/02 12:14:36 wazuh-modulesd:vulnerability-scanner[2220215] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/07/02 12:14:36 indexer-connector[2220215] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1086' sync omitted due to abuse control.
2024/07/02 12:14:46 wazuh-modulesd:vulnerability-scanner[2220215] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/07/02 12:14:46 indexer-connector[2220215] indexerConnector.cpp:437 at operator()(): DEBUG: Syncing agent '1103' with the indexer.
2024/07/02 12:14:46 indexer-connector[2220215] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '1103' with the indexer.
2024/07/02 12:14:46 indexer-connector[2220215] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server
2024/07/02 12:14:47 wazuh-modulesd:vulnerability-scanner[2220215] scanOrchestrator.hpp:299 at run(): DEBUG: Event type: 11 processed
2024/07/02 12:14:47 indexer-connector[2220215] indexerConnector.cpp:129 at abuseControl(): DEBUG: Agent '1103' sync omitted due to abuse control.

Differences I made:

1. Changes in ossec.conf  (indexer block) are on both master and worker:
   
   OLD:
   <indexer>
       <enabled>yes</enabled>
       <hosts>
         <host>https://10.10.10.11:9200</host>
       </hosts>
       <ssl>
         <certificate_authorities>
           <ca>/etc/filebeat/certs/intermed-ca.pem</ca>
           <ca>/etc/filebeat/certs/root-ca.pem</ca>
         </certificate_authorities>
         <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
         <key>/etc/filebeat/certs/filebeat-key.pem</key>
       </ssl>
     </indexer>
   
   NEW:
   <indexer>
       <enabled>yes</enabled>
       <hosts>
         <host>https://10.10.10.11:9200</host>
         <host>https://10.10.10.12:9200</host>
         <host>https://10.10.10.13:9200</host>
       </hosts>
       <ssl>
         <certificate_authorities>
           <ca>/etc/filebeat/certs/intermed-ca.pem</ca>
           <ca>/etc/filebeat/certs/root-ca.pem</ca>
         </certificate_authorities>
         <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
         <key>/etc/filebeat/certs/filebeat-key.pem</key>
       </ssl>
     </indexer>
   
   
2. Check filebeat cert files on both master and worker:
    Wazuh - Filebeat configuration file
   output.elasticsearch:
     hosts: ["10.10.10.11 :9200","10.10.10.12:9200","10.10.10.13 :9200"]
   #  hosts: ["10.10.10.11:9200"] #Tried with on or all hosts in cluster
     protocol: https
     username: ${username}
     password: ${password}
     ssl.certificate_authorities: ["/etc/filebeat/certs/intermed-ca.pem","/etc/filebeat/certs/root-ca.pem"]
     ssl.certificate: "/etc/filebeat/certs/filebeat.pem"
     ssl.key: "/etc/filebeat/certs/filebeat-key.pem"
   setup.template.json.enabled: true
   setup.template.json.path: '/etc/filebeat/wazuh-template.json'
   setup.template.json.name: 'wazuh'
   setup.ilm.overwrite: true
   setup.ilm.enabled: false
   
   
3. Wazuh-keystore (I did it again to be sure):
   
   /var/ossec/bin/wazuh-keystore -f indexer -k username -v {same_user_for_login_to_wazuh_or_curl_wazuh_indexer} (default: admin)
   /var/ossec/bin/wazuh-keystore -f indexer -k password -v {same_password_for_login_to_wazuh_or_curl_wazuh_indexer} (default: admin)
   
   
4.  Chuck cluster health (did curl on all three nodes to be sure every node have expected response):
   
   curl -u admin:admin --cacert /etc/filebeat/certs/intermed-ca.pem --cert /etc/filebeat/certs/filebeat.pem --key /etc/filebeat/certs/filebeat-key.pem -X GET "https://10.10.10.11:9200/_cluster/health?pretty"
   {
     "cluster_name" : "MyCluster",
     "status" : "green",
     "timed_out" : false,
     "number_of_nodes" : 3,
     "number_of_data_nodes" : 3,
     "discovered_master" : true,
     "discovered_cluster_manager" : true,
     "active_primary_shards" : 236,
     "active_shards" : 600,
     "relocating_shards" : 0,
     "initializing_shards" : 0,
     "unassigned_shards" : 0,
     "delayed_unassigned_shards" : 0,
     "number_of_pending_tasks" : 0,
     "number_of_in_flight_fetch" : 0,
     "task_max_waiting_in_queue_millis" : 0,
     "active_shards_percent_as_number" : 100.0
   }
   
   

One last thing regarding certificates:

Every certificate is signed by intermed-ca in my case. (intermed-ca is signed by root-ca)

Ceritifcates for every component have CN hostname and SAN as IP address of the host where this component should be.

So filebeat certs (/etc/filebeat/certs/filebeat.yml) in:
wazuh master is:
CN: master01
SAN: 10.10.10.11

wazuh worker is:
CN: worker01
SAN: 10.10.10.14

Wazuh indexer cert for indexer01:
CN: indexer01
SAN: 10.10.10.11 (on same server as wazuh master)

Wazuh indexer cert for indexer02:
CN: indexer01
SAN: 10.10.10.12 

and so on..

I apologize for longer message but I am trying to specify as much details as possible for someone who will see these afterwards

Thank you

[Alan Baltic]

UPDATE:

I can still see inventory packages only for one agent in vulnerability detector, but relatively good news is that I am able to see Events for 3 agents for now (31 hits).

Inventory shows only for this one agent 7138 hits  which is quite a lot (OS updated), but in Events I am able to see only 9,10 vulnerabilities per agent for those 3 agents.

Hopefully it takes time to scan all of them. Failed to sync messages are still present though.

Thank you

[Javier Sanchez Gil]

Hi Alan Baltic,

Thank you for providing all the information!

Could you summarize where we are right now so that we can continue investigating if necessary?

I look forward to your response.

[Alan Baltic]

Hi Javier,

So the problem is half solved after adding all nodes from Wazuh indexer cluster to indexer block in ossec.conf. 

In last 24hours I am able to see the following:

Dashboard/Inventory which was updated but not for every agent (I could not find a lot of them using filter - all agent are 4.8.0): 


Today there are no events: (Only from yesterday)

Sync error still present:

Is there any chance that wazuh-indexer is getting to many requests? 
HW for indexers are:
4 CPU
16GB RAM (but 4gb set jvm.options )

Thank you

[Alan Baltic]

UPDATE for today: 
Vulnerability detector is still not updating nor I can see new events.

[Daniele Mutti]

Hi guys,

same situation here:

[Paweł Horosz]

I have the same problem as the predecessors. The situation is not changed by the number of agents, because it does not matter whether online there are 200 or 60 are type errors:
 
2024/07/08 10:24:27 indexer-connector[183883] indexerConnector.cpp:446 at operator()(): WARNING: Failed to sync agent '163' with the indexer.
2024/07/08 10:24:27 indexer-connector[183883] indexerConnector.cpp:447 at operator()(): DEBUG: Error: No available server

occur all the time. What I noticed interestingly is that after each restart of wazuh-manager randomly for some of the agents the information completes, and then disappears again.

[Javier Sanchez Gil]

Hi Alan Baltic,

Sorry for the delay! I have read the latest updates you mentioned.

Regarding the recommended minimum hardware for the indexer, it seems to meet the minimum requirements to function well: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/index.html#recommended-operating-systems

Also, let me know if it meets the general hardware requirements mentioned here: https://documentation.wazuh.com/current/quickstart.html

Are you still receiving the log: Error: No available server           ?

If so, let's check the health status of the indexer cluster:

# curl -X GET "http://127.0.0.1:9200/_cat/health?v"

Replace the address with the address of your indexer host.

[Alan Baltic]

Hi Javier,

the problem is still present.
Hardware is OK. This is more of a testing environment so there is no many events present.
As per my previous posts the cluster is in green state, and also the green state is now. Configurations that I sent are the same that I am using with different IP's.

curl -u admin:admin --cacert /etc/filebeat/certs/intermed-ca.pem --cert /etc/filebeat/certs/filebeat.pem --key /etc/filebeat/certs/filebeat-key.pem -X GET "https://10.10.10.12:9200/_cluster/health?pretty"

{
  "cluster_name" : "MyCluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,

  "active_primary_shards" : 259,
  "active_shards" : 664,

  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

I am out of ideas. 
Vulnerability detection is still the same for a few days:

Thank you

[Javier Sanchez Gil]

Hi Alan Baltic,

Everything seems to be in order with the indexer cluster.

Let's proceed with troubleshooting in the environment to see if we can uncover anything else related to the issue.

I'll provide you with troubleshooting steps from the Upgrade guide in the Wazuh documentation:

https://documentation.wazuh.com/current/upgrade-guide/troubleshooting.html#troubleshooting

Please let me know if you notice anything unusual or encounter any errors!

[Alan Baltic]

Hi Javier,

I already looked troubleshooting documentation. 
Unfortunately it didn't help because it's not related to the current problem.
The interesting part is that I have some vulnerabilities and alerts as you can see from my previous posts, but not for all agents. 

So the question is why the syncing suddenly stopped. I red from other tickets that the only reasons for this "No server" error are wrong credentials in Wazuh keystore, cluster health != green or the issues with certificates.
From my perspective it all seems good based on checks like using CURL with the same certs which are configured in indexer, monitoring vulnerability index creation.

Just to confirm. Wazuh vulnerability module is communicating only and directly to Wazuh indexer cluster (Wazuh manager -> Wazuh indexer TCP 9200) and getting the vulnerabilities database from cti.wazuh.com:443 or offline in my case.

[Javier Sanchez Gil]

Hi Alan Baltic,

Yes, the Wazuh vulnerability module communicates directly with the Wazuh indexer cluster (from core to indexer directly).

We will continue investigating to see if we can find the cause of this!

First, let's rule out any issues with the environment by verifying that everything is functioning perfectly. We'll check the manager status to be sure:

systemctl status wazuh-manager

I was also considering whether HaProxy could be involved, even though you tested with and without it. There were reports from others experiencing issues with the update due to HaProxy.

Reason for the Problem:

- In the configuration file /etc/systemd/system/wazuh-manager.service.d/override.conf, the http_proxy and https_proxy environment variables were set.

- These variables cause the wazuh-manager service to send all its requests through a proxy.

- Due to this configuration, wazuh-manager could not contact the wazuh-indexer on the internal private network via HTTPS requests because the proxy did not authorize these requests.

Solution:

- An environment variable called no_proxy was added to the configuration.

- This no_proxy variable specifies that certain addresses, such as the internal FQDN (fully qualified domain name) of the wazuh-indexer, should be excluded from using the proxy.

- By doing this, the wazuh-manager was able to contact the wazuh-indexer directly without going through the proxy.

[Alan Baltic]

Hi Javier,

here is wazuh-manager status screenshot.

Regarding the configuration file /etc/systemd/system/wazuh-manager.service.d/override.conf I do not have it so can you please write what configuration should be in it for HAProxy to work properly?
Also I am using HAProxy only for loadbalancing agents and Wazuh nodes are communicating directly in Wazuh cluster. In order to make debugging easier I excluded HAProxy from my setup so agents are now directly connected only to Wazuh master.
One more thing that I tried is that I created vulnerability index with 3 primary shards (default is 1) hoping that somehow this will help. 
Also worth mentioning is that I have archives enabled. 
Now the interesting thing is that somehow vulnerability inventory is updated (also not for all agents), but I still can not see any events in VD.




The problem is I am not sure if its the reason HAProxy, disabling/enabling archives, that all agents are now only on master or adding primaries to vulnerability index (which I am pretty sure it did not help)
Nevertheless, with my current setup I still have "No server available" error.

Is it somehow possible to "speed up" the vulnerability updates so that I can test if functionality works after making changes in environment?

Thank you very much for your time

[Javier Sanchez Gil]

Hi Alan Baltic,

Regarding the file .../override.conf, it was a solution offered where a user had the http_proxy and https_proxy environment variables, which were causing similar errors. If you don't have them, we can also rule out that part

Regarding what you mentioned, did I understand correctly that all your agents are pointing to the master server?

About the "speed up" comment, are you referring to the feed-update-interval? https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/vuln-detector.html#feed-update-interval

On the other hand, is there any new information on this to continue looking for more solutions?

[Alan Baltic]

Hi Javier,

Yes all agents are now pointing only to master in order to make debugging easier. 

Vulnerability detector is same as per last post and I am still investigating.

Next thing I will try is leave only one agent connected to see if vulnerabilities index will start to update

[Javier Sanchez Gil]

Hi Alan Baltic,

Agreed! Let's try leaving just one agent and see if it starts updating.

Please inform me of any new information that may arise so we can investigate and find the solution!

[Alan Baltic]

Hi Javier,

After I had one agent connected for some time the error "No server available" was still present. 
Cluster was restarted a few times and I saw some updates in vulnerability. I really can not understand where the problem is. 
Today, I generated certificates for Wazuh indexers again but I did not manage to add another SAN name using wazuh certificate tools so I extracted commands
For every Indexer node  a CN, hostname and IP changed:
ssl.conf:
[ req ]
prompt = no
default_bits = 2048
default_md = sha256
distinguished_name = req_distinguished_name
x509_extensions = v3_req

[req_distinguished_name]
C = XX
L = City
O = Organization
OU = Organization Unit
CN = name-of-indexer

[ v3_req ]
authorityKeyIdentifier=keyid,issuer
basicConstraints = CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
IP.1 = 10.10.10.11
DNS.1 = hostname-of-indexer

openssl req -new -nodes -newkey rsa:2048 -keyout wi01-key.pem -out wi.csr -config ssl.conf
openssl x509 -req -in wi01.csr -CA intermed-ca.pem -CAkey intermed-key.pem -CAcreateserial -out wi01.pem -extfile ssl.conf -extensions v3_req -days 3650

As you can see certificates are signed by my internal intermediate ca.
Can you confirm that indexer block is properly configured regarding CA's?

  <indexer>
    <enabled>yes</enabled>
    <hosts>
      <host>https://10.10.10.11:9200</host>
      <host>https://10.10.10.12:9200</host>
      <host>https://10.10.10.13:9200</host>
    </hosts>
    <ssl>
      <certificate_authorities>
        <ca>/etc/filebeat/certs/intermed-ca.pem</ca>
        <ca>/etc/filebeat/certs/root-ca.pem</ca>
      </certificate_authorities>
      <certificate>/etc/filebeat/certs/filebeat.pem</certificate>
      <key>/etc/filebeat/certs/filebeat-key.pem</key>
    </ssl>
  </indexer>

How can I confirm if the Wazuh manager (Indexer module) is verifying the chain of trust, besides using those same certificates in CURL towards wazuh indexers? Is it enough if the Indexer module created the vulnerability index?
I just want to mention again that I have  Wazuh manager, Wazuh indexer, Wazuh Dashboard and filebeat on one machine, and two other Wazuh indexers are on separated machines.
I point that out because I saw some tickets where people need to put 127.0.0.1 in the host block if they have all-in-one installation. 

I think that in my case that is not possible because of a cluster.

Also, I assume that we need all hosts in the indexer block because there is only one primary shard by default for the vulnerability index. If the node where this shard is located goes down, the cluster will be yellow, and the shard will be unassigned. 

I would say that adding at least one replica is recommended if you have an indexer cluster.  

Now the main problem is that I am unable to upgrade my Wazuh production cluster until this issue is resolved.

Please advise what should I try next
Thank you

[Javier Sanchez Gil]

Hi Alan Blactic,

I have been talking directly with the team in charge of the indexer to discuss your case!

Could you provide us with the output of journalctl -u wazuh-indexer for a few minutes before and after the timestamp of the error in ossec.log to see if there was a problem with the indexing?

I look forward to your response to continue with this

[Alan Baltic]

Hi Javier,

before I saw your message I did some changes.
1. I deleted vulnerability index. Wazuh manager created new index successfully. Also changed <feed-update-interval>6h</feed-update-interval>. From my understanding this interval is just for updating index with new vulnerabilities online or offline

Waited for a few hours - index was empty.

2. Afterwards I changed the name from Wazuh indexer cluster (not wazuh server cluster). The reason was that the name contained spaces ("this is mycluster"). Now is "this-is-mycluster". This was one of a desperate tries..

Waited for a few hours - index was empty. 

3. I updated one machine and now I can see vulnerability data alongside with events for this machine only.

4. I am seeing vulnerabilities for other machines - lets see If all machines will be synced. Also can I can see "sync omitted due to abuse control" but from other conversations that is normal because of cooldown

I really don't know what to think.

Is it possible that Wazuh indexer cluster name was the problem?

[Javier Sanchez Gil]

Hi Alan Baltic,

I'm glad to hear that you are starting to see vulnerability data and events.

Yes, leaving spaces in the name could have created discrepancies or problems in the system configuration and communication.

Anyway, let's hope all the machines update properly.

Regarding the abuse control message, it is just a protection to prevent agents from overloading the indexer.

When you have more information, please let me know!

[Alan Baltic]

Hi Javier,

I can see changes in vulnerability dashboard and inventory which is good. What confuses me is how is dashboard is updated exactly. 

Yesterday I upgraded a few machines which were on RHEL 9.2 and now they are on RHEL 9.4. I can still see both versions and many many vulnerabilities. Here are screenshots.



Now the problem is that I can't show such status to auditor. 

Another problem is I didn't receive any alerts (events) except for one machine. Regarding the vulnerability alerts/events they are generating as any other alerts consuming the EPS? 
if so, then I can understand which unfortunately leads to another problem. Due to our policy it is a must to record  all root actions. During the upgrades of machines there are overwhelming amount of events many are discarded due to

"full bucket" logic. If everything is as I stated than I can't turn off Wazuh agents during upgrades because I would not receive vulnerability alerts/events about package statuses etc..?

I really appreciate your time and effort in helping me,

[Javier Sanchez Gil]

Hi Alan Baltic,

Regarding what you mentioned about not receiving any alerts except from one machine, is this related to the update or in general? I'm a bit confused about whether you are currently receiving events from the other agents or not.

About the EPS limitation on the agent, there is a parameter called max_eps that limits the maximum rate of events per second.

You can find it within the agent's configuration. In the example configuration for the vulnerability detector, you can see it:

https://documentation.wazuh.com/current/user-manual/capabilities/vulnerability-detection/configuring-scans.html

Here is more information about this parameter:

https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-syscollector.html#max-eps

[Alan Baltic]

Hi Javier,

I am receiving globally all alerts.  But regarding the Vulnerability detector alerts called Events (third tab) where you can check if vulnerability is resolved, there I can see only for one agent. 

Regarding the EPS limit, I have set it up but during the system upgrades there are to much events because we are monitoring all root actions. I would suggest for some sort of silence option during maintenance work.
I think it would be great feature.

[Alan Baltic]

HI Javier,

just one more thing regarding the Events for Vulnerability. I am not sure how this should work (I red in Wazuh documentation). 
I just tested it like this.
1. Example

• I saw VIM package have vulnerabilities.
• Delete vim package (dnf remove vim)
• Agent restart so that Inventory is scanned (do not want to wait for 1h)
• Can see that Dashboard is updated (I do not see VIM in my inventory).
• There is no events regarding these.

2. Example

1. Updating machine
2. I assume due to to much logs because of a audit rule where we need to monitor ALL root actions there are no events
3. After upgrade in Dashboard I can still see both version of RHEL (9.3 and 9.4). I need to see only current version. Also after upgrade I have thousands and thousands vulnerabilities
4. For example comparing to Nessus report where I have 0 critical, in Wazuh I have 5. So unfortunately It's not very useful at the moment. I hope it will get better sometime.

Thank you

[Javier Sanchez Gil]

Hi Alan Baltic,

Could you let me know if you did anything differently for that agent? Regarding certificates, OS version, configuration, permissions, etc., that might give us a clue as to why it is the only one showing events on the dashboard.

On the other hand, for Example 1, yes, it follows the Wazuh documentation:

1. Install a Vulnerable Version of Vim
2. Wait for Syscollector to Perform a New Scan
3. Remove the Vim Package
4. Wait for Syscollector to Perform a New Scan
5. This should display the alerts on the Wazuh dashboard.

Regarding the second issue, I will continue investigating why both OS versions are being shown!

[Paweł Horosz]

I have a similar problem as Alan, while I observe very strange behavior of vulnerability counters. As I restart the wazuh-manager unit several times the counters increase, but after some time with each dashboard refresh they drop practically to 0 and so on and so forth.

[Alan Baltic]

Hi Javier,

I am on vacation until 6.8. so I wont be active. 

Regarding the configuration of the agent there is no difference. 

You wrote that alerts are generated when Syscolector perform a new scan?

I will need to confirm that because it seems that alerts are generating at the same moment when

I was upgrading the machine but because of alert flood (root command alerts) I did not recieve vulnerability 

alerts but I can see updated inventory.

When I get back I will also check what Pawel wrote to see if I have same behavior regarding counters.

Thanks,

Alan

[Alan Baltic]

Hi Javier,

hope you are doing well.

I can confirm that events are not shown in vulnerability detector while queue is full while performing update. I tested on few machines.
Machine1 where I can see events is without audit rule where I record every root command.
Machine2 where I have audit rule for recording every root command is overwhelmed by events and it fills the buffer. 
Syscheck is regularly executing but not triggering alerts for second machine, the one where I have buffer overflow while doing updates.

Machine1 

Machine2

ossec.log
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:27 wazuh-logcollector: ERROR: Discarding audit message because cache is full.
2024/08/14 11:52:48 wazuh-syscheckd: INFO: (6009): File integrity monitoring scan ended.
2024/08/14 11:52:48 wazuh-syscheckd: INFO: FIM sync module started.
2024/08/14 11:52:48 wazuh-syscheckd: INFO: (6012): Real-time file integrity monitoring started.
2024/08/14 11:52:53 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2024/08/14 11:53:03 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_rhel9_linux.yml'
2024/08/14 11:53:03 sca: INFO: Security Configuration Assessment scan finished. Duration: 39 seconds.
2024/08/14 11:53:44 rootcheck: INFO: Ending rootcheck scan.
2024/08/14 12:52:54 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2024/08/14 12:53:04 wazuh-modulesd:syscollector: INFO: Evaluation finished.

I have ERROR, but after syscollector scan finished still do not have evets about update packages.


One more thing about counter and showing both versions of OS after update is due to time range which as I far as I can see there is none. So every time it shows whole index or if there is hardcoded time stamp which then includes both records with old and new OS version.

Thank you

[Alan Baltic]

Hi,

Did you have time to look at this?

Appriciate your help,

Alan