File Integrity Monitoring - Time difference

[WENWEN H]

Hello everyone:
My Wazuh is currently experiencing some issues, and there is a delay in the log generation. For example: I found that the installation date of the screen on our server was August 29th, and the log read "Aug 29 17:47:47 Installed: screen-4.1.0-0.27.20120314git3c2946.amzn2.0.2.x86_64"; The alarm received by the Wazuh platform was on August 30th. Below, I will post the detailed screenshot 1.
I'm not sure if this is related to the performance of my Wazuh agent server. The screenshot 2 shows that the Wazuh-agent-linux3 will issue an alert every certain period of time, stating "rule.id: 204", "rule.description: Agent event queue is flooded. Check the agent configuration." Will this affect the delay in sending my logs?

[Stuti Gupta]

Yes, the "Agent event queue is flooded. Check the agent configuration." warning is related to delays in alerts and can explain why there's a lag between when an event occurs and when the alert appears in Wazuh.
When this buffer (or queue) overflows—because too many events are generated in a short period, or because the agent/server is processing events too slowly—the "queue flooded" alert is triggered.
The Wazuh agent in your case is experiencing buffer saturation, meaning it is collecting more events than it can send to the manager. The buffer is almost full (4999 out of 5000), and only about a third of the collected messages have been sent. This suggests that the agent is not able to forward events fast enough, even though the event rate is limited to 500 events per second by default. This could be caused by a large burst of events that floods the network of the manager. https://documentation.wazuh.com/current/user-manual/agent/agent-management/antiflooding.html#flooding-status-red-area
To solve this issue, you can increase the queue size and reduce the event per second on the agent's configuration file.
Navigate to the Wazuh configuration directory /var/ossec/etc/ossec.conf and edit the file to reflect the changes:

<client_buffer>
 <disabled>no</disabled>
 <queue_size>50000</queue_size>
 <events_per_second>800</events_per_second>
</client_buffer>

Save the changes.
Then restart the Wazuh components

However, increasing the queue size is not recommended as it can impact the agent's footprint and the environment's network.
Therefore, it is important to identify the root cause of the issue by understanding what kind of logs the agents are ingesting, their frequency, and when the issue began to occur.
Wazuh has a buffer mechanism on the agents to prevent a large burst of events from negatively impacting the network of the manager. More information on the anti-flooding mechanism can be found in the Wazuh documentation - https://documentation.wazuh.com/current/user-manual/agents/antiflooding.html
You need to review the event logs is crucial for detecting anomalies or patterns in the generated events. This information can help reduce false positives and provide valuable insights into the underlying cause of the flooding.
You could share what your agent’s operating system is and what events are triggering the logs for it to be flooded.
https://documentation.wazuh.com/current/user-manual/agent/agent-management/antiflooding.html#flooding-status-red-area

For Example:
A frequent culprit for this issue on Windows machines is audit failure events related to Chrome. These events (Event ID: 4673) are often harmless and can be safely filtered.
https://support.google.com/chrome/a/thread/15440066/excessive-windows-10-audit-failures-from-chrome-exe?hl=en
Check the 'C:\Program Files (x86)\ossec-agent\ossec.conf and add the EventID != 4673 with the configuration of <location>Security</location> inside <localfile> existing configuration.

<localfile>
  <location>Security</location>
  <log_format>eventchannel</log_format>
    <query>Event/System[EventID != 5145 and EventID != 5156 and EventID != 5447 and EventID != 4656 and EventID != 4658 and EventID != 4663 and EventID != 4660 and EventID != 4670 and EventID != 4690 and EventID != 4703 and EventID != 4907 and EventID != 5152 and EventID != 5157 and EventID != 4673 ]
  </query>
<localfile>

Save the config file.
Then, restart the agent and check if the alert has stopped triggering.
Restart-Service -Name wazuh

Based on your alert, you can change your event ID as mentioned above.
You can find the detailed explanation of how the agent's events are buffered in the following documentation: https://documentation.wazuh.com/current/user-manual/agents/antiflooding.html
Also, ensure the agents are equal to the wazuh-manager or the wazuh-manager version is higher than the wazuh-agent.

Let me know if that works for you!

[WENWEN H]

Hi, Stuti.
I have found a solution. There are two events, namely rule.id = 80454 and 80456. These two events account for half of my total events. My lowest alert level is 3. If I set their alert levels to 1, there won't be any alerts. So, can this solve the buffer (or queue) overflow problem?
If the documentation is correct, they are mainly responsible for monitoring the alerts of resource additions and deletions on AWS.

[Stuti Gupta]

Hi 

If you want to stop those noisy ResourceDiscovered / ResourceDeleted events (rule IDs 80454 & 80456) from ever hitting the manager queue, you need to discard them earlier, at the ingestion stage. That means using <discard_regex> in the AWS module config (on the manager if you pull logs directly, or on the agent if it’s the collector).

Please refer to https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/wodle-s3.html#discard-regex