How to prevent "queue flooding" and scale log ingestion to 500,000+ events?
Alex
I am experiencing a persistent "queue flooding" issue on one of my Wazuh agents. To resolve the issue, I modified the <client_buffer> configuration in the agent's ossec.conf file, significantly increasing the queue size and EPS limits:
<client_buffer>
<disabled>no</disabled>
<queue_size>100000</queue_size>
<events_per_second>1000</events_per_second>
</client_buffer>
Despite applying this configuration, I am still getting the queue flooding logs.
My goal here is not to reduce the noise or filter these logs out. I actually want to ingest this massive volume of logs successfully. For example, I need the architecture to be able to capture and process around 500,000 logs without causing any flooding on the agent side or dropping any events.
How can I achieve this? What are the necessary advanced tuning steps on both the agent side and the manager side (such as wazuh-remoted worker pools, internal queues, etc.) to handle this level of log volume seamlessly without triggering flood warnings?
Thanks in advance,
Alex
Md. Nazmur Sakib
Hi Alex,
You are getting this error due to the anti-flooding mechanism placed on the agent and your agent is generaating more events than the limit you have configured.
You can disable the event queue if you like from the agent’s ossec.conf.
<client_buffer>
<disabled>yes</disabled>
<queue_size>100000</queue_size>
<events_per_second>1000</events_per_second>
</client_buffer>
Ref: client_buffer
Keep in mind that disabling the client buffer will allow sending bursts of alerts, and the agent loses its shock absorber, so the manager and network must be able to sustain the rate of logs per second.
Configure the Wazuh agent communication queue by editing the <queue_size> in the <remote> block of the /var/ossec/etc/ossec.conf file on the Wazuh server
And adjust the worker_pool size in the /var/ossec/etc/internal_options.conf file.
Agent queuing mechanisms
Keep an eye on the static files to monitor if there are any potential drops of events.
cat /var/ossec/var/run/wazuh-analysisd.state
cat /var/ossec/var/run/wazuh-remoted.state
Let me know if this works for you.
Alex
Hi,
Thanks for the suggestions! I applied the configurations you recommended:
remoted.worker_pool: 16
<queue_size>262144</queue_size>
<disabled>yes</disabled>
This method helped a little bit, but when I monitor the queues, I can still see that event drops are occurring.
It looks like wazuh-remoted is successfully handling the load now (the remoted/event queue is at 0%), but the bottleneck has clearly shifted to analysisd. Specifically, the syscollector queue is completely maxed out at 100%, which is causing a massive amount of dropped events.
Here are the statistics I am seeing right now:
Queue usage (all monitored queues, highest first): analysisd/syscollector: 100.00% (size 16384) analysisd/dbsync: 3.00% (size 16384) analysisd/syscheck: 0.00% (size 16384) analysisd/rootcheck: 0.00% (size 16384) analysisd/sca: 0.00% (size 16384) analysisd/hostinfo: 0.00% (size 16384) analysisd/winevt: 0.00% (size 16384) analysisd/upgrade: 0.00% (size 16384) analysisd/event: 0.00% (size 16384) analysisd/rule_matching: 0.00% (size 16384) analysisd/alerts: 0.00% (size 16384) analysisd/firewall: 0.00% (size 16384) analysisd/statistical: 0.00% (size 16384) analysisd/archives: 0.00% (size 16384) remoted/event: 0.00% (size 262144) Data-loss counters that grew since the previous check: analysisd:events_dropped: +117009 (total 117009) Current data-loss counters: analysisd:events_dropped: 117009 remoted:dequeued_after_close: 0 remoted:discarded_count: 0
How can I resolve this new bottleneck in analysisd? Is there a specific parameter in internal_options.conf to increase the syscollector queue size, or any other tuning you would recommend to prevent analysisd from dropping these events?
Thanks again,
Alex
Md. Nazmur Sakib
You can increase the analysis engine queue following this doc:
Wazuh analysis engine queue (queue_and)
Go to this file:
/var/ossec/etc/local_internal_options.conf
And update the specific queue sizes in the analysisd
Keep in mind that increasing this can overwarm your server. In this case, you can add more Wazuh manager nodes to balance the load.
Ref: Adding new Wazuh server nodes
Let me know if this solves your issue.