Agent event queue is full. Events may be lost

[Todor Dimitrov]

Hello, 

I am very new to Wazuh and recently got this message: "Agent event queue is full. Events may be lost". What can i do to increase the buffer size for the agent or maybe filter what events are being processed? If anyone can help me with advice that would be greatly appreciated. 

Kind regards,

Todor

[Jeremiah Kolawole]

Hello Todor,

If you get an alert saying "agent's event queue is full. Events may be lost," it means the agent's event queue has hit its limit and can't take in any more events. This usually happens when there's a huge spike in events, overwhelming the manager's network.

To fix it, you can increase the queue size in the affected agent’s configuration file. Here's how:

1. Go to the Wazuh configuration directory at `/var/ossec/etc/ossec.conf` and edit the file.
2. Add the new configuration below for a specific group that includes just the affected agents. Then, tweak the values gradually to avoid making the bucket too big.

For more details on how to manage this through centralized config, check here.

<client_buffer>
  <disabled>no</disabled>
  <queue_size>50000</queue_size>
 <events_per_second>800</events_per_second>
</client_buffer>

Save the changes.
Then restart the Wazuh manager for the changes to take effect

systemctl restart wazuh-manager

You should know that It's not a good idea to increase the queue size because it can put more strain on both the agents and the network. Instead, it's better to figure out what's causing the issue by looking into the type of logs the agents are processing, how often they come in, and when the problem started.

Wazuh agents have a buffer system in place to prevent a flood of events from overwhelming the manager's network. You can learn more about how this anti-flooding system works in the Wazuh documentation: https://documentation.wazuh.com/current/user-manual/agents/antiflooding.html.

The Wazuh manager also sends out alerts about the queue's flow levels, which are categorized into different types. More details can be found here: https://github.com/wazuh/wazuh/blob/master/ruleset/rules/0016-wazuh_rules.xml#L22.

To avoid the queue filling up again, you might want to filter out unnecessary events and make sure there aren’t any network connectivity problems between the agent and the manager. You can also keep an eye on the "noisiest" events using a different visualization in the Wazuh Dashboard. For more on managing Wazuh centrally, check this link: https://documentation.wazuh.com/current/user-manual/reference/centralized-configuration.html.

Regard,

[Todor Dimitrov]

Hello Jeremiah, 

Thanks for the useful information. I got some of the things i wanted to work but i have a question - Is there a way to just clear the client buffer without increasing the EPS and the buffer size? And if i don't have the client_buffer block in /var/ossec/etc/ossec.conf i can just add it right?

Regards, 

Todor

[Jeremiah Kolawole]

Hello Todor,

Since this alert occurs when the agent's queue is full, if you want to clear the buffer without increasing the EPS, you will need to finetune your different log sources for example your FIM directories, lower the manager-agent connection frequency, implement log rotation or simply restarting the agent service, you should know that restarting is a temporary fix and some alerts might be dropped.

For your second question, Yes. You'll add it to the conf file - see here

I hope this helps

[Todor Dimitrov]

Hello Jeremiah, 

So from what i understand you can't just clear the buffer contents with a command so it can start filling up again from scratch and I just have to configure the way the logs are collected and the amount that is collected and that is the only way and with time it should just empty itself because there are not that many logs collected any more, correct? So now that i have made some changes and stopped one of the events that generates most of my alerts it should go back to normal eventually, correct? Also one last thing - Is there a way to check what is the current capacity of the buffer manually with a command maybe or someplace where it is displayed? Thank you for your help again. Have a good day. 

Regards, 

[Jeremiah Kolawole]

Hello Todor,

I apologize for my delayed response.

To empty the buffer, you can simply restart the agent, a way to automate this might be to setup a cron job or a scheduled task with intervals you have noticed it takes to fill up and yes, if the event sources are reduced it'll also prevent the queue from filling up fast. 

To know the status of the buffer, you can check /var/ossec/var/run/wazuh-agentd.state (locally), or consult it from the API (from the manager): /agents/{agent_id}/stats/agent.

I hope this helps