WARNING: Agent buffer is full: Events may be lost.

[C. L. Martinez]

Hi all,

 As subject says:

2018/07/02 13:45:27 ossec-agentd: WARNING: Agent buffer at 90 %.
2018/07/02 13:45:29 ossec-agentd: WARNING: Agent buffer is full: Events may be lost.
2018/07/02 13:45:44 ossec-agentd: WARNING: Agent buffer is flooded: Producing too many events.

 According to doc, EPS is limited to 1000 and queue_size to 100000. Is it not possible to increment these values?

Thanks.

[Chema Martinez]

Hi C. L. Martinez,

If these messages have appeared in your log file, one or more of your agent components are flooding the agent queue and it is not a normal behavior. If you want to find out what is causing that amount of events you could check what module (syscheck, OpenSCAP, etc...) have a big load of work or show us the configuration files if you prefer.

Those parameters for the agent queue configuration are limited due to OS limitations, so it is not possible to make them bigger. You also have the possibility to disable the queue with the "<disabled>" option. However, it exists the possibility of flooding the manager and lost events incoming from every agent.

I hope it helps.

Best regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CAEjQA5JjSq6dehP9QSpXTDKAfS3WZ5AytAj-tCdvHbKL_3duew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[C. L. Martinez]

Thanks Chema. These message errors appears when one agent read logs from a file, previously downloaded from another server (every 15 min). But I don't understand this sentence " Those parameters for the agent queue configuration are limited due to OS limitations ". What is the limitation?. Cannot be by a limitation in the linux kernel. I have serveral RHEL servers sending more than 5k EPS to a central SIEM infrastructure..

Exists any option to configure in agent side to read log files without exceeding that limit?

[Chema Martinez]

Hi C. L. Martinez,

I should have been clearer, sorry about that. I was referring about a limitation on Windows, the function Sleep() on that OS doesn't allow to sleep a thread less than one millisecond. That is why the maximum EPS is 1000. To be consistent with Windows agents, it makes sense for Linux agents to set the same limitation, it is also a logical EPS to avoid flooding the manager with many events.

On the other hand, it exists one option in the internal_options.conf file to limit the number of lines read by Logcollector from the same file in a read cycle. This option is called logcollector.max_lines and by default is set to 1000 lines. Here you can find more information about how to configure this option:

https://documentation.wazuh.com/current/user-manual/reference/internal-options.html#logcollector

Best regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

[C. L. Martinez]

Perfect, many thanks Chema. Only the last question: what is the EPS limit on the server side? is it configurable?

[Chema Martinez]

You are welcome Carlos,

On the manager side, there is no EPS limit. It ingests incoming data as fast as possible.

However, since v3.2.2 the manager includes an input buffer to avoid congestion issues whose size is configurable in the <remote> tag of the configuration file. This buffer discards new events when it is full, triggering warning messages in the log file.

Regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.