Agent Buffer is full message

[Mark Rafa]

Hi Team,

I am trying to collect logs from 2 different txt files on a windows machine where wazuh agent is installed. Database logs are written to these 2 files and I route these logs as syslog to my wazuh manager.  However since there is too much logs first I get agent buffer is full log. After I disabled client buffer, I get the log below:

2021/02/02 11:50:36 ossec-agent: WARNING: Target 'agent' message queue is full (1024). Log lines may be lost.

And I cant see any logs on Kibana. 

I can see the logs in archives.log when I first restart the agent, but then I can not able to see them also in archives.log. 

I have already create a decoder and rule, (rule.level:3) and tested for the logs. 

My agent buffer conf:

  <!-- Agent buffer options -->

  <client_buffer>

    <disabled>no</disabled>

    <queue_size>100000</queue_size>

    <events_per_second>1000</events_per_second>

  </client_buffer>

Are there any other ways to increase the buffer size or any other solutions?

King regards, 

[jeremias...@wazuh.com]

Hi Mark!
Thank you for using Wazuh!
As you may know, the client_buffer limits (link) target to avoid flooding the agent and/or the manager with too much information to process. Your configuration is using the maximum values of this config.
The events_per_second tag specifies the number of events that can be sent to the manager per second but isn't possible to increase because in windows agents there is an operating system limitation on the sleep used to wait for sending events to the manager (isn't possible to set it lower than 1ms)
The queue_size  tag sets the capacity of the agent buffer in a number of events, and the maximum accepted is 100000 events retained.
The disabled tag disables the leaky buffer on the agent at all. Disabling it can lead to overflowing the manager with events, and this seems to be your last problem when you observed only the events right after the restart.

You can try adding a line like logcollector.max_lines=5000 in local_internal_options.conf. The default value of this configuration is 10000.
This will limit the number of lines read in one read cycle. If these logs are being written irregularly, this can help to read them slower allowing the agent and manager to process the whole information.
I suggest you take a look at this link, where all these anti-flooding mechanisms are explained.

Let me know how this test goes and if you have further doubts, please don't hesitate to ask.
Best regards!

[jeremias...@wazuh.com]

Hi Mark,
I want to add something:
Since Wazuh 3.12, in Windows agents, events_per_second limit isn't working properly, and the agents send events as fast as possible (1000 EPS). This issue is being handled here.
Best regards

[Mark Rafa]

Hi,

Thanks for valuable informations.

I have tried to add   logcollector.max_lines=5000 but I had the agent log below:

2021/02/02 18:04:29 ossec-agent: WARNING: Target 'agent' message queue is full (1024). Log lines may be lost.

Still I can see the first two logs but then I can not able to see the upcoming ones. 

I have decreased the     logcollector.max_lines=1000  but still the same issue. 

Are there any other solution to solve this issue or this is something unsolveable due to windows operating system limitation?

King regards, 

2 Şubat 2021 Salı tarihinde saat 17:14:41 UTC+3 itibarıyla jeremias...@wazuh.com şunları yazdı:

[jeremias...@wazuh.com]

Hi Mark,
Do we have an estimation of how often these two txt files are written to know which should be the speed of Wazuh Agent to handle this information?
On the other hand, is this the only information collected by the agent? Maybe, this flooding is because of another source. Can you filter only the information of the problematic agent in archives.log and check if there are many events from other sources different than the txt files?
I will ask other coworkers to see if there is another possibility to test.
Best regards.

[jeremias...@wazuh.com]

Hi Mark,
When you disabled client_buffer the error that you observed is because logcollector reaches the limit number of events that can be dispatched to the agent module. This limit can be expanded by adding a line like the following in local_internal_options.conf.
logcollector.queue_size=10000

By doing so you will make this limit 10 times bigger than the default (1024, the value written in the log).
Keep in mind that any configuration change requires a restart on the agent.
Please let me know if this helps.
Best regards.

[Mark Rafa]

Hi again,

I have edit the local_internal_options.conf file on the agent, also disabled the client buffer. Now I dont see any errors on agent log file. Analyzing file: /path.txt is seen on the logs. After restart I can see again a couple of logs related to my issue but then no logs are shown.

A database logs are written to these two txt files continously. On that agent I dont monitor any other events other than logon/logoff. 

Regards, 

3 Şubat 2021 Çarşamba tarihinde saat 05:01:10 UTC+3 itibarıyla jeremias...@wazuh.com şunları yazdı:

[jeremias...@wazuh.com]

Hi Mark.
Maybe we are facing another problem then. You have enabled logall option and you don't find the expected entries in archives.log, right?
Can you share with me the configuration of the agent that you are using to read these txt files?
Also, can you restart the agent (when you do observe some entries regarding the txt logs) and share with me archives.log and ossec.log from the manager and ossec.log from the agent?
I will try to find another problem that can be causing this.
On the other hand: Do you have an estimation of how many lines per second (or bytes) are these txt files being written?
Regards

[Mark Rafa]

Hello,

Thanks for your help.

When I enable logall and restart agent, I can see related logs send from txt. files. First they are flooding but in 5-6 seconds it stops. Logon/logoff activities are continuing to come but txt. file logs are not coming. 

Below is my agent config to read the txt. files:

  <localfile>

    <location>C:\path1.txt</location>

    <log_format>syslog</log_format>

  </localfile>

  <localfile>

    <location>C:\path2.txt</location>

    <log_format>syslog</log_format>

  </localfile>

My ossec.log from agent (hkey registry logs are removed):

2021/02/04 12:56:49 ossec-agent: INFO: (1950): Analyzing file: 'C:\path1.txt'.

2021/02/04 12:56:49 ossec-agent: INFO: (1950): Analyzing file: 'C:\pat2h.txt'.

2021/02/04 12:56:49 ossec-agent: INFO: (6207): Ignore 'file' sregex '.log$|.htm$|.jpg$|.png$|.chm$|.pnf$|.evtx$'

2021/02/04 12:56:49 ossec-agent: INFO: (6207): Ignore 'registry' sregex '\Enum$'

2021/02/04 12:56:49 ossec-agent: INFO: Started (pid: 4892).

2021/02/04 12:56:49 ossec-agent: INFO: Started (pid: 4892).

2021/02/04 12:56:49 sca: INFO: Starting evaluation of policy: 'C:\Program Files (x86)\ossec-agent\ruleset\sca\sca_win_audit.yml'

2021/02/04 12:56:49 ossec-agent: INFO: (6000): Starting daemon...

2021/02/04 12:56:49 ossec-agent: INFO: (6010): File integrity monitoring scan frequency: 43200 seconds

2021/02/04 12:56:49 ossec-agent: INFO: (6008): File integrity monitoring scan started.

2021/02/04 12:56:49 rootcheck: INFO: Starting rootcheck scan.

2021/02/04 12:56:50 ossec-agent: INFO: (6009): File integrity monitoring scan ended.

2021/02/04 12:56:50 ossec-agent: INFO: (6012): Real-time file integrity monitoring started.

2021/02/04 12:56:50 wazuh-modulesd:syscollector: INFO: Starting evaluation.

2021/02/04 12:56:52 sca: INFO: Evaluation finished for policy 'C:\Program Files (x86)\ossec-agent\ruleset\sca\sca_win_audit.yml'

2021/02/04 12:56:52 sca: INFO: Security Configuration Assessment scan finished. Duration: 0 seconds.

2021/02/04 12:56:54 rootcheck: INFO: Ending rootcheck scan.

2021/02/04 12:56:55 wazuh-modulesd:syscollector: INFO: Evaluation finished.

ossec.log from manager:

tail -f ossec.log

2021/02/04 12:51:41 sca: INFO: Starting Security Configuration Assessment scan.

2021/02/04 12:51:41 wazuh-modulesd:download: INFO: Module started.

2021/02/04 12:51:41 wazuh-modulesd:ciscat: INFO: Module disabled. Exiting...

2021/02/04 12:51:41 wazuh-modulesd:database: INFO: Module started.

2021/02/04 12:51:41 sca: INFO: Skipping policy '/var/ossec/ruleset/sca/cis_rhel7_linux.yml': 'Check RHEL7 family platform'

2021/02/04 12:51:41 sca: INFO: Security Configuration Assessment scan finished. Duration: 0 seconds.

2021/02/04 12:51:42 wazuh-modulesd:syscollector: INFO: Starting evaluation.

2021/02/04 12:51:43 ossec-syscheckd: INFO: (6009): File integrity monitoring scan ended.

2021/02/04 12:51:49 wazuh-modulesd:syscollector: INFO: Evaluation finished.

2021/02/04 12:52:06 rootcheck: INFO: Ending rootcheck scan.

In the archives log when I restart the agent, I see a lot of logs flooding regarding my txt files. 

But only 2 or 3 logs are shown in Kibana. 

When I check inside of txt files around 20-30 events per second is written inside txt. files

Thanks for your support!

King regards, 

4 Şubat 2021 Perşembe tarihinde saat 02:48:46 UTC+3 itibarıyla jeremias...@wazuh.com şunları yazdı:

[jeremias...@wazuh.com]

Hi Mark,
I don't see anything problematic with the information that you provided.
But, if 20-30 events per second are written on the txt files, this amount of events doesn't explain the first error of the queue overflooded.
I think we must found what was really causing the first overflood. Keep in mind that the agent is capable to handle until 1000 EPS, then, the txt read shouldn't have generated the first issue. Maybe problems with the connection to send the events to the manager. Can you share with me the complete ossec.conf of the agent? Don't you identify any other source that can be flooding the agent with events?
Also, let me ask:
How are the hardware specifications of the computer where the agent is running?
Is the connection between agent and manager stable and fast enough?
Which Wazuh version is the agent and the manager?

I will ask other coworkers if anybody knows about a similar issue.

Regards.

[Mark Rafa]

Hi,

ossec.conf is attached. 

My environment: 

wazuh 3.13

wazuh manager: centos 8 4vcpu 8gb RAM

elasticsearc, kibana: centos 8 4 vcpu 16 gb RAM

windows agent: 8 vcpu 16 gb RAM

There is no connection problem between agent and manager.

Maybe there is something I am missing on agent ossec.conf.

King Regards, 

5 Şubat 2021 Cuma tarihinde saat 06:16:40 UTC+3 itibarıyla jeremias...@wazuh.com şunları yazdı:

[jeremias...@wazuh.com]

Hi Mark,
Your configuration doesn't have anything suspicious. It's the default Wazuh configuration just with these two local files to be analyzed.
If these files are written at 20-30 events per second, and the agent is getting flooded with just this information, I suspect now of the logs format...
- Are these files being written by a unique application.
- Do we know that these files are only being written in Syslog format?
- Can you make the following test?
  - Restart the agent.
  - Observe on archives.log which was the last event registered.
  - Find the log of this event in the log files (the analyzed txt)
  - Check the next logs after this log.
  - Repeat the test and check if the logs after the last obtained are similar.
  - Also, can you share with me the events of archives.log and the logs of these files after the restart? In order to find if there is a problematic format on these lines.

Best regards.