Lag in ingestion of JSON log from windows endpoint via agent to Wazuh Manager Version 4.4.1

[taimur ahmed]

Hi Community!
                             The scenario is that I created an exe that tends to tell the score whether the file at windows endpoint is malicious or not. The application writes the logs in JSON Format into a file at windows endpoint. The logs are decoded by the wazuh default JSON decoder and are displayed in kibana under the archive index. I only applied the path and configuration at agent configuration:
                                                                   
<localfile>
    <log_format>json</log_format>
    <location><Path_to_Json_file></location>
</localfile>

Now the Problem is that the last entry appended at the log file is not pushed for instance if there are six entries in the JSON file then 5 entries are being pushed but the sixth one is not. And when seventh entry is added then the sixth (which is now second-last) is pushed to manager and can be viewed but not the last seventh one. Kindly guide me how to solve it such that the last entry can also be viewed at wazuh-dashboard. Thanks!  

[Damian Nicastro]

Hi taimur:

I hope you are fine.

Please, send me some logs sample in order to reproduce this issue.

Also consider that there is a hard limit of 64KiB for log lines that is 65279 bytes or characters.

If this is the case, you will see a line like this in the wazuh-agent (or wazuh-manager) LogColletor module:

#less /var/ossec/logs/ossec.log | grep -i logcollector

2023/05/04 12:29:10 wazuh-logcollector: ERROR: Large message size from file '/var/log/suri-test.json' (length = 65279): '{"timestamp":"2023-05-03T17:52:15.186949+0200","flow_id":2085930'...

If you don't have this message, we can look for errors in the same log. You might need to set the DEBUG mode to have more information:

# vi /var/ossec/etc/local_internal_options.conf

...

logcollector.debug=2

# systemctl restart wazuh-agent

After that, send me the output of:

#less /var/ossec/logs/ossec.log | grep -iE 'WARN|ERROR'

I hope this helps.

Thanks

[taimur ahmed]

The log sample is attached in the mail. Along with other configurations that I applied and the output of the ossec.log w.r.t logcollector at manager. The issue is persistent. Please guide me thanks! 

[Damian Nicastro]

Hi taimur:

I hope you are fine.

It seems that your agent buffer is overflowed. Since the Bufffer control is already configured with max values as mentioned in this document:
client_buffer - Local configuration (ossec.conf) · Wazuh documentation
You can try disabling it completely:
<client_buffer>
  <!-- Agent buffer options -->
  <disabled>yes</disabled>
  <queue_size>100000</queue_size>
  <events_per_second>1000</events_per_second>
</client_buffer> 

Do not forget to restart your wazuh-agent to load these changes:

# systemctl restart wazuh-agent

However, this configuration is not recommended because it can generate overload on your wazuh-agent machine (CPU and RAM), network saturation and  overload on your wazuh-manager machine. In these cases, it is strongly recommended to restrict the generation of log lines in the source. This can be done by logging only logs with certain severity or change the occurrences of them.

Additionally, with <client_buffer> disabled, you can modify the following values to increase the Events Per Second (EPS) setn to the wazuh-manager:
logcollector.input_threads
logcollector.queue_size

You can check the number of EPS and buffer status of the agent in the following file:

/var/ossec/var/run/wazuh-agentd.state

For more information about this, you can check the following document:

Internal configuration - Reference · Wazuh documentation