Logs Stop Reporting at 20:00

[Ryan Mohr]

Hi All,

Currently, I have network traffic logs being sent to a sensor (nids), more specifically they are suricata logs. On random days the logs will stop reporting, then the next day they will report as normal. Any idea on the reason? 

Troubleshooting:

- We've restarted the sensors, suricata and wazuh-agent.

- We've reconnected the agent to the wazuh master.

- Double checked the ossec.conf file.

- We've checked crontab and nothing looks unusual or would effect it. 

- We check and the ntp port.

Any suggestions would be greatly appreciated.  

Best,

Ryan

[Ryan Mohr]

Hi All,

I'm looking at the errors and I'm able to see this under ossec.log: 

"00:15:33 ossec-logcollector: ERROR: Large message size from file"

"14:41:04 ossec-logcollector: WARNING: Target 'agent' message queue is full (1024). Log lines may be lost."

Also the sensor time is UTC, when the linux box hits 00:00 the logs stop.

Best,

Ryan

[Ryan Mohr]

Hi All,

Any recommendations for troubleshooting would be appreciated!

Thank you!

[jeremias...@wazuh.com]

Hi Ryan.
Thank you for using Wazuh!

The error that you are having surely is because the agent buffer is getting flooded. This means that the module in charge of collecting logs is generating more events than what the agent can deliver to the manager.
You can increase the value of the events queue by modifying the client_buffer section of the agent ossec.conf. i.e.

<client_buffer>
<disabled>no</disabled>
<queue_size>100000</queue_size>
<events_per_second>500</events_per_second>
</client_buffer>

If the queue is still getting flooded (Logs like "Agent buffer is full" are still visible), you can even disable the client_buffer with

<client_buffer>
<disabled>yes</disabled>
<queue_size>100000</queue_size>
<events_per_second>500</events_per_second>
</client_buffer>

but, as this is a manager antiflooding mechanism this last option isn´t recommended, otherwise, an agent can generate as many events to flood the manager with them, which will lead to a similar error on the manager side.

On the other hand, the log ERROR "Large message size from file" must be related to a unique log line that is so big that can´t be dispatched to the manager. It should be a line bigger than 65,279 bytes.
The complete log should look something like "Large message size from file 'FILE' (length = LENGTH(...)"
- Can you share with me ossec.log to identify if these two errors are isolated?
- Also, can you check in the file of "Large message size from file 'FILE' (length = LENGTH(...)" if there is, in fact, a log line as big as it is reported?

I suggest you try the first approach of increasing the buffer queue size first, and then, try to reproduce the issue again.
If you have further doubts, please don´t hesitate to ask. And let me know how the tests go.

Best regards.

[jeremias...@wazuh.com]

Hi Ryan,
I misunderstood the source of your ERROR log. Sorry for the inconvenience:
The "message queue full" log proceeds from logcollector module, not from agentd as I first understood. Logcollector has its own leaky bucket and is being flooded.
To extend this buffer, you can open the file /var/ossec/etc/local_internal_options.conf and add the following line:

logcollector.queue_size=220000

You can read more about these configurations here.

This is the same approach of increasing client_buffer but, in logcollector module.

On the other hand: Can you share with me ossec.conf too, so I can take a look if there is any wrong configuration regarding this log file to be improved?

Best regards.

[Ryan Mohr]

Hi Jeremias,

Thank you for the suggestion. I've updated both the ossec.conf file and the local_internal_options.conf file. We will see within the week if that helps. Thanks again!

Anything I need to update on the manager to ensure it doesn't get flooded with logs?

Best,

[jeremias...@wazuh.com]

Hi Ryan,
The anti-flooding mechanism is on the agent side only. That is why it isn´t recommended to completely disable it. If you only incremented the buffer size of one of your agents, then you should be ok with the manager.
If want to read more about the anti-flooding mechanism I suggest you the following links:
https://documentation.wazuh.com/current/user-manual/capabilities/antiflooding.html
https://documentation.wazuh.com/current/learning-wazuh/survive-flood.html

Let me know how the tests goes.
Best regards!