Queue Statistics

[Russell Butturini]

Hi List,

Another separate issue I am working on troubleshooting is getting my queue sizes tuned for different servers.  For example, the queue is constantly filling up on my Exchange servers.  I've upped the queue size considerably, but it hasn't completely solved the problem.  Is there any way to tell how much overrun the queue is experiencing to aid in tweaking this setting? 

-Russell

[Chema Martinez]

Hi Russell,

I think the first step here is to know which is the source of the high load of events which are flooding your agent. I have some questions to find it out:

- Do the flooding logs appear in every agent restart? This can mean that the FIM engine is monitoring too many directories (it performs a pre-scan when starting).

- Are the Windows audit monitoring noisy logs coming from the Exchange server? Maybe the event channel is generating a big amount of events related to Exchange, and the logcollector is monitoring all of them.

You can enable the <logall>  option on the manager side to watch which events are coming from the flooded agents while the queue alerts appear. Raw events from the agent are stored at /var/ossec/logs/archives/archives.log when that setting is enabled.

On the other hand, you can tune the queue settings in the ossec.conf file (queue size and eps) and internal values (thresholds) at local_internal_options.conf. However, values which affect the performance of the queue are the ones located at ossec.conf.

If you find the queue is getting fully in exceptional times (when one of the components performs any scan), I suggest you increase the queue size to soften the burst of events. However, if the flooding is constant, I would increase the EPS of the queue as well as try to filter the noisy events.

I hope it helps.

Best regards,

Chema.

Chema Martinez | IT Engineer — Wazuh, Inc.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CA%2Bms%2BuJJdr6sC4fNptz6gCKO-e_EEAsswYUGmNPiknthLVjN-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Jason Youngquist]

Hi Russell.

I have the same problem with my Exchange server and a handful of other boxes (ie. domain controllers).  I've tweaked the "client_buffer" information (see below).  Did you ever solve this problem?  If so, what did you do?  

My next step is to turn on the "logall" option to see what type of events I'm getting from the flooded agents, but I'm monitoring ~300 agents, so turning on the "logall" option would eat up a lot of disk space.

<client_buffer>

  <!-- Agent buffer options -->

  <disabled>no</disabled>

  <queue_size>50000</queue_size>

  <events_per_second>1000</events_per_second>

</client_buffer>

[Chema Martinez]

Hi Jason,

I think in one of your flooded agents you can use the native Event Viewer to look for the source of the flooding. 

In the Overview tab of the Event Viewer you can find the type of events are being generated as you can see below:

Inside that list, you could look for the noisy event IDs (and evaluate whether they are important for you) to filter them in the agent configuration. For example, to filter by the event ID 1102 of the Security channel the configuration would like as follows:

<localfile>

  <location>Security</location>

  <log_format>eventchannel</log_format>

  <query>Event/System[EventID != 1102]

</localfile>

Queries located at the <query> field can be as complex as you think they fit your needs.

I hope it helps.

Best regards,

Chema Martinez IT Security Engineer  The Open Source Security Platform

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/60df2fed-4b8a-4f20-8e50-f90c9581efca%40googlegroups.com.

[Jason Youngquist]

Hey Chema.

Thanks for the information.  Is there a way to filter these on a per server basis?

ie.ignore eventid 1102 on exchange01

<localfile>

  <location>Security</location>

  <log_format>eventchannel</log_format>

  <query>Event/System[EventID != 1102]

</localfile>

Thanks.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[Chema Martinez]

Hi Jason,

Sure, you can use metadata of events to filter them, writing queries as complex as you need. I have no logs from Exchange on my hand but it has to contain any field that fit your needs. I will show you an example with a random event, and if you need further assistance I would be happy to help if you can provide us a raw event for your use case.

We have the following event log:

If we want to filter events with the ID 4658 and user SID S-1-5-18. The configuration to ignore events with those characteristics would look like this one:

<localfile>

  <location>Security</location>

  <log_format>eventchannel</log_format>

  <query>Event[System/EventID != 1102 and (EventData[Data[@Name="SubjectUserSid"]!="S-1-5-18")]</query>

</localfile>

As I said before, queries can be as complex as they fit your needs. Here you have more related information:

- Wazuh documentation about querying Windows logs: 

https://documentation.wazuh.com/current/user-manual/reference/ossec-conf/localfile.html#query

https://documentation.wazuh.com/current/user-manual/capabilities/log-data-collection/how-to-collect-wlogs.html#filtering-events-from-windows-event-channel-with-queries

- An interesting article about queries:

https://blogs.technet.microsoft.com/askds/2011/09/26/advanced-xml-filtering-in-the-windows-event-viewer/

You can find more examples in those links.

I hope it helps.

Best regards.

Chema Martinez IT Security Engineer  The Open Source Security Platform

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/60df2fed-4b8a-4f20-8e50-f90c9581efca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/f36f6901-4097-4cb9-9c61-dd5c39c4b4f0%40googlegroups.com.