Queue Size

[Sam Heuchert]

Hi!

I'm running the latest 4.3.5 version of Wazuh, All in One.  I noticed rather high disk usage on my SAN (I'm running a VM in Hyper V).   I am wondering if there is a backlog of events that need to be decoded and processed.  Is there a way to check this in the logs?

Thanks!

[Andres Micalizzi]

Hi Sheuchert,
Thanks for using Wazuh.

In wazuh's architecture, you will have all your agents, collecting logs and information and sending it to the manager to be analyzed by it. It might be possible that if you have a lot of agents connected to your manager, and they have many logs to forward to the manager, that it will be active doing the analysis, until it can process them.

It is also possible that you manager is being flooded by the agent's messages because they have an improper configuration with no limit on the ammount of events or messages they send over to the manager.

Here you can check how to configure the agent's queue size, to limit how many events they send over to the manager.

As to your question if it is possible to check the backlog of messages or logs to read, you can find that informationin the files inside the /var/ossec/var/run folder. Inside you will find the state files, that will give you statistics about current usage of that daemon. For example, you can check wazuh-analysisd.state file to check how much of the queue is being used at the moment for different actions by the manager.

Also, you can check here, what is the expected disk usage and calculate how much storage you would be using with your current setup.

I hope this clears up your question. In case of further doubt do not hesitate to ask.
Cheers,
Andrés.

[Sam Heuchert]

Thanks for the quick response!  Everything looks clean to me based on your instructions.  However, my Manager is now running about 10 minutes behind in analyzing events.  Is there a variable in the wazuh-analysisd.state file that would show me where the backlog is?  As mentioned, everything is showing normal from what I can see, but I'm not familiar with the exact meaning of each value.  I could post the contents of the file if that would be helpful.

Thanks!

[Andres Micalizzi]

Well, that is weird that the manager is now lagging behind.

What is your current setup? How many manager+worker nodes do you have? How many agents are connected to them?

If you share the state file, and the ossec.log file, we can check what is causing it to lag behind.

[Sam Heuchert]

I have a single AIO node with no worker nodes.  I have 300 agents.  Here is the state file!

[Sam Heuchert]

I finally figured it out!  My RAID card was actually failing in my hypervisor running the VM.  I replaced it and everything is back to normal.