Delayed logs and log archiving

[Wazuh Server]

Hi  Team,

We are using Wazuh 4.3V( indexer, server, manager is on same ubuntu server) from last two months and EPS is around 800. Currently we are facing 6+hours delay in logs to wazuh dashboard from couple of days, These delay logs are getting sorted during non-business time and then again logs are been delayed during business hours. 

Kindly suggest me what changes i have to make to get the logs on-time without any delay during the business hours. 

Also please do let us know if there is any provision to store the only Syslogs archival instead of having all archives(Agent based+Syslog) 

[Isaiah Daboh]

Hello,

Thank you for using Wazuh.

Please allow me to review and revert shortly.

Regards,

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/fe9761d3-c8e2-4d20-adcb-e887e960f8e9n%40googlegroups.com.

[Isaiah Daboh]

Hello,

Kindly provide the following information to enable me assist you further?

Number of agents connected

Manager resources (CPU, RAM and HDD).

Regards,

[Wazuh Server]

Hi Isaiah,
Currently 47 agents are connected to the manager and actively sending logs.

Manager, indexer, dashboard is on same ubuntu server of 32GB RAM and 1TB HDD is assigned initially and recently we have increased to 3TB .

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   48 bits physical, 48 bits virtual
CPU(s):                          8
On-line CPU(s) list:             0-7
Thread(s) per core:              2
Core(s) per socket:              4
Socket(s):                       1

[Isaiah Daboh]

Hello,

Please note that the problem with delay is usually due to resources. When there is a delay, the EPS (Events Per Second) is too high for the manager’s resources. 

A fix could be increasing the manager resources and/or create a cluster or increase the number of nodes for a current cluster.

Regards,

[Isaiah Daboh]

Hello,

In addition, when there is a delay, this could be an indicator that the wazuh-indexer is running low on resources.

Maybe filebeat is claiming about this too. Check filebeat logs and try to find any warning/error that will give you some feedback.If you are running an all-in-one system, a fix could be increasing the resources and/or splitting components for a distributed deployment.

Best regards,

[Wazuh Server]

Could you please help me on how to increase the manager resources! 
Also we encountering frequent Indexer stoppage issue, YES we also observed the filebeat test output is connection refused sometimes. 

Requesting you to please guide on how to make the console available all the time and how to avoid the delay in logs with existing infra of above mentioned.

[Wazuh Server]

Hi Isaiah,

Could you please help me on below request! 

[Isaiah Daboh]

Hello,

You should consider distributed deployment with multi-node clusters for high availability and load balancing. 

https://documentation.wazuh.com/current/deployment-options/elastic-stack/distributed-deployment/index.html

https://documentation.wazuh.com/current/deployment-options/elastic-stack/distributed-deployment/wazuh-cluster/index.html

Regards,

Regards,

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/73e1c524-78bb-4816-a912-613c095aa637n%40googlegroups.com.

[Shiva Gujjanti]

Hi Isaiah,

Is there any possibility to have load balancing with existing setup ?
If not please do let us know how to upgrade the  current setup. 

[Shiva Gujjanti]

Hi Isaish,

i have checked on if any events are dropped with the help of /var/ossec/var/run/wazuh-analysisd.state command but it is ZERO.

# Events dropped
events_dropped='0'

Output of  /var/ossec/var/run/wazuh-remoted.state is 

# Queue size
queue_size='0'

# Total queue size
total_queue_size='231072'

# TCP sessions
tcp_sessions='47'

# Events sent to Analysisd
evt_count='178732566'

# Control messages received
ctrl_msg_count='1554070'

# Discarded messages
discarded_count='13936'

# Messages queued
queued_msgs='2464770'

# Total number of bytes sent
sent_bytes='280468114'

# Total number of bytes received
recv_bytes='93654361644'

# Messages dequeued after the agent closes the connection
dequeued_after_close='400598'

[Shiva Gujjanti]

Hi Isaiah,

Can you please check and update on below request! 

[Shiva Gujjanti]

Hi Isaiah,

We are still awaiting for an update from you. 

[ismailctest C]

Hi,

We are also facing the same issue on wazuh 4.3.8 and 4.3.11.

We are using distributed installation.

Note: 

We have installed the above-mentioned version in 3 different locations and facing the same issue for all locations.

There were no issues when using older versions which are 4.2, 4.1 & 4.0

Kindly help to fix it.

One server conf mentioned below,

Log sources:

Agent count: 10

Syslog: 10

Distributed installation:

Wazuh (CPU: 2 core, Mem: 4GB)

Elastic (CPU: 4 core, Mem: 16GB)