Wazuh Cluster shutdown and lost all the logs

[Saddique Khan]

Hello Team,

 

      My wazuh cluster stops working when I don't refresh it or leave it for some hour. I see the following error in the work pod. When I restart the manager, everything comes up but no logs. Please help

wazuh-modulesd: ERROR: (1278): Unable to reconnect to 'queue/sockets/queue': Con

nection refused (111).

Regards,

Saddique

[John Ebuka Onyejegbu]

Hi Saddique,

What version of wazuh are you running?

you might also want to check if wazuh-modulesd daemon is enabled: /var/ossec/bin/ossec-control status . If not execute /var/ossec/bin/wazuh-modulesd

[Saddique Khan]

Hey John,

         I am using 4.5.0 version on Kubernetes: This is the status for control API. 

      root@wazuh-manager-master-0:/# /var/ossec/bin/wazuh-control status
wazuh-clusterd is running...
wazuh-modulesd is running...
wazuh-monitord is running...
wazuh-logcollector is running...
wazuh-remoted is running...
wazuh-syscheckd is running...
wazuh-analysisd is running...
wazuh-maild not running...
wazuh-execd is running...
wazuh-db is running...
wazuh-authd is running...
wazuh-agentlessd not running...
wazuh-integratord is running...
wazuh-dbd not running...
wazuh-csyslogd not running...
wazuh-apid: Process 14187 not used by Wazuh, removing...
wazuh-apid not running...

The master goes down very frequently.

Regards,

Saddique

[John Ebuka Onyejegbu]

Hi Saddique,

can you upload the contents of the following files (please mask sensitive data were appropraite)

/var/ossec/logs/ossec.log

/var/ossec/etc/ossec.conf

[Saddique Khan]

Hello John,

       These are the files. I restarted the Wazuh manager, I believe that there will not be any specific error right now. However, I will also share it once it stops working. However, I have obeserved one thing that when It stops working, the Unable to reconnect to 'queue/sockets/queue': Con error
      comes in the worker pod.

[John Ebuka Onyejegbu]

Hi Saddique,

From the logs am noticing alerts.json is not created and perhaps some other dependencies.

can you run the following: 
/var/ossec/bin/ossec-control start

[Saddique Khan]

Hello John,

           Here is the output.

  Starting Wazuh v4.5.0...

wazuh-apid already running...
Started wazuh-csyslogd...
Started wazuh-dbd...
wazuh-integratord already running...
Started wazuh-agentlessd...
wazuh-authd already running...
wazuh-db already running...
wazuh-execd already running...
wazuh-analysisd already running...
wazuh-syscheckd already running...
wazuh-remoted already running...
wazuh-logcollector already running...
wazuh-monitord already running...
wazuh-modulesd already running...
wazuh-clusterd already running...
Completed.

Regards,

Saddiqe

[John Ebuka Onyejegbu]

Hello Saddique,

output looks good.

Please do share the  /var/ossec/logs/ossec.log file once you encounter the issue again.

Regards.

[Saddique Khan]

Hello John,

           Here is the actual issue. Could you please check the logs?

 Regards,

Saddique

[John Ebuka Onyejegbu]

Dear Saddique,

I have looked at the logs and can see that the issue is related to ossec-analysisd failing repeatedly.

This is usually caused by the wazuh manager using a low core CPU. increasing the CPU core can solve this problem.

But if you have a very good CPU core and the problem is still persisting then you have follow the below steps.

1. Set the analysisd.sca_threads option from 0 to 1 in /var/ossec/etc/internal_options.conf

2. Restart your wazuh manager.

[Saddique Khan]

Hello John,

          Thanks for the suggestion. I have updated wazuh manager core. It looks to some extend. However, The other setup of wazuh is getting down. I am monitoring it. I will let you know If i will find something fishi..

Regards,

Saddique

[John Ebuka Onyejegbu]

Dear Saddique,

This is noted, Do let me know if anything changes.

Regards.

[Saddique Khan]

Hello John,

            I checked it for two days and it is working perfectly fine now.

           Thank you

Regards,

Saddique