Wazuh Cluster shutdown and lost all the logs

187 views
Skip to first unread message

Saddique Khan

unread,
Sep 28, 2023, 3:43:37 AM9/28/23
to Wazuh | Mailing List
Hello Team,
 
      My wazuh cluster stops working when I don't refresh it or leave it for some hour. I see the following error in the work pod. When I restart the manager, everything comes up but no logs. Please help

wazuh-modulesd: ERROR: (1278): Unable to reconnect to 'queue/sockets/queue': Con
nection refused (111).


Regards,
Saddique

John Ebuka Onyejegbu

unread,
Sep 28, 2023, 6:05:51 AM9/28/23
to Wazuh | Mailing List
Hi Saddique,

What version of wazuh are you running?
you might also want to check if wazuh-modulesd daemon is enabled: /var/ossec/bin/ossec-control status . If not execute /var/ossec/bin/wazuh-modulesd

Saddique Khan

unread,
Sep 28, 2023, 6:08:48 AM9/28/23
to Wazuh | Mailing List
Hey John,

         I am using 4.5.0 version on Kubernetes: This is the status for control API. 

      root@wazuh-manager-master-0:/# /var/ossec/bin/wazuh-control status
wazuh-clusterd is running...
wazuh-modulesd is running...
wazuh-monitord is running...
wazuh-logcollector is running...
wazuh-remoted is running...
wazuh-syscheckd is running...
wazuh-analysisd is running...
wazuh-maild not running...
wazuh-execd is running...
wazuh-db is running...
wazuh-authd is running...
wazuh-agentlessd not running...
wazuh-integratord is running...
wazuh-dbd not running...
wazuh-csyslogd not running...
wazuh-apid: Process 14187 not used by Wazuh, removing...
wazuh-apid not running...

The master goes down very frequently.

Regards,
Saddique

John Ebuka Onyejegbu

unread,
Sep 28, 2023, 7:03:08 AM9/28/23
to Wazuh | Mailing List
Hi Saddique,

can you upload the contents of the following files (please mask sensitive data were appropraite)
/var/ossec/logs/ossec.log
/var/ossec/etc/ossec.conf

Saddique Khan

unread,
Sep 28, 2023, 7:14:32 AM9/28/23
to Wazuh | Mailing List
Hello John,

       These are the files. I restarted the Wazuh manager, I believe that there will not be any specific error right now. However, I will also share it once it stops working. However, I have obeserved one thing that when It stops working, the Unable to reconnect to 'queue/sockets/queue': Con error
      comes in the worker pod.
ossec.xml
ossec.log

John Ebuka Onyejegbu

unread,
Sep 28, 2023, 8:50:15 AM9/28/23
to Wazuh | Mailing List
Hi Saddique,

From the logs am noticing alerts.json is not created and perhaps some other dependencies.
can you run the following: 
/var/ossec/bin/ossec-control start

Saddique Khan

unread,
Sep 28, 2023, 10:17:00 AM9/28/23
to Wazuh | Mailing List
Hello John,

           Here is the output.

  Starting Wazuh v4.5.0...

wazuh-apid already running...
Started wazuh-csyslogd...
Started wazuh-dbd...
wazuh-integratord already running...
Started wazuh-agentlessd...
wazuh-authd already running...
wazuh-db already running...
wazuh-execd already running...
wazuh-analysisd already running...
wazuh-syscheckd already running...
wazuh-remoted already running...
wazuh-logcollector already running...
wazuh-monitord already running...
wazuh-modulesd already running...
wazuh-clusterd already running...
Completed.

Regards,
Saddiqe

John Ebuka Onyejegbu

unread,
Sep 29, 2023, 3:46:20 AM9/29/23
to Wazuh | Mailing List
Hello Saddique,

output looks good.
Please do share the  /var/ossec/logs/ossec.log file once you encounter the issue again.

Regards.

Saddique Khan

unread,
Sep 29, 2023, 9:36:52 AM9/29/23
to Wazuh | Mailing List
Hello John,

           Here is the actual issue. Could you please check the logs?

 Regards,
Saddique
worker-ossec.log

John Ebuka Onyejegbu

unread,
Oct 1, 2023, 6:29:14 PM10/1/23
to Wazuh | Mailing List
Dear Saddique,

I have looked at the logs and can see that the issue is related to ossec-analysisd failing repeatedly.
This is usually caused by the wazuh manager using a low core CPU. increasing the CPU core can solve this problem.
But if you have a very good CPU core and the problem is still persisting then you have follow the below steps.
1. Set the analysisd.sca_threads option from 0 to 1 in /var/ossec/etc/internal_options.conf
2. Restart your wazuh manager.

Saddique Khan

unread,
Oct 9, 2023, 4:26:22 AM10/9/23
to Wazuh | Mailing List
Hello John,

          Thanks for the suggestion. I have updated wazuh manager core. It looks to some extend. However, The other setup of wazuh is getting down. I am monitoring it. I will let you know If i will find something fishi..

Regards,
Saddique

John Ebuka Onyejegbu

unread,
Oct 9, 2023, 7:22:11 AM10/9/23
to Wazuh | Mailing List
Dear Saddique,

This is noted, Do let me know if anything changes.

Regards.

Saddique Khan

unread,
Oct 12, 2023, 3:49:56 AM10/12/23
to Wazuh | Mailing List
Hello John,

            I checked it for two days and it is working perfectly fine now.
           Thank you

Regards,
Saddique

Reply all
Reply to author
Forward
0 new messages