wazuh-modulesd->failed

[BS]

Hello,

I just upgraded my version of Wazuh to 3.11.3 and now it throws an error saying "Wazuh API seems to be down." It shows that an API is configured but gives the following error 

"3099 - ERROR3099 - Some Wazuh daemons are not ready in node 'node01' (wazuh-modulesd->failed)"

What can I do to troubleshoot?

Thanks,

BS

[Victor Santaella]

Hi BS,

As far as I can see in the error you send me, there is an error connecting the app to the API because the API service is in a failed state.

Have you tried restarting the API service?

systemctl restart wazuh-api

On the other hand, it seems that also some deamos are not active. Try restarting Wazuh,s manager as well.

systemctl restart wazuh-manager

Once restarted and checked that they are running correctly,

systemctl status wazuh-api
systemctl status wazuh-manager

Try again to enter the API connection credentials from the Kibana app.

Let us know the result.

Regards,

Victor Santaella

[BS]

Hi Victor,

I've done a restart and I get the below message when I check the status.

root@wazuh-manager:/# service wazuh-api status

WAZUH-API is running.

root@wazuh-manager:/# service wazuh-manager status

wazuh-clusterd is running...

wazuh-modulesd not running...

ossec-monitord not running...

ossec-logcollector not running...

ossec-remoted not running...

ossec-syscheckd not running...

ossec-analysisd not running...

ossec-maild: Process 53378 not used by Wazuh, removing...

ossec-maild not running...

ossec-execd not running...

wazuh-db not running...

ossec-authd not running...

ossec-agentlessd is running...

ossec-integratord not running...

ossec-dbd not running...

ossec-csyslogd not running...

root@wazuh-manager:/#

In case it helps, I am running this as a container deployment.

[Victor Santaella]

Hi BS

It seems that the problem is in ossec, could you please share with me the ossec log?

cat /var/ossec/logs/ossec.log

Regards

Victor Santaella

[BS]

Hi Victor,

The below is a sample of the entries I see in the ossec.log file

2020/02/27 10:53:33 ossec-remoted: WARNING: (1408): Invalid ID 1272 for the source ip: '10.2.3.138' (name 'unknown').

2020/02/27 10:53:33 ossec-remoted: WARNING: (1408): Invalid ID 494 for the source ip: '10.2.3.138' (name 'unknown').

2020/02/27 10:53:33 wazuh-modulesd[45015] wdb.c:360 at wdb_step(): DEBUG: Maximum attempts exceeded for sqlite3_step()

2020/02/27 10:53:33 wazuh-modulesd[53424] wdb.c:360 at wdb_step(): DEBUG: Maximum attempts exceeded for sqlite3_step()

2020/02/27 10:53:33 wazuh-modulesd[45015] wdb.c:360 at wdb_step(): DEBUG: Maximum attempts exceeded for sqlite3_step()

2020/02/27 16:23:33 wazuh-modulesd[37953] wdb.c:360 at wdb_step(): DEBUG: Maximum attempts exceeded for sqlite3_step()

2020/02/27 10:53:33 wazuh-modulesd[53424] wdb.c:360 at wdb_step(): DEBUG: Maximum attempts exceeded for sqlite3_step()

2020/02/27 10:53:33 ossec-remoted: WARNING: (1408): Invalid ID 1277 for the source ip: '10.2.3.138' (name 'server1').

2020/02/27 10:53:33 ossec-remoted: WARNING: (1408): Invalid ID 1343 for the source ip: '10.2.3.138' (name 'server1').

2020/02/27 10:53:33 ossec-remoted: WARNING: (1408): Invalid ID 1294 for the source ip: '10.2.3.138' (name 'unknown').

root@wazuh-manager:/var/ossec/logs#

Are you looking for any specific entries?

- BS

[Victor Santaella]

Hi BS,

Have you updated the user API too?

Wazuh and Wazuh API have the same version. To check the version you have, you can run the following command:

cat /var/ossec/api/package.json | grep version

If the version of your API is lower you must update it, for this you can follow the documentation:

https://documentation.wazuh.com/3.11/upgrade-guide/upgrading/latest_wazuh3_minor.html#upgrade-the-wazuh-manager-and-api

Regards

Victor Santaella.

[BS]

Hi Victor,

The version is 3.11.3

root@wazuh-manager:/# cat /var/ossec/api/package.json | grep version

  "version": "3.11.3",

root@wazuh-manager:/#

Regards,

Bhowmik

[Victor Santaella]

Hi BS,

If the version of API and App is the same, we will try something else to solve this error.

The first thing will be to stop the wazuh manager service, for this use this:

systemctl stop wazuh-manager

The next thing is to kill the wazuh-api process while it is still running. After doing this on these two services try again to start the services and check that everything works.

This error may be that this is happening because some api or app daemons have not started correctly.

Please share if this has solved your problem.

Regards,

Victor Santaella.

[BS]

Hi Victor,

I did that but still no luck. Also, I restarted all the containers as well but that also hasn't fixed it and wazuh-modulesd is still not running.

- BS

[Victor Santaella]

Hi BS,

Try the following steps on the machine where Wazuh manager is installed:

rm -rf /var/ossec/var/db/global.db*
/var/ossec/bin/ossec-control restart

This will erase your databases and restart your manager. Don't worry, the databases will be regenerated again.

Let us know if this helps.

Regards,
Victor Santaella.

[BS]

Hi Victor,

Tried that as well, I still keep getting the same error as below,

"3099 - ERROR3099 - Some Wazuh daemons are not ready in node 'node01' (wazuh-modulesd->failed)"

- BS

[Victor Santaella]

Hi Bs

Could you please provide me with information about the whole environment you are using to work (versions, containers and their versions ...).

Regards,

Victor Santaella.

[BS]

Hi Victor,

Below are the details about my environment,

1) Host Operating System - CentOS Linux release 7.7.1908 (Core)

2) Containers

     2a) wazuh/wazuh-nginx:3.11.3_7.5.2

     2b) wazuh/wazuh-kibana:3.11.3_7.5.2

     2c) wazuh/wazuh:3.11.3_7.5.2

     2d) wazuh/wazuh-elasticsearch:3.11.3_7.5.2

Let me know if you need any more information.

- BS

[Victor Santaella]

Hi BS,

Reviewing the previous messages I have noticed that the problem may be in the ossec configuration, since with the log error you shared:

2020/02/27 16:23:33 wazuh-modulesd [37953] wdb.c: 360 at wdb_step (): DEBUG: Maximum attempts exceeded for sqlite3_step ()

This does not start modules d. When sqlite3_step() is executed and SQLITE_BUSY returns, it means that the database engine was unable to acquire the database locks it needs to do its job. If the statement is a COMMIT or occurs outside of an explicit transaction, then you can retry the statement. If the statement is not a COMMIT and occurs within an explicit transaction then you should rollback the transaction before continuing.

As the error persists, please share with me the ossec.conf file so that I can simulate your environment and investigate the problem.

On the other hand I would need you to share more logs about the problem, for them it is necessary that you activate the debug mode in Wazuh, you can guide how to do it from the following link:

https://documentation.wazuh.com/3.11/user-manual/reference/internal-options.html#wazuh-modules

Regards

Victor Santaella.

[BS]

Hi Victor,

I've shared the config file separately with you and also enabled the debugging.

I'll share the debug logs as well.

-BS

[Victor Santaella]

Hi BS,

I have been simulating your environment in containers with the same configuration that you passed me and with the same version and everything works correctly.

We are going to try something else to focus on the error since it seems that the source is modulesd which does not start and does not allow the API to start.

systemctl restart wazuh-manager
systemctl restart wazuh-api

When you restart both services, wait about 20 seconds for the service to restart correctly and use this command to share the ossec logs again with me.

tail /var/ossec/logs/ossec.log -n 100

I need more logs because the ones we have are not giving me enough information about the problem.

Regards,

Victor Santaella