API failing with 400 code - wazuh-modulesd -> failed

[Samuel Kajiwara]

Greetings team,

We are having an issue with our Elastic API failing to connect to Wazuh server. Over the last few days, we have been getting error: 

INFO: Current API id [default]
INFO: Checking current API id [default]...
INFO: Current API id [default] has some problem: 3002 - Request failed with status code 400
INFO: Getting API hosts...
INFO: API hosts found: 1
INFO: Checking API host id [default]...
INFO: Could not connect to API id [default]: 3099 - ERROR3099 - Some Wazuh daemons are not ready yet in node "node01" (wazuh-modulesd->failed)
INFO: Removed [navigate] cookie
ERROR: No API available to connect

When viewing the api.log file, we see INFO errors where the user changes from "wazuh" to "unknown_user". After doing some research we have consistently had these errors in the past (around 150 per day) but this week we are seeing an uptick (600-1300 per day) and when the API goes down, the api.log contains only "unknown_user" and the modulesd service stops. 

Example from api.log

2022/08/04 01:20:00 INFO: wazuh [REDACTED IP] "GET /manager/stats/analysisd" with parameters {"pretty": ""} and body {} done in 0.008s: 200

2022/08/04 01:25:00 INFO: unknown_user [REDACTED IP]  "GET /manager/stats/analysisd" with parameters {"pretty": ""} and body {} done in 0.002s: 401

2022/08/04 01:25:00 INFO: unknown_user [REDACTED IP]  "GET /manager/stats/remoted" with parameters {"pretty": ""} and body {} done in 0.001s: 401

2022/08/04 01:25:00 INFO: wazuh [REDACTED IP]  "GET /security/user/authenticate" with parameters {} and body {} done in 0.439s: 200

Example from wazuhapp.log

info: 2022/08/04 10:5:0: Cron-scheduler: {"message":"Request failed with status code 400","stack":"Error: Request failed with status code 400\n    at createError (/usr/share/kibana/plugins/wazuh/node_modules/axios/lib/core/createError.js:16:15)\n    at settle (/usr/share/kibana/plugins/wazuh/node_modules/axios/lib/core/settle.js:17:12)\n    at IncomingMessage.handleStreamEnd (/usr/share/kibana/plugins/wazuh/node_modules/axios/lib/adapters/http.js:260:11)\n    at IncomingMessage.emit (events.js:412:35)\n    at endReadableNT (internal/streams/readable.js:1317:12)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21)","config":{"url":[REDACTED]:55000/security/user/authenticate","method":"get"}}

error: 2022/08/04 10:5:35: wazuh-api:getToken: Some Wazuh daemons are not ready yet in node "node01" (wazuh-modulesd->failed)

error: 2022/08/04 10:5:35: wazuh-api:checkStoredAPI: Request failed with status code 400

error: 2022/08/04 10:5:35: wazuh-api:checkStoredAPI: Request failed with status code 400

We can start the API again by restarting the wazuh-manager or starting the modulesd service again, however it keeps going down. At the moment with the vulnerability scanner disabled, the influx in this occurrence is down, though still occurring. I'm not sure if this is the cause or just a coincidence though. 

 

[Emiliano Zorn]

Hello team! Hope you are doing good.

Could you please specify which Wazuh version you are currently using? In the meantime, let's execute this request and check the API response:

curl -u USER:PASS -k -X GET "https://<your-api-ip>:55000/security/user/authenticate"

The API connection configuration should be available in wazuh.yml file.

You can also check if the API service is running with the following command:

service wazuh-manager status

Also, there's a chance that you have the wazuh manager daemons stopped. Could you please restart the wazuh manager using systemctl restart wazuh-manager and please share the ossec.log?

grep -E "WARN|ERR" /var/ossec/logs/ossec.log

Regards.

[Samuel Kajiwara]

Hi Emilia,

We are running version 4.2.5 

Running the curl command returns the token value with ERROR: 0.

service wazuh-manager status currently is as follows:

wazuh-clusterd not running...

wazuh-modulesd is running...

wazuh-monitord is running...

wazuh-logcollector is running...

wazuh-remoted is running...

wazuh-syscheckd is running...

wazuh-analysisd is running...

wazuh-maild is running...

wazuh-execd is running...

wazuh-db is running...

wazuh-authd is running...

wazuh-agentlessd not running...

wazuh-integratord not running...

wazuh-dbd not running...

wazuh-csyslogd not running...

wazuh-apid is running...

When the API goes down, we will see wazuh-modulesd stop running. 

Restarting the wazuh-manager and running the grep command returns no results.

Since disabling the Vulnerability module, we have seen no instances of the API dropping.  

[Samuel Kajiwara]

"Since disabling the Vulnerability module, we have seen no instances of the API dropping." - we do still see "unknown_user" in appearing in the logs, just not to the extent that it takes down the API

[Emiliano Zorn]

Hello Samuel!

Thanks for the information provided, however, I would like to see the logs I have requested to focus on the problem, and if you can attach the API logs as well.

I see that your version is out of date, and in newer versions, bugs with the vulnerability module have been fixed, attached:

https://documentation.wazuh.com/current/release-notes/release-4-2-7.html
https://documentation.wazuh.com/current/release-notes/release-4-3-0.html

Regards.