Wazuh API seems to be down

[Aram]

Hello Wazuh Community, 

I am trying to troubleshoot an issue with the Kibana App.  When attempting to access it I receive the "Wazuh API seems to be down" with the following error message "3005 - read ECONNRESET". 

When running 

 curl -k -X GET "https://redacted:55000/" -H "Authorization: Bearer $(curl -u redacted:redacted -k -X GET 'https://redacted:55000/security/user/authenticate?raw=true')"

I receive: curl: (35) TCP connection reset by peer

When listening on the manger over port 55000 I see the following traffic (with the hostnames redacted): 

Flags [.], ack 1, win 229, options [nop,nop,TS val 3164180166 ecr 4024952085], length 0

Flags [P.], seq 1:189, ack 1, win 229, options [nop,nop,TS val 3164180321 ecr 4024952085], length 188

 Flags [.], ack 189, win 235, options [nop,nop,TS val 4024952241 ecr 3164180321], length 0

Flags [R.], seq 1, ack 189, win 235, options [nop,nop,TS val 0 ecr 3164180321], length 0

Flags [S], seq 268711885, win 29200, options [mss 1460,sackOK,TS val 3164180338 ecr 0,nop,wscale 7], length 0

 Flags [S.], seq 238465093, ack 268711886, win 28960, options [mss 1460,sackOK,TS val 4024952258 ecr 3164180338,nop,wscale 7], length 0

This connection has been working for over 6 months with no changes to Wazuh or Elastic. 

In the /usr/share/kibana/optimize/wazuh/config  directory I can see that the "wazuh-registry.json was modified today. Is this expected behavior? There were no human user logged in during the time of the file modification. 

-rw-r--r--. 1 kibana kibana  732 Jul  2 16:28 wazuh-registry.json

-rw-------. 1 kibana kibana 4065 Nov 24  2020 wazuh.yml

These are the only errors in the Wazuh manager log:

2021/07/02 10:12:15 ossec-remoted: ERROR: TCP peer [4729] at x.x.x.x: No route to host (113)

2021/07/02 12:57:26 wazuh-db: ERROR: at wdb_process_insert(): sqlite3_step(): UNIQUE constraint failed: sys_processes.scan_id, sys_processes.pid

2021/07/02 12:57:26 wazuh-db: ERROR: Unable to update 'sys_processes' table for agent '4847'

I have restarted Wazuh, Elasticsearch and Kibana to no avail. Are there any further troubleshooting steps available? 

Thank you!

- Aram 

[Aram]

Additional Information:

4 node Elasticsearch cluster

2 node Wazuh manager

Wazuh Version  4.0 

Elasticsearch 7.9.3

-----------------------------------------------------------------------------

sudo ./ossec-control status

wazuh-clusterd is running...

wazuh-modulesd is running...

ossec-monitord is running...

ossec-logcollector is running...

ossec-remoted is running...

ossec-syscheckd is running...

ossec-analysisd is running...

ossec-maild not running...

ossec-execd is running...

wazuh-db is running...

ossec-authd is running...

ossec-agentlessd not running...

ossec-integratord not running...

ossec-dbd not running...

ossec-csyslogd not running...

wazuh-apid not running...

------------------------------------------------------------------------------

sudo systemctl status wazuh-manager.service

● wazuh-manager.service - Wazuh manager

   Loaded: loaded (/etc/systemd/system/wazuh-manager.service; enabled; vendor preset: disabled)

   Active: active (running) since Fri 2021-07-02 16:27:50 PDT; 2h 4min ago

  Process: 21072 ExecStop=/usr/bin/env ${DIRECTORY}/bin/ossec-control stop (code=exited, status=0/SUCCESS)

  Process: 21212 ExecStart=/usr/bin/env ${DIRECTORY}/bin/ossec-control start (code=exited, status=0/SUCCESS)

   CGroup: /system.slice/wazuh-manager.service

           ├─21312 /var/ossec/bin/ossec-authd

           ├─21329 /var/ossec/bin/wazuh-db

           ├─21353 /var/ossec/bin/ossec-execd

           ├─21368 /var/ossec/bin/ossec-analysisd

           ├─21382 /var/ossec/bin/ossec-syscheckd

           ├─21427 /var/ossec/bin/ossec-remoted

           ├─21461 /var/ossec/bin/ossec-logcollector

           ├─21481 /var/ossec/bin/ossec-monitord

           ├─21525 /var/ossec/bin/wazuh-modulesd

           └─21638 /var/ossec/framework/python/bin/python3 /var/ossec/framework/scripts/wazuh-clusterd.py

Jul 02 16:27:41 wazuh-mgr1 env[21212]: Started ossec-analysisd...

Jul 02 16:27:42 wazuh-mgr1 env[21212]: Started ossec-syscheckd...

Jul 02 16:27:43 wazuh-mgr1 env[21212]: Started ossec-remoted...

Jul 02 16:27:44 wazuh-mgr1 env[21212]: Started ossec-logcollector...

Jul 02 16:27:45 wazuh-mgr1 env[21212]: Started ossec-monitord...

Jul 02 16:27:45 wazuh-mgr1 env[21212]: 2021/07/02 16:27:45 wazuh-modulesd: WARNING: A deprecated Vulnerability Detector configuration block was found. It will be ignored.

Jul 02 16:27:46 wazuh-mgr1 env[21212]: Started wazuh-modulesd...

Jul 02 16:27:48 wazuh-mgr1 env[21212]: Started wazuh-clusterd...

Jul 02 16:27:50 wazuh-mgr1 env[21212]: Completed.

Jul 02 16:27:50 wazuh-mgr1 systemd[1]: Started Wazuh manager.

---------------------------------------------------------------------------------------------------------------------------

# Set the logging level for the Wazuh App log files.

# Default value: info

# Allowed values: info, debug

#logs.level: info

wazuh.monitoring.replicas: 1

wazuh.replicas: 1

hosts:

  - default:

      url: https://wazuh-mgr1

      port: 55000

      username: redacted

      password: redacted

[Franco Hielpos]

Hello Aram!

It seems that your Wazuh API daemon is not running:

sudo ./ossec-control status

wazuh-clusterd is running...

wazuh-modulesd is running...

ossec-monitord is running...

ossec-logcollector is running...

ossec-remoted is running...

ossec-syscheckd is running...

ossec-analysisd is running...

ossec-maild not running...

ossec-execd is running...

wazuh-db is running...

ossec-authd is running...

ossec-agentlessd not running...

ossec-integratord not running...

ossec-dbd not running...

ossec-csyslogd not running...

wazuh-apid not running...

Is this from your master node? Can you share the logs inside /var/ossec/logs/api.log to check what may have failed? Feel free to sanitize these logs by removing any sensitive data.

Additionally you can restart your Wazuh Manager master node to restart the wazuh-apid service.

systemctl restart wazuh-manager

After this, can you check if your wazuh-apid daemon is running again?

/var/ossec/bin/ossec-control | grep apid

Hope this solves your issue! Will be waiting for your feedback.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/2c0f2209-3c8e-4067-9b53-93ae71fa0c8dn%40googlegroups.com.

--

Franco Hielpos
IT Security Engineer — Wazuh, Inc.

[Aram]

Hi Franco, 

Thanks for getting back to me on this. Yes this is from the master node. I have tried restarting the Wazuh service several times with no luck. The wazuh-apid daemon is still not running. When looking through the var/ossec/logs/api.log log files I only see the following entry: 

INFO: Listening on 0.0.0.0:55000..

My next thought is to either try switching the api config to the other Wazuh manger, or to unistall/reinstall the Wazuh plugin. Please let me know what you think. 

Thank you, 

- Aram 

[Aram]

Hi Franco, 

I was able to resolve the issue by rebooting the VM. 

Cheers, 

- Aram