After today's update to kibana, all wazuh servers are down

[J J Sloan]

All Wazuh platforms are Centos 8.4
selinux enforcing or not makes no difference

[root@wazuh kibana]# systemctl status kibana

● kibana.service - Kibana

   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Tue 2021-05-18 10:59:34 PDT; 12min ago

  Process: 1844 ExecStart=/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml (code=exited, >

 Main PID: 1844 (code=exited, status=1/FAILURE)

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: kibana.service: Main process exited, code=exit>

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: kibana.service: Failed with result 'exit-code'.

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: kibana.service: Service RestartSec=100ms expir>

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: kibana.service: Scheduled restart job, restart>

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: Stopped Kibana.

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: kibana.service: Start request repeated too qui>

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: kibana.service: Failed with result 'exit-code'.

May 18 10:59:34 wazuh.mainphrame.net systemd[1]: Failed to start Kibana.

[root@wazuh kibana]# rpm -qi opendistroforelasticsearch-kibana

Name        : opendistroforelasticsearch-kibana

Version     : 1.13.2

Release     : 1

Architecture: x86_64

Install Date: Tue 18 May 2021 10:06:04 AM PDT

Group       : default

Size        : 691907093

License     : ASL 2.0

Signature   : RSA/SHA256, Wed 14 Apr 2021 09:56:02 AM PDT, Key ID 96b3ee5f29111145

Source RPM  : opendistroforelasticsearch-kibana-1.13.2-1.src.rpm

Build Date  : Mon 05 Apr 2021 01:57:24 PM PDT

Build Host  : 61ab017b841a

Relocations : / 

Packager    : Opendistro Team <opendistrofor...@amazon.com>

Vendor      : Amazon Web Services, Inc.

URL         : https://aws.amazon.com/

Summary     : Explore and visualize your Elasticsearch data

Description :

Explore and visualize your Elasticsearch data

Jake

[J J Sloan]

Root cause:

 FATAL  Error: Failed to initialize plugins:

Plugin "wazuh" is only compatible with Kibana version "7.10.0", but used Kibana version is "7.10.2". (incompatible-version, /usr/share/kibana/plugins/wazuh/kibana.json)

(I sent an earlier reply to this thread by email but it has not shown up)

[J J Sloan]

So, the question is, do we edit the config file to make it allow kibana 7.10.2?

Or do we downgrade to kibana 7.10.0?

Or should we expect an imminent upgrade of wazuh-manager?

Joe

[Alberto Rodriguez]

Hello 

  The Wazuh Kibana plugin you tried to install is not the corresponding one to Wazuh v4.1.5 and Opendistro 1.13.2 (which works with Elasticsearch/Kibana 7.10.2). Following the Upgrade guide, please make sure that the commands

cd /usr/share/kibana/

sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/4.x/ui/kibana/wazuh_kibana-4.1.5_7.10.2-1.zip

are correctly run. Please let me know when you try again with the results. 

Regards, 

Alberto R

[J J Sloan]

Hi Alberto.

I didn't "try to install" anything. 

I did a system package upgrade, and the wazuh repo provided incompatible packages. This will affect many users.

We need to stay with the managed packages, rather than downloading zip files. 

Your thoughts?

Jake

[Alberto Rodriguez]

Hello J J Sloan

You are completely right. We suggest always perform manual upgrades because the Wazuh Kibana component it's needs to be manually added. 
You are not losing alerts or events, so there is no problem. The only issue is related to the Wazuh kibana plugin that which must be installed manually as described in the Upgrade guide. 

Having said this, the problem you pointed is annoying and we are working on providing a package with the Wazuh Web User Interface, it could be upgraded with the system upgrade like others and we will avoid problems like the one you had. Until then, please take into account the manual upgrade in case of new Wazuh / Opendistro packages in our repositories. 

Sorry for the inconvenience.

Regards, 

Alberto R

[J J Sloan]

Thank you Alberto.

I know the email alerts are still working, so I'm prepared to wait a bit for the wazuh updates in the repo.

Joe

[J J Sloan]

Hi Alberto,

Is there anywhere we can download the previous opendistroforelasticsearch-kibana rpm?

J

[Alberto Rodriguez]

Sure, here: https://packages.wazuh.com/4.x/yum/opendistroforelasticsearch-kibana-1.12.0.rpm

Regards, 

Alberto R

[J J Sloan]

Hi Alberto,

I've installed Kibana 1.12.0 on the Centos 8 Wazuh servers, but kibana refuses to start. Perhaps you can provide some advice based on the error message:

May 20 12:58:03 wazuh kibana[41716]: {"type":"log","@timestamp":"2021-05-20T19:58:03Z","tags":["warning","config","deprecation"],"pid":41716,"message":"\"server.defaultRoute\" is deprecated and has been replaced by \"uiSettings.overrides.defaultRoute\""}

May 20 12:58:03 wazuh kibana[41716]: {"type":"log","@timestamp":"2021-05-20T19:58:03Z","tags":["fatal","root"],"pid":41716,"message":"{ Error: listen EACCES: permission denied 0.0.0.0:443\n    at Server.setupListenHandle [as _listen2] (net.js:1263:19)\n    at listenInCluster (net.js:1328:12)\n    at doListen (net.js:1461:7)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  code: 'EACCES',\n  errno: 'EACCES',\n  syscall: 'listen',\n  address: '0.0.0.0',\n  port: 443 }"}

May 20 12:58:03 wazuh kibana[41716]: FATAL  Error: listen EACCES: permission denied 0.0.0.0:443

May 20 12:58:03 wazuh systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE

We did the original install using the all-in-one installer script around March 4th, so whatever version was in the repos at that time would have been used.

FWIW, I fired up the Centos 7 Wazuh server, which had been offline, and so not upgraded, and it's running opendistroforelasticsearch-kibana-1.12.0-1.x86.

J

[Alberto Rodriguez]

Looks like the new Kibana it's trying to start but some other process (probably old Kibana) it's listening to the same port as the new. So, try to stop kibana 

systemctl stop kibana

Make sure that no more process is listening to 443 port:

netstat -tunap | grep 443

If you have not installed netstat, you could by running: yum install net-tools

If you have any other process listening to port 443, kill it using: kill -9 id-process  

Then, make sure that you give right permissions to Kibana: 
setcap 'cap_net_bind_service=+ep' /usr/share/kibana/node/bin/node

And try to start it again by systemctl start kibana 

Let me know if it worked. 

Regards, 

Alberto R

[jjs - mainphrame]

This seems to be the root cause -  

 log   [18:21:34.345] [error][plugins-service] { Error: Plugin "wazuh" is only compatible with Kibana version "7.10.0", but used Kibana version is "7.10.2". (incompatible-version, /usr/share/kibana/plugins/wazuh/kibana.json)

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/naYtxiyYqpw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9a41dcd4-bf9f-4e74-a1de-e42a53c5c723n%40googlegroups.com.

[jjs - mainphrame]

Hi Alberto,

Thanks for the assist.

It turns out that node was indeed missing capabilities, but adding them per your note allowed kibana to start. so it's all good now.

Apparently the all-in-one installer sets the capabilities, but the normal package install doesn't

BTW Do you recommend disabling the wazuh repo after install, to prevent any future incompatible updates?

J

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/naYtxiyYqpw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/83d2494a-cce6-46a3-8337-cb1a8d88ed44n%40googlegroups.com.

[Alberto Rodriguez]

Hello

 Yes, but this can be easily fixed by following the steps explained in this document: https://documentation.wazuh.com/current/upgrade-guide/elasticsearch-kibana-filebeat/upgrading-open-distro.html#upgrading-kibana

You should skip step 4 and other non-corresponding to your previous version. Please let me know if you have any questions. 

Regards, 

Alberto R

You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/CABkzw7zQTEWrUmcUiXxydxPa7vcJ7fN_E-GFBbkArs89MCOUSA%40mail.gmail.com.

--

Alberto Rodriguez CICD TL  The Open Source Security Platform

* This message and the information contained in or attached to it are private and confidential and intended exclusively for the addressee. Any dissemination, copying or distribution to third parties without the express consent of the sender is strictly prohibited. If you have received this message in error, please delete it immediately and notify the sender. Thank you for your collaboration.