New 4.1 install - port 443 connection refused

[Kris Springer]

Hi, I'm attempting to install Wazuh Server 4.1 (All-in-one) on a new Ubuntu Server v20.04 system.  I first tried the 'Unattended Install' option, but that failed to successfully complete, so I tried the 'Step-by-Step' option.  That went fine until I got to the 'Install Kibana' section.  There seems to be an error in how the opendistroforelasticsearch-kibana apt installs.  It fails to auto create the /usr/share/kibana/optimize folder, and the apt install fails because it's trying to chown a non-existent folder.  After I manually created that folder and then ran 'apt reinstall' on kibana, it seemingly completed the install successfully.  After finishing all the remaining steps in the instructions I try to access the new system from a browser and all I get is 'connection refused'.  This error means the server is responding on port 443, but it's just refusing to serve it's pages.  I've searched this mailing list for answers, but found none. I've searched Google and found none.  I've tried changing a few things in the kibana.yml file to see if it would magically fix anything, but it doesn't.  Any idea what could be going on here?

Here's what my kibana.yml file looks like.

server.host: 0.0.0.0

server.port: 443

elasticsearch.hosts: https://localhost:9200

elasticsearch.ssl.verificationMode: certificate

elasticsearch.username: kibanaserver

elasticsearch.password: kibanaserver

elasticsearch.requestHeadersWhitelist: ["securitytenant","Authorization"]

opendistro_security.multitenancy.enabled: true

opendistro_security.readonly_mode.roles: ["kibana_read_only"]

server.ssl.enabled: true

server.ssl.key: "/etc/kibana/certs/kibana-key.pem"

server.ssl.certificate: "/etc/kibana/certs/kibana.pem"

elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/certs/root-ca.pem"]

server.defaultRoute: /app/wazuh?security_tenant=global

I also suspected that Ubuntu firewall may be blocking, but 'ufw status' shows 'inactive', meaning that the firewall is allowing all ports.

[Federico Rodriguez]

Hi Kris!

the optimize folder is used in older than 4.1 Wazuh versions, recent versions use the data folder. Could you please provide the guide link which you used to install Wazuh?
Also to dive deeper into what happened, we can check Wazuh and Kibana logs:

To read the journal of kibana service you can run:
journalctl -u kibana.service -n100 -r 

Wazuh logs file location depends on Wazuh version:

KIBANA_PATH/data/wazuh/logs/wazuhapp-plain.log

If Wazuh version is lower than 4.1 you can check the optimize folder instead of the data folder

[Kris Springer]

I first tried the 'Unattended' method, but that failed.

https://documentation.wazuh.com/current/installation-guide/open-distro/all-in-one-deployment/unattended-installation.html

I then followed all the steps for 'Manual' install.

https://documentation.wazuh.com/current/installation-guide/open-distro/all-in-one-deployment/all_in_one.html

[Kris Springer]

the 'journal of kibana' command shows this.

-- Logs begin at Tue 2021-06-22 18:31:46 UTC, end at Wed 2021-06-23 20:07:04 UTC. --

-- No entries --

Looking in the kibana data folder located in /usr/share/kibana/data/ shows nothing but 1 file called 'uuid'.  No logs folder.

On Wednesday, June 23, 2021 at 10:27:23 AM UTC-6 federico....@wazuh.com wrote:

[Federico Rodriguez]

There are 2 strange things we should look into:
The current wazuh installation script has no reference to optimize folder, so it seems very strange you got an optimize folder error. This might indicate an old script was used.

Please verify Kibana installation steps, in particular data folder step 3
https://documentation.wazuh.com/current/installation-guide/open-distro/all-in-one-deployment/all_in_one.html#installing-kibana

On the other hand, if the journalctl is empty it probably means Kibana never actually started. We can check the daemon status by running service kibana status. In case its inactive/stopped try to start it running service kibana start. Then please check if the journalctl has new entries.

[Kris Springer]

I verified step 3 as suggested.  The data folder exists with the correct kibana user:group permissions.  I then checked 'service kibana status' and this was the output.

● kibana.service - Kibana

     Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)

     Active: active (running) since Thu 2021-06-24 13:32:32 UTC; 1s ago

   Main PID: 319304 (node)

      Tasks: 7 (limit: 19110)

     Memory: 43.7M

     CGroup: /system.slice/kibana.service

             └─319304 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist -c /etc/kibana/kibana.yml

I checked the journalctl command again and this is what I see now.

Jun 24 13:37:18 xymon systemd[1]: Stopped Kibana.

Jun 24 13:37:18 xymon systemd[1]: kibana.service: Scheduled restart job, restart counter is at 11.

Jun 24 13:37:18 xymon systemd[1]: kibana.service: Failed with result 'exit-code'.

Jun 24 13:37:18 xymon systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE

Jun 24 13:37:18 xymon kibana[320309]:  FATAL  Error: EACCES: permission denied, open '/etc/kibana/certs/kibana-key.pem'

Jun 24 13:37:18 xymon kibana[320309]: {"type":"log","@timestamp":"2021-06-24T13:37:18Z","tags":["fatal","root"],"pid":320309,"message":"{ Error: EACCES: permission denied, open '/etc/kibana/certs/kibana-key.pem'\n    at Object.ope>

Jun 24 13:37:18 xymon kibana[320309]: {"type":"log","@timestamp":"2021-06-24T13:37:18Z","tags":["warning","config","deprecation"],"pid":320309,"message":"\"server.defaultRoute\" is deprecated and has been replaced by \"uiSettings.>

Jun 24 13:37:17 xymon kibana[320309]: {"type":"log","@timestamp":"2021-06-24T13:37:17Z","tags":["info","plugins-service"],"pid":320309,"message":"Plugin \"visTypeXy\" is disabled."}

Jun 24 13:37:10 xymon systemd[1]: Started Kibana.

There appears to be a permissions error on the /etc/kibana/certs/ folder.  So I ran ' chown -R kibana:kibana /usr/share/kibana/data ' and then restarted the kibana service again, then checked the journalctl and this time it showed this.

Jun 24 13:42:56 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:56Z","tags":["info","http","server","Kibana"],"pid":321492,"message":"http server running at https://0.0.0.0:443"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["listening","info"],"pid":321492,"message":"Server running at https://0.0.0.0:443"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["error","plugins","wazuh","initialize"],"pid":321492,"message":"Response Error"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["error","plugins","wazuh","initialize"],"pid":321492,"message":"Response Error"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["error","elasticsearch","data"],"pid":321492,"message":"[ResponseError]: Response Error"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["error","elasticsearch","data"],"pid":321492,"message":"[ResponseError]: Response Error"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["info","plugins-system"],"pid":321492,"message":"Starting [50] plugins: [opendistroAlertingKibana,usageCollection,telemetryCollectionM>

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["info","savedobjects-service"],"pid":321492,"message":"Starting saved objects migrations"}

Jun 24 13:42:55 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:55Z","tags":["info","savedobjects-service"],"pid":321492,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before startin>

Jun 24 13:42:54 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:54Z","tags":["info","plugins-system"],"pid":321492,"message":"Setting up [50] plugins: [opendistroAlertingKibana,usageCollection,telemetryCollectio>

Jun 24 13:42:54 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:54Z","tags":["warning","config","deprecation"],"pid":321492,"message":"\"server.defaultRoute\" is deprecated and has been replaced by \"uiSettings.>

Jun 24 13:42:54 xymon kibana[321492]: {"type":"log","@timestamp":"2021-06-24T13:42:54Z","tags":["info","plugins-service"],"pid":321492,"message":"Plugin \"visTypeXy\" is disabled."}

Jun 24 13:42:47 xymon systemd[1]: Started Kibana.

There's still some errors and warnings shown, but I tried to access the webpage and I successfully got the login page and was able to log in!

Perhaps you're missing a permissions chown command in step 5 of your 'Installing Kibana' instructions??

---------------------------------------------------

Jun 24 13:32:32 xymon systemd[1]: Started Kibana.

[Federico Rodriguez]

Great! May be you missed the permissions changes step described here:

I'm glad it's working. Should you stumble with something else don't hesitate to ask for help.

[KSpringer]

The chown permission you referenced is not the same path as certs. The certs command is in step 5 and is in /etc

--
Kris Springer

-----Original Message-----
From: Federico Rodriguez <federico....@wazuh.com>
To: Wazuh mailing list <wa...@googlegroups.com>
Subject: Re: New 4.1 install - port 443 connection refused
Date: Thu, 24 Jun 2021 08:15:28 -0700 (PDT)

Great! May be you missed the permissions changes step described here:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/bLyC0EjfKKI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/wazuh/9c401a64-13d8-4a70-8629-4f69acbd9c5en%40googlegroups.com.

[Federico Rodriguez]

My bad. I thought you ran chown -R  just in data folder. Thanks for the clarification!