Failed to start Wazuh

[Tung Ton]

Hi Supporter,

One day I got the error " Kibana server is not ready yet" when I access to wazuh UI. I already checked status of elasticsearch(running) and elasticsearch.log ( not anything abnormaly). I run journalctl -u kibana | egrep -i "error":

{"type":"log","@timestamp":"2021-09-30T22:42:36Z","tags":["error","elasticsearch","data"],"pid":663,"message":"[ConnectionError]: connect ECONNREFUSED

and Run sudo -u kibana /usr/share/kibana/bin/kibana --verbose -c /etc/kibana/kibana.yml:

 got the error last line :FATAL  Error: Port 443 is already in use. Another instance of Kibana may be running!

I remembered that I install Wazuh, I change admin/password and update on filebeat.yml and kibana.yml also. But I figured out that I should keep default of elasticsearch.username and password of kbana.yml ( kibanaserver) instead of change alo so and it worked. 

Please help me to troubleshoot this issue above

Thanks and regards,

Tung

[victor....@wazuh.com]

Hello Tung,

I recommend you to use the wazuh-passwords-tool.sh script and follow this documentation page.

You can change your kibanaserver password easily using that script, you only need to run the following command:

bash wazuh-passwords-tool.sh -u kibanaserver -p new_password

And change it accordingly your kibana.yaml file.

Regarding changing the elasticsearch.username kibanaserver admin user, this is the default Open Distro user for Elasticsearch, so, maybe it could be a better idea to add a new user with the same permissions in order to make Kibana works.

If you have any doubt doing this process do not doubt to ask.

​

[Tung Ton]

Hi Victor,

Thanks for your response.

I tried to run  bash wazuh-passwords-tool.sh -u kibanaserver -p new_password but i encountered an error with " Creating backup... Error: The backup could not be created"

What's next step I should do to verify?

Thanks,

[victor....@wazuh.com]

Hello Tung,

It seems that the backup fails, this could be motivated by multiple reasons.

Please run the script using the verbose mode:

bash wazuh-passwords-tool.sh -u kibanaserver -p new_password -v

Also, we need more information about your environment, please, could you provide:

• Wazuh version

• Are you using Elasticsearch single-node or multi-node?

• Elasticsearch related logs: /var/log/elasticsearch/wazuh-cluster.log

​

[Tung Ton]

Hi Victor,

Thank for your response. I run  bash wazuh-passwords-tool.sh -u kibanaserver -p new_password -v , it appear error :

Creating backup...

mkdir: cannot create directory ‘/usr/share/elasticsearch/backup’: File exists

Open Distro Security Admin v7

Will connect to 127.0.0.1:9300

ERR: Seems there is no Elasticsearch running on 127.0.0.1:9300 - Will exit

Error: The backup could not be created

• Wazuh version -> 4.2

• Are you using Elasticsearch single-node or multi-node? -> single-node

• Elasticsearch related logs: /var/log/elasticsearch/wazuh-cluster.log -> -bash: /var/log/elasticsearch/wazuh-cluster.log: No such file or directory

Thanks,

[victor....@wazuh.com]

Hello Tung,

In order to change elasticsearch users' passwords, it is necessary to have elasticsearch running, so please restart elasticsearch and try again the command.

In case of failure, please, send back the output of the command, probably it would be related to the elasticsearch configuration, check this documentation page and ensure everything is well configured.

​

[Tung Ton]

Hi Victor,

Thank so much for your response. I restart elasticsearch and I checked that service in running. I run again  bash wazuh-passwords-tool.sh -u kibanaserver -p new_password -v get the error:

mkdir: cannot create directory ‘/usr/share/elasticsearch/backup’: File exists Open Distro Security Admin v7 Will connect to 127.0.0.1:9300 ... done 13:40:42.860 [elasticsearch[_client_][transport_worker][T#1]] ERROR com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLNelishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?] at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?] at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?] at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) ~[?:?] at sun.security.ssl.TransportContext.dispatch(TransportContext.java:202) ~[?:?] at sun.security.ssl.SSLTransport.decode(SSLTransport.java:171) ~[?:?] at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?] at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?] at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?] at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:637) ~[?:?] at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:282) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1372) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1. at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4. at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1. at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Fi at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4. at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4. at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Fina at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final] at java.lang.Thread.run(Thread.java:832) [?:?] ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information Trace: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{PWxlRtsVRMO3Qzy_kE-g0Q}{127.0.0.1}{127.0.0.1:9300}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:352) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:248) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:57) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:391) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:401) at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.execute(OpenDistroSecurityAdmin.java:524) at com.amazon.opendistroforelasticsearch.security.tools.OpenDistroSecurityAdmin.main(OpenDistroSecurityAdmin.java:157) Error: The backup could not be created But I figured out that I can access wazuh by Web UI. But in case I need to change admin password seem its still error? I install Wazuh all in one option in previous time.

Thanks,

[Tung Ton]

Hi Victor,

I figureded out that we haven't encountered "Kibana server is not ready yet" but when I type username/password Web UI always loading and pop up again with sign in portal. I tried restart elasticsearch and wazu-manager also, but it could access Web UI in short time. What's the issue that we encounterd?

Thanks so much for your support,

[victor....@wazuh.com]

Hello Tung,

It seems there are errors in your elasticsearch certificates.

If you are using a testing or development environment and you have an All-in-one deployment, it will be easier to reinstall it. You can do it using this command:

curl -so ~/unattended-installation.sh https://packages.wazuh.com/resources/4.2/open-distro/unattended-installation/unattended-installation.sh && bash ~/unattended-installation.sh --overwrite

In another case, I recommend removing your olds certificates and following these steps to generate new ones.

​

[Muhammad Samiul Haq]

This solved my issue , thank you. 

[Tung Ton]

Hi Victor,

Thanks so much for your response.

If I reinstall it, does it clear all wazuh configuration and log also? 

I tried to remove old certificate with your url instructions but I got the error when run bash ~/wazuh-cert-tool.sh :

 ERROR: The given information does not match with an IP or a DNS
I implemented All in one with single node and with below ~/instances.yml :

elasticsearch-nodes: - name: elasticsearch-nodes ip: - myIP   # Wazuh server nodes wazuh-servers: - name: wazuh-servers ip: - myIP   # Kibana node kibana: - name: kibana ip: - myIP

Thanks so much for your support,

Regards,

[victor....@wazuh.com]

• Regarding the backup:

  • If you want to keep your manager’s configuration you only have to keep the following list of files:

    • /var/ossec/etc/ossec.conf

    • /var/ossec/etc/local_internal_options.conf

    • /var/ossec/etc/decoders/local_decoder.xml

      • /var/ossec/etc/rules/local_rules.xml

      • /var/ossec/etc/client.keys

      • /var/ossec/api/configuration/api.yaml

  • If you want to keep the alerts, you could save the logs folder and restore it after the reinstall process.

• Regarding the creation of new certifications, I suggest running the following script. This will generate new certs and passwords. After that, it will restart your elasticsearch, filebeat and kibana.
  

# Remove existent certs
rm -rf /etc/elasticsearch/certs/
rm -rf /etc/kibana/certs/
rm -rf /etc/filebeat/certs/
rm /etc/elasticsearch/esnode-key.pem /etc/elasticsearch/esnode.pem /etc/elasticsearch/kirk-key.pem /etc/elasticsearch/kirk.pem /etc/elasticsearch/root-ca.pem -f

# Generate new certs
curl -so ~/wazuh-cert-tool.sh https://packages.wazuh.com/resources/4.2/open-distro/tools/certificate-utility/wazuh-cert-tool.sh
curl -so ~/instances.yml https://packages.wazuh.com/resources/4.2/open-distro/tools/certificate-utility/instances_aio.yml
bash ~/wazuh-cert-tool.sh

mkdir /etc/elasticsearch/certs/
mv ~/certs/elasticsearch* /etc/elasticsearch/certs/
mv ~/certs/admin* /etc/elasticsearch/certs/
cp ~/certs/root-ca* /etc/elasticsearch/certs/

# Restart elasticsearch
systemctl daemon-reload
systemctl enable elasticsearch
systemctl restart elasticsearch

# Apply new certifications
export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem

# Copy new certifications in filebeat
mkdir /etc/filebeat/certs
cp ~/certs/root-ca.pem /etc/filebeat/certs/
mv ~/certs/filebeat* /etc/filebeat/certs/

# Restart filebeat
systemctl daemon-reload
systemctl enable filebeat
systemctl start filebeat

# Copy new certifications in kibana
mkdir /etc/kibana/certs
cp ~/certs/root-ca.pem /etc/kibana/certs/
mv ~/certs/kibana* /etc/kibana/certs/
chown kibana:kibana /etc/kibana/certs/*

# Restart kibana
systemctl daemon-reload
systemctl enable kibana
systemctl start kibana

# Generate new passwords for your environment
curl -so wazuh-passwords-tool.sh https://packages.wazuh.com/resources/4.2/open-distro/tools/wazuh-passwords-tool.sh
bash wazuh-passwords-tool.sh -a

# Restart all services to apply new passwords
systemctl restart elasticsearch
systemctl restart kibana
systemctl restart filebeat

If you get any error running this script, send it back and we will help you.

Also, if you have any doubt do not hesitate to ask.

​

[Tung Ton]

Hi Victor,

What's a amazing your help! I will try in my situation and let you know the result soon.

Thanks so much for your great support.

Regards,

Tung