Dashboard auth issue after upgrade to 4.3

1,388 views
Skip to first unread message

Boriša

unread,
Jul 31, 2022, 9:37:45 AM7/31/22
to Wazuh mailing list
Hello Wazuh team, 

I have an issue after upgrading my docker deployment from 4.2 to 4.3. 
I followed this procedure and all went ok, but dashboard can't connect to indexer nodes.

root@dockerserver:/opt/wazuh-docker/multi-node# curl -k -u admin:******** https://127.0.0.1:9200/_cat/nodes?v

ip           heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name

192.168.48.2           73          99  41    6.04    7.61     8.56 dimr      -      wazuh3.indexer

192.168.48.4           73          99  41    6.04    7.61     8.56 dimr      *      wazuh1.indexer

192.168.48.5           72          99  41    6.04    7.61     8.56 dimr      -      wazuh2.indexer


These are error messages from dashboard and indexer respectively:

An OpenSearch Dashboards keystore already exists. Overwrite? [y/N] Created OpenSearch Dashboards keystore in /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore

Wazuh APP already configured

{"type":"log","@timestamp":"2022-07-31T06:40:35Z","tags":["info","plugins-service"],"pid":39,"message":"Plugin \"visTypeXy\" is disabled."}

{"type":"log","@timestamp":"2022-07-31T06:40:35Z","tags":["info","plugins-system"],"pid":39,"message":"Setting up [42] plugins: [alertingDashboards,usageCollection,opensearchDashboardsUsageCollection,opensearchDashboardsLegacy,mapsLegacy,share,expressions,data,home,console,apmOss,management,indexPatternManagement,advancedSettings,savedObjects,securityDashboards,indexManagementDashboards,opensearchUiShared,reportsDashboards,charts,embeddable,dashboard,visualizations,visTypeTimeseries,visTypeVega,visTypeTable,visTypeVislib,visTypeTimeline,timeline,visTypeTagcloud,visTypeMarkdown,visTypeMetric,tileMap,regionMap,inputControlVis,ganttChartDashboards,visualize,discover,wazuh,savedObjectsManagement,legacyExport,bfetch]"}

{"type":"log","@timestamp":"2022-07-31T06:40:36Z","tags":["info","savedobjects-service"],"pid":39,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}

{"type":"log","@timestamp":"2022-07-31T06:40:36Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:37Z","tags":["error","savedobjects-service"],"pid":39,"message":"Unable to retrieve version information from OpenSearch nodes."}

{"type":"log","@timestamp":"2022-07-31T06:40:39Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:41Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:44Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:46Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:49Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:51Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:54Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:56Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

{"type":"log","@timestamp":"2022-07-31T06:40:59Z","tags":["error","opensearch","data"],"pid":39,"message":"[ResponseError]: Response Error"}

============ error from indexer node

[2022-07-31T06:39:22,613][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

[2022-07-31T06:39:25,208][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48828

[2022-07-31T06:39:27,696][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

[2022-07-31T06:39:30,194][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48828

[2022-07-31T06:39:32,618][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

[2022-07-31T06:39:35,127][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48828

[2022-07-31T06:39:37,611][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

[2022-07-31T06:39:40,125][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48828

[2022-07-31T06:39:42,617][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

[2022-07-31T06:39:45,120][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48828

[2022-07-31T06:39:47,615][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

[2022-07-31T06:39:50,118][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48828

[2022-07-31T06:39:52,618][WARN ][o.o.s.a.BackendRegistry  ] [wazuh1.indexer] Authentication finally failed for kibanaserver from 172.28.0.7:48816

I tried to change pass for kibanaserver and all other users in /config/wazuh_indexer/internal_users.yml and then updated pass in dashboard keystore, but no luck:

  1. Execute the following command to get the hash of the password:
    docker exec -it multi-node_wazuh1.indexer_1 /bin/bash /usr/share/wazuh-indexer/plugins/opensearch-security/tools/hash.sh
  2. Change the password hash for the admin user in /config/wazuh_indexer/internal_users.yml
  3. Apply the change by executing the securityadmin.sh with the following command:
    docker exec -it multi-node_wazuh1.indexer_1 /bin/bash /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -cd /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig/ -nhnv -cacert /usr/share/wazuh-indexer/config/certs/root-ca.pem -cert /usr/share/wazuh-indexer/config/certs/admin.pem -key /usr/share/wazuh-indexer/config/certs/admin-key.pem -h wazuh1.indexer -icl
  4. Change the INDEXER_PASSWORD for the two Wazuh Nodes
  5. Re up the container with docker-compose up -d

wazuh-dashboard@wazuh:~$ ./bin/opensearch-dashboards-keystore --allow-root add opensearch.password

Setting opensearch.password already exists. Overwrite? [y/N] y

Enter value for opensearch.password: ****************


This is strange..looks like dashboard keystore is recreated every time dashboard container is restarted? 

keystore is not in volume, is my change persistent after I restart container?

An OpenSearch Dashboards keystore already exists. Overwrite? [y/N] Created OpenSearch Dashboards keystore in /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore


I even tried to set default credentials, as from our previous setup, but that also didn't work:

bash-4.2$ cat  /usr/share/kibana/config/kibana.yml

---

# Copyright 2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License").

# You may not use this file except in compliance with the License.

# A copy of the License is located at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# or in the "license" file accompanying this file. This file is distributed

# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either

# express or implied. See the License for the specific language governing

# permissions and limitations under the License.

# Description: 

# Default Kibana configuration from kibana-docker.

server.name: kibana

server.host: "0"

elasticsearch.hosts: https://elasticsearch:9200

elasticsearch.ssl.verificationMode: none

elasticsearch.username: kibanaserver

elasticsearch.password: kibanaserver

Thank you in advance, 

Boriša

Antonio David Gutiérrez

unread,
Aug 1, 2022, 5:43:27 AM8/1/22
to Wazuh mailing list
Hi Boriša,

Thank you for using Wazuh!

Are you using a Docker deployment with Kibana or Wazuh dashboard? I see references to both in your message.

If you are trying to use Wazuh dashboard, depens on the version you are using, when the Wazuh dashboard Docker container is started, it sets the `opensearch.username` and `opensearch.password` settings in the keystore as you can see here: https://github.com/wazuh/wazuh-docker/blob/v4.3.6/build-docker-images/wazuh-dashboard/config/entrypoint.sh#L11-L12 (for Wazuh dashboard v4.3.6 using the Docker image). It will use environment variables if defined or set a default value otherwise. You could define these environment variables for the Wazuh dashboard Docker container:
- DASHBOARD_USERNAME : set the username to use in Wazuh dashboard
- DASHBOARD_PASSWORD : set the password for the user

Check these environment variables are used depending on the Wazuh dashboard Docker image version that you are using.
Reply all
Reply to author
Forward
0 new messages