Wazuh Dashboard upgrade to 4.9.0-2 - "Wazuh dashboard server is not ready yet"

[Benny Moy]

Hi,

I upgraded from 4.8.2 to 4.9.0 and have the same issue as others, but all my services load. I am unsure how to troubleshoot the following error when I run journalctl -u wazuh-dashboard -iE "err|warn":

Sep 16 22:55:01 wlfd-wazuh-dashboard opensearch-dashboards[8393]: {"type":"log","@timestamp":"2024-09-16T22:55:01Z","tags":["error","opensearch","data"],"pid":8393,"message":"[ResponseError]: Response Error"}
Sep 16 22:55:03 wlfd-wazuh-dashboard opensearch-dashboards[8393]: {"type":"log","@timestamp":"2024-09-16T22:55:03Z","tags":["warning","savedobjects-service"],"pid":8393,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 16 22:55:03 wlfd-wazuh-dashboard opensearch-dashboards[9046]:   throw generateError(state, message);
Sep 16 22:55:03 wlfd-wazuh-dashboard opensearch-dashboards[9046]:     at generateError (/usr/share/wazuh-dashboard/node_modules/js-yaml/lib/js-yaml/loader.js:167:10)
Sep 16 22:55:03 wlfd-wazuh-dashboard opensearch-dashboards[9046]:     at throwError (/usr/share/wazuh-dashboard/node_modules/js-yaml/lib/js-yaml/loader.js:173:9)
Sep 16 22:55:03 wlfd-wazuh-dashboard opensearch-dashboards[9046]:       'uiSettings.overrides.defaultRoute: /app/wz-home\n' +

Thank you in advance!

Benny

[Lamya Imam]

Hello Benny Moy,

I would need you to ensure that you have updated the /etc/wazuh-dashboard/opensearch_dashboards.yml file with the backup of the previous configuration file. Make sure to update the IP address and the certificates in the dashboard configuration. [Screenshot added for reference]

After that I would need you to change the password for kibanaserver by using this command:

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<password>'

Then restart the Wazuh dashboard:

systemctl restart wazuh-dashboard

For distributed environments, update the password using the following command:

echo <kibanaserver>| /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch

Refer to: https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html#password-management

The issue is already open here:  

https://github.com/wazuh/wazuh/issues/25607

Also, check the Wazuh Dashboard certificates using this command:

ls -lrt /etc/wazuh-dashboard/certs/

Ensure that the path and file names match in:

/etc/wazuh-dashboard/opensearch_dashboards.yml

Let me know if this solves your issue!

[Benny Moy]

Hi Lamya,

Thank you for taking a look at my issue!  I have already done the following:

- Reset kibanaserver user password from one of my indexers using the command: /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver

- Updated keystore on dashboard:  echo "<generated_password from previous step>" | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

- Restarted wazuh-dashboard via systemctl restart wazuh-dashboard.service

- Confirmed that /etc/wazuh-dashboard/opensearch_dashboards.yml file contains the correct IPs for my indexers and path to certs:

#server.host: 0.0.0.0
server.host: "172.24.135.20"
server.port: 443
#opensearch.hosts: https://localhost:9200
opensearch.hosts:
  - https://172.24.135.11:9200
  - https://172.24.135.12:9200
  - https://172.24.135.13:9200
  - https://172.24.135.14:9200
  - https://172.24.135.15:9200

opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]

uiSettings.overrides.defaultRoute: /app/wz-home

ls -lrt /etc/wazuh-dashboard/certs/
total 12
-r-------- 1 wazuh-dashboard wazuh-dashboard 1204 Jul 25  2022 root-ca.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1704 Jul 25  2022 dashboard-key.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1237 Jul 25  2022 dashboard.pem

[Lamya Imam]

Hello Benny Moy,

Can you please check if you have the correct IP address at /usr/share/wazuh-dashboard/config/opensearch_dashboards.yml file of your dashboard server?

Also, can you please check if there is any indentation at /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml file?
It should look like this:
hosts:
  - default:
      url: https://127.0.0.1
      port: 55000
      username: wazuh-wui
      password: wazuh-wui
      run_as: false

Please share the output of the following log:
cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

For further troubleshooting reference, please check out this documentation:
https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html#troubleshooting

Let me know the update on the issue!

[Benny Moy]

Hi Lamya,

Confirmed that IPs are correct for my 5 indexers:

cat /usr/share/wazuh-dashboard/config/opensearch_dashboards.yml

#server.host: 0.0.0.0
server.host: "172.24.135.20"
server.port: 443
#opensearch.hosts: https://localhost:9200
opensearch.hosts:
  - https://172.24.135.11:9200
  - https://172.24.135.12:9200
  - https://172.24.135.13:9200
  - https://172.24.135.14:9200
  - https://172.24.135.15:9200

opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home

There are a lot when I run the following but here are the last lines:

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://172.24.135.21:55000/cluster/server-3/stats/analysisd"},"message":"Request failed with status code 500","stack":"AxiosError: Request failed with status code 500\n    at settle (/usr/share/wazuh-dashboard/plugins/wazuh/node_modules/axios/lib/core/settle.js:19:12)\n    at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/node_modules/axios/lib/adapters/http.js:585:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1359:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"},"date":"2024-08-20T16:18:20.827Z","level":"info","location":"Cron-scheduler"}
{"date":"2024-08-22T02:00:00.466Z","level":"error","location":"monitoring:cronTask","message":"Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
{"data":{"config":{"data":"{}","method":"get","params":{},"url":"https://172.24.135.21:55000/cluster/server-1/stats/remoted"},"message":"Request failed with status code 400","stack":"AxiosError: Request failed with status code 400\n    at settle (/usr/share/wazuh-dashboard/plugins/wazuh/node_modules/axios/lib/core/settle.js:19:12)\n    at IncomingMessage.handleStreamEnd (/usr/share/wazuh-dashboard/plugins/wazuh/node_modules/axios/lib/adapters/http.js:585:11)\n    at IncomingMessage.emit (node:events:525:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1359:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"},"date":"2024-08-22T09:20:00.282Z","level":"info","location":"Cron-scheduler"}
{"date":"2024-08-31T01:30:10.698Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default. Error: Request failed with status code 500"}
{"date":"2024-08-31T01:30:10.698Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status code 500"}

[Lamya Imam]

Hello Benny Moy,

Can you check the status of the indexers individually if they are up and running properly?
# systemctl status wazuh-indexer

I would suggest you to make a curl request from the dashboard node to the indexer node with kibanaserver user, and check if it can connect with every indexer node:
# curl -k -u kibanaserver:<pass> https://<Indexer_IP>:9200

Please share the findings with us for further analysis! 

[Benny Moy]

Hi Lamya,

I checked all 5 and they are online and active.  Here's the systemctl status result for one of them:

systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2024-09-16 13:31:32 UTC; 1 weeks 0 days ago
       Docs: https://documentation.wazuh.com
   Main PID: 963 (java)
      Tasks: 128 (limit: 38374)
     Memory: 27.3G
     CGroup: /system.slice/wazuh-indexer.service
             └─963 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.>

Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at org.opensearch.performanceanalyzer.commons.collectors.MetricStatus.serialize(MetricStatus.java:19)
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector.collectMetrics(CacheConfigMetricsCollector.java:>
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at org.opensearch.performanceanalyzer.commons.collectors.PerformanceAnalyzerMetricsCollector.lambda$run$0(PerformanceAnalyzer>
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at org.opensearch.performanceanalyzer.commons.util.Util.lambda$invokePrivileged$1(Util.java:57)
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at org.opensearch.performanceanalyzer.commons.util.Util.invokePrivileged(Util.java:53)
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at org.opensearch.performanceanalyzer.commons.collectors.PerformanceAnalyzerMetricsCollector.run(PerformanceAnalyzerMetricsCo>
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Sep 23 00:00:02 wlfd-wazuh-indexer01 systemd-entrypoint[963]:         at java.base/java.lang.Thread.run(Thread.java:1583)

When I run curl -ku kibanaserver https://<IP>:9200 and enter the password for each, i get a blank return.  This happens on all 5 indexers.

Benny

[Lamya Imam]

Hello Benny Moy,

It seems like there might be a connection or authentication issue. Please ensure that the firewall rules are set up correctly for communication.

Also, could you share the output of the following command? It might help us investigate the problem further:
# curl -ku kibanaserver https://<IP>:9200

I was expecting an error message about the connection refused error from the output of the command, like this:
curl: (7) Failed to connect to 127.0.0.2 port 9200 after 0 ms: Connection refused

Let me know!

[Benny Moy]

Hi Lamya,

I confirmed that I can telnet to each index's IP on port 9200 so I've ruled out network connectivity issues.  For example, here is the curl command to indexer-01:

root@wlfd-wazuh-dashboard:~# curl -ku kibanaserver :'<password_redacted>' https://172.24.135.11:9200
root@wlfd-wazuh-dashboard:~#

The odd thing is that when i browse to https://172.24.135.11:9200 and try to login as kibanauser, it doesn't seem like the password is taking:

Benny

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/D6DiO9ixrHA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/55dc5d87-2de4-44b9-8507-cff3e147c160n%40googlegroups.com.

[Lamya Imam]

Hi  Benny Moy,

Thank you for providing me the information. Now, we can conclude that the issue is related to authentication.

I would suggest you to change the password for the kibanaserver user by using this command:
# bash /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p Secr3tP4ssw*rd
You can change the password as you like.

After that, on your Wazuh dashboard node, run the following command to update the kibanaserver password in the Wazuh dashboard keystore. Replace <KIBANASERVER_PASSWORD> with the new password:
# echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

Restart the Wazuh dashboard to apply the changes:
# systemctl restart wazuh-dashboard

For further reference, please check out this documentation:
https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

Let me know if this solved your issue!

[Benny Moy]

Hi Lamya,

I ended up fixing this rerunning the following from an indexer:

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver

and then importing into the dashboard host:

echo '<password_redacted>' | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password (edited) 

Thank you very much for your help!