botched 4.8.2 upgrade

[Fred Bret-Mounet]

Hello,

I accidentally upgraded from 4.8.1 to 4.8.2 without going through the upgrade process.

Trying to retroactively apply the steps, I have ended with:
- dashboard refusing https connections

- systemctl status wazuh-dashboard

● wazuh-dashboard.service - wazuh-dashboard

   Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled)

   Active: failed (Result: exit-code) since Tue 2024-08-27 21:19:56 UTC; 5min ago

  Process: 6558 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (code=exited, status=1/FAILURE)

 Main PID: 6558 (code=exited, status=1/FAILURE)

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: {"type":"log","@timestamp":"2024-08-27T21:19:56Z","tags":["info","plugins-service"],"pid":6558,"message":"Plugin \"visTypeXy\" is disabled."}

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: {"type":"log","@timestamp":"2024-08-27T21:19:56Z","tags":["fatal","root"],"pid":6558,"message":"Error: ENOENT: no such file or dir...(/usr/share/

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: at Object.openSync (fs.js:498:3)

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: at readFileSync (fs.js:394:35)

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: at readFile (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:181:31)

Aug 27 21:19:56 redacted.compute.internal opensearch-dashboards[6558]: at new SslConfig (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:131:18)

Aug 27 21:19:56 redacted.compute.internal systemd[1]: wazuh-dashboard.service: main process exited, code=exited, status=1/FAILURE

Aug 27 21:19:56 redacted.compute.internal systemd[1]: Unit wazuh-dashboard.service entered failed state.

Aug 27 21:19:56 redacted.compute.internal systemd[1]: wazuh-dashboard.service failed.

- cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

[2024-08-27T21:11:03,187][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms7705m, -Xmx7705m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-436197384683995072, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=4041211904, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]

[2024-08-27T21:11:13,586][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes

[2024-08-27T21:11:13,644][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.

[2024-08-27T21:11:13,645][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.

[2024-08-27T21:11:15,028][WARN ][o.o.s.p.SQLPlugin        ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information

[2024-08-27T21:11:16,399][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually

[2024-08-27T21:11:18,001][ERROR][o.o.s.t.SecurityRequestHandler] [node-1] OpenSearchException[Transport client authentication no longer supported.]

[2024-08-27T21:11:18,004][ERROR][o.o.s.t.SecurityRequestHandler] [node-1] OpenSearchException[Transport client authentication no longer supported.]

[2024-08-27T21:11:18,011][WARN ][o.o.d.HandshakingTransportAddressConnector] [node-1] handshake failed for [connectToRemoteMasterNode[[::1]:9300]]

[2024-08-27T21:11:18,011][WARN ][o.o.d.HandshakingTransportAddressConnector] [node-1] handshake failed for [connectToRemoteMasterNode[127.0.0.1:9300]]

[2024-08-27T21:11:18,155][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.

[2024-08-27T21:11:18,501][WARN ][o.o.o.i.ObservabilityIndex] [node-1] message: index [.opensearch-observability/CnLN75CdS1SP-vxgd3Kafg] already exists

[2024-08-27T21:11:18,505][WARN ][o.o.s.SecurityAnalyticsPlugin] [node-1] Failed to initialize LogType config index and builtin log types

[2024-08-27T21:11:21,677][WARN ][c.a.d.a.h.s.Saml2SettingsProvider] [node-1] The IdP does not provide a Single Logout Service. In order to ensure that users have to re-enter their password after logging out, OpenSearch Security will issue all SAML authentication requests with a mandatory password input (ForceAuthn=true)

[2024-08-27T21:15:10,250][WARN ][o.o.s.a.BackendRegistry  ] [node-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

[2024-08-27T21:15:25,610][WARN ][o.o.s.a.BackendRegistry  ] [node-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

[2024-08-27T21:15:34,319][WARN ][o.o.s.a.BackendRegistry  ] [node-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

[2024-08-27T21:15:39,750][WARN ][o.o.s.a.BackendRegistry  ] [node-1] No 'Authorization' header, send 401 and 'WWW-Authenticate Basic'

What did I miss?!

-Fred

[Hatem]

Hi Fred

Can you share the right step?

> --
> You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9be07335-84d6-4a11-8327-186d53c79b3dn%40googlegroups.com.



--
BR
Hatem Enaami

[Fred Bret-Mounet]

Here's the upgrade guide:

https://documentation.wazuh.com/4.8/upgrade-guide/upgrading-central-components.html

[Felix Bocco]

Hello Fred,

According to the provided logs, we found this error message:
Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'

First of all, you should check if the corresponding certificates exist in that path. Secondly, check that they have the correct permissions (highlighted):
[rootxxxxx ~]# ll /etc/wazuh-dashboard/certs/
total 12
-r--------. 1 wazuh-dashboard wazuh-dashboard size Month Time dashboard-key.pem
-r--------. 1 wazuh-dashboard wazuh-dashboard size Month Time dashboard.pem
-r--------. 1 wazuh-dashboard wazuh-dashboard size Month Time  root-ca.pem

Also, please check that you have the corresponding users configured as expected:

On all your Wazuh server nodes, run the following command to update the admin password in the Filebeat keystore and in the ossec.conf file for the Wazuh server. Replace <ADMIN_PASSWORD> with the admin password.

# echo <ADMIN_PASSWORD> | filebeat keystore add password --stdin --force # sed -i 's/<password>.*<\/password>/<password><ADMIN_PASSWORD><\/password>/g' /var/ossec/etc/ossec.conf

Restart Filebeat and the Wazuh server to apply the change:
systemctl restart filebeat

systemctl restart wazuh-manager

On your Wazuh dashboard node, run the following command to update the kibanaserver password in the Wazuh dashboard keystore. Replace <KIBANASERVER_PASSWORD> with the kibanaserver password.

# echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

Update the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml configuration file with the wazuh-wui password.

hosts: - default: url: https://127.0.0.1 port: 55000 username: wazuh-wui password: "<wazuh-wui-password>" run_as: false

Restart the Wazuh dashboard to apply the changes.

systemctl restart wazuh-dashboard

After performing the previous steps, check again if it works or what error prompts.

Let us know how it goes.

[Fred Bret-Mounet]

Thanks for the pointer. Fixing the config to point to the right cert was all that was needed.

Turns out the upgrade process overwrote my custom letsencrypt config and saml auth... :-(