Wazuh Dashboard not able to start after upgrade

[Andrehens Chicfici]

Hey,

My wazuh upgraded automatically over the weekend and now the dashboard is broken.

journalctl -u wazuh-dashboard.service | grep -i -E "error|critical|warning|fatal"

tells me:

Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]: {"type":"log","@timestamp":"2024-09-09T11:35:55Z","tags":["fatal","root"],"pid":3009540,"message":"Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n    at Object.openSync (node:fs:596:3)\n    at readFileSync (node:fs:464:35)\n    at readFile (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:170:31)\n    at new SslConfig (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:130:18)\n    at new HttpConfig (/usr/share/wazuh-dashboard/src/core/server/http/http_config.js:174:16)\n    at MapSubscriber.project (/usr/share/wazuh-dashboard/src/core/server/http/http_service.js:67:177)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at CombineLatestSubscriber.notifyNext (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/observable/combineLatest.js:97:34)\n    at InnerSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/InnerSubscriber.js:28:21)\n    at InnerSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:30:22)\n    at MapSubscriber.shareReplayOperation (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/shareReplay.js:53:32) {\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n}"}
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]: Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'

ls -lrt /etc/wazuh-dashboard/certs/

gives me:

total 12
-rwxrwxrwx 1 wazuh-dashboard wazuh-dashboard 1204 Feb 20  2024 root-ca.pem
-rwxrwxrwx 1 wazuh-dashboard wazuh-dashboard 1289 Feb 20  2024 wazuh-dashboard.pem
-rwxrwxrwx 1 wazuh-dashboard wazuh-dashboard 1704 Feb 20  2024 wazuh-dashboard-key.pem

status wazuh-dashboard.service

tells me

× wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2024-09-09 13:35:55 CEST; 27min ago
   Main PID: 3009540 (code=exited, status=1/FAILURE)
        CPU: 3.641s

Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at ReplaySubject._subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/ReplaySubject.js:80:28)
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at ReplaySubject.Observable._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:44:25)
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at ReplaySubject.Subject._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subject.js:102:51)
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at ReplaySubject.Observable.subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:30:22)
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]:     at MapSubscriber.shareReplayOperation (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/shareReplay.js:53:32)
Sep 09 13:35:55 wazuh systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 13:35:55 wazuh systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.
Sep 09 13:35:55 wazuh systemd[1]: wazuh-dashboard.service: Consumed 3.641s CPU time.

What do I need to do, to get my dashboard back?

cheers

chic

[hasitha.u...@wazuh.com]

Hello Andrehens,

As part of Wazuh 4.9.0, we published Wazuh dashboard revision 1 packages:
wazuh-dashboard-4.9.0-1.deb
wazuh-dashboard-4.9.0-1.rpm
https://documentation.wazuh.com/current/installation-guide/packages-list.html#wazuh-indexer

These packages introduced a change in the keystore location to /usr/share/wazuh-dashboard/config, which caused issues when upgrading from previous versions, leading to the "Dashboard is not ready" error. Additionally, the kibanaserver user displayed failed authentication messages in the indexer logs.
To resolve this, we have rolled out Wazuh dashboard revision 2 of these packages:
wazuh-dashboard-4.9.0-2.deb
wazuh-dashboard-4.9.0-2.rpm
https://documentation.wazuh.com/current/installation-guide/packages-list.html#wazuh-dashboard

Wazuh 4.x repositories are available again.
Important:  If you upgraded using wazuh-dashboard-4.9.0-1, follow these steps:
Install the new Wazuh dashboard revision 2 packages.

Backup your current keystore:
cp /etc/wazuh-dashboard/opensearch_dashboard.keystore /etc/wazuh-dashboard/opensearch_dashboard.keystore.bak

3. Backup the keystore from the new location:
cp /usr/share/wazuh-dashboard/config/opensearch_dashboard.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboard.keystore.bak

4. Move the keystore to the correct location:
mv /usr/share/wazuh-dashboard/config/opensearch_dashboard.keystore /etc/wazuh-dashboard/opensearch_dashboard.keystore

5. Restart the Wazuh dashboard.
systemctl restart wazuh-dashboard

Additionally Please check the /etc/wazuh-dashboard/opensearch_dashboards.yml file you have include correct names of the certificate files.
server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"

Let me know if this helps.

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

Hey Hasita,

I followed your steps but still get:

systemctl status wazuh-dashboard.service

× wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)

     Active: failed (Result: exit-code) since Mon 2024-09-09 15:33:14 CEST; 1min 11s ago
    Process: 3024728 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (code=exited, status=1/FAILURE)
   Main PID: 3024728 (code=exited, status=1/FAILURE)
        CPU: 3.712s

Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at ReplaySubject._subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/ReplaySubject.js:80:28)
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at ReplaySubject.Observable._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:44:25)
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at ReplaySubject.Subject._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subject.js:102:51)
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at ReplaySubject.Observable.subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:30:22)
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]:     at MapSubscriber.shareReplayOperation (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/shareReplay.js:53:32)
Sep 09 15:33:14 wazuh systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE
Sep 09 15:33:14 wazuh systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.
Sep 09 15:33:14 wazuh systemd[1]: wazuh-dashboard.service: Consumed 3.712s CPU time.

[hasitha.u...@wazuh.com]

Hello Andrehens,

It would be great, if you could provide the output of the following commands so I can take a closer look:
cat /etc/wazuh-dashboard/opensearch_dashboards.yml ls -lh /etc/wazuh-dashboard/opensearch_dashboards.yml ls -lh /etc/wazuh-dashboard/certs

journalctl -u wazuh-dashboard.service | grep -i -E "error|critical|warning|fatal"

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

For further troubleshooting, you can follow this guide.
Ref: https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html

Let me know the update. Thanks!

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

cat /etc/wazuh-dashboard/opensearch_dashboards.yml

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://localhost:9200
opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home

ls -lh /etc/wazuh-dashboard/opensearch_dashboards.yml

-rw-r----- 1 wazuh-dashboard wazuh-dashboard 634 May  5  2023 /etc/wazuh-dashboard/opensearch_dashboards.yml

ls -lh /etc/wazuh-dashboard/certs

total 12K
-rwxrwxrwx 1 wazuh-dashboard wazuh-dashboard 1.2K Feb 20  2024 root-ca.pem
-rwxrwxrwx 1 wazuh-dashboard wazuh-dashboard 1.7K Feb 20  2024 wazuh-dashboard-key.pem
-rwxrwxrwx 1 wazuh-dashboard wazuh-dashboard 1.3K Feb 20  2024 wazuh-dashboard.pem

 journalctl -u wazuh-dashboard.service | grep -i -E "error|critical|warning|fatal"

Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]: {"type":"log","@timestamp":"2024-09-09T11:35:55Z","tags":["fatal","root"],"pid":3009540,"message":"Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n    at Object.openSync (node:fs:596:3)\n    at readFileSync (node:fs:464:35)\n    at readFile (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:170:31)\n    at new SslConfig (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:130:18)\n    at new HttpConfig (/usr/share/wazuh-dashboard/src/core/server/http/http_config.js:174:16)\n    at MapSubscriber.project (/usr/share/wazuh-dashboard/src/core/server/http/http_service.js:67:177)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at CombineLatestSubscriber.notifyNext (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/observable/combineLatest.js:97:34)\n    at InnerSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/InnerSubscriber.js:28:21)\n    at InnerSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:30:22)\n    at MapSubscriber.shareReplayOperation (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/shareReplay.js:53:32) {\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n}"}
Sep 09 13:35:55 wazuh opensearch-dashboards[3009540]: Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'

Sep 09 14:03:49 wazuh opensearch-dashboards[3014802]: {"type":"log","@timestamp":"2024-09-09T12:03:49Z","tags":["fatal","root"],"pid":3014802,"message":"Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n    at Object.openSync (node:fs:596:3)\n    at readFileSync (node:fs:464:35)\n    at readFile (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:170:31)\n    at new SslConfig (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:130:18)\n    at new HttpConfig (/usr/share/wazuh-dashboard/src/core/server/http/http_config.js:174:16)\n    at MapSubscriber.project (/usr/share/wazuh-dashboard/src/core/server/http/http_service.js:67:177)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at CombineLatestSubscriber.notifyNext (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/observable/combineLatest.js:97:34)\n    at InnerSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/InnerSubscriber.js:28:21)\n    at InnerSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:30:22)\n    at MapSubscriber.shareReplayOperation (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/shareReplay.js:53:32) {\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n}"}
Sep 09 14:03:49 wazuh opensearch-dashboards[3014802]: Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]: {"type":"log","@timestamp":"2024-09-09T13:33:14Z","tags":["fatal","root"],"pid":3024728,"message":"Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n    at Object.openSync (node:fs:596:3)\n    at readFileSync (node:fs:464:35)\n    at readFile (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:170:31)\n    at new SslConfig (/usr/share/wazuh-dashboard/src/core/server/http/ssl_config.js:130:18)\n    at new HttpConfig (/usr/share/wazuh-dashboard/src/core/server/http/http_config.js:174:16)\n    at MapSubscriber.project (/usr/share/wazuh-dashboard/src/core/server/http/http_service.js:67:177)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:49:35)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at CombineLatestSubscriber.notifyNext (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/observable/combineLatest.js:97:34)\n    at InnerSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/InnerSubscriber.js:28:21)\n    at InnerSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at DistinctUntilChangedSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/distinctUntilChanged.js:69:30)\n    at DistinctUntilChangedSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at MapSubscriber._next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/map.js:55:26)\n    at MapSubscriber.Subscriber.next (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subscriber.js:66:18)\n    at ReplaySubject._subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/ReplaySubject.js:80:28)\n    at ReplaySubject.Observable._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:44:25)\n    at ReplaySubject.Subject._trySubscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Subject.js:102:51)\n    at ReplaySubject.Observable.subscribe (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/Observable.js:30:22)\n    at MapSubscriber.shareReplayOperation (/usr/share/wazuh-dashboard/node_modules/rxjs/internal/operators/shareReplay.js:53:32) {\n  errno: -2,\n  syscall: 'open',\n  code: 'ENOENT',\n  path: '/etc/wazuh-dashboard/certs/dashboard-key.pem'\n}"}
Sep 09 15:33:14 wazuh opensearch-dashboards[3024728]: Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'

cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

{"date":"2024-09-03T14:17:38.078Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 10.122.0.253:55000"}
{"date":"2024-09-03T14:17:40.112Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 10.122.0.253:55000"}
{"date":"2024-09-03T14:17:42.127Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 10.122.0.253:55000"}
{"data":{"dapi_errors":{"node01":{"error":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"detail":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2024-09-03T14:17:44.207Z","level":"error","location":"wazuh-api:makeRequest"}
{"data":{"dapi_errors":{"node01":{"error":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"detail":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2024-09-03T14:17:46.235Z","level":"error","location":"wazuh-api:makeRequest"}
{"data":{"dapi_errors":{"node01":{"error":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"detail":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2024-09-03T14:17:48.258Z","level":"error","location":"wazuh-api:makeRequest"}
{"data":{"dapi_errors":{"node01":{"error":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)"}},"detail":"Some Wazuh daemons are not ready yet in node \"node01\" (wazuh-modulesd->restarting, wazuh-analysisd->restarting, wazuh-execd->restarting, wazuh-db->restarting, wazuh-remoted->restarting)","error":1017,"title":"Bad Request"},"date":"2024-09-03T14:17:50.296Z","level":"error","location":"wazuh-api:makeRequest"}

Cheers chic

[Dmitry Mikheev]

adjust to your files name

server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"

server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"

and restart

[Andrehens Chicfici]

@Dmitry Mikheev

This "helped" a bit.

/etc/wazuh-dashboard/certs# systemctl status wazuh-dashboard.service

● wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)

     Active: active (running) since Tue 2024-09-10 11:16:00 CEST; 5s ago
   Main PID: 3158371 (node)
      Tasks: 11 (limit: 57669)
     Memory: 310.5M
        CPU: 4.149s
     CGroup: /system.slice/wazuh-dashboard.service
             └─3158371 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist

Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: {"type":"log","@timestamp":"2024-09-10T09:16:05Z","tags":["info","savedobjects-service"],"pid":3158371,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: {"type":"log","@timestamp":"2024-09-10T09:16:05Z","tags":["error","opensearch","data"],"pid":3158371,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 10 11:16:05 wazuh opensearch-dashboards[3158371]: {"type":"log","@timestamp":"2024-09-10T09:16:05Z","tags":["error","savedobjects-service"],"pid":3158371,"message":"Unable to retrieve version information from OpenSearch nodes."}

Now I get:  "Wazuh dashboard server is not ready yet" when I open the dashboard.

Cheers

chic

[Dmitry Mikheev]

curl -u admin:paswd -k https://127.0.0.1:9200/_cat/indices

are all the indices green?

curl -u admin:paswd -k https://127.0.0.1:9200/_cluster/health?pretty

"unassigned_shards" : 0 or other?

[Andrehens Chicfici]

@Dmitry Mikheev

root@wazuh:/etc/wazuh-dashboard/certs# curl -u admin:paswd -k https://127.0.0.1:9200/_cat/indices
curl: (7) Failed to connect to 127.0.0.1 port 9200 after 0 ms: Connection refused
root@wazuh:/etc/wazuh-dashboard/certs# curl -u admin:paswd -k https://127.0.0.1:9200/_cluster/health?pretty

curl: (7) Failed to connect to 127.0.0.1 port 9200 after 0 ms: Connection refused

which admin:passwd credential should I use? I tried admin:passwd and the regular admin:myadminpassword combination...

Cheers

chic

[Dmitry Mikheev]

they are created when installing the system.

they can be changed using /usr/share/wazuh-indexer/plugins/opensearch-security/tools/ wazuh-passwords-tool.sh 

BUT read how to change only for dashboard

[Andrehens Chicfici]

@Dmitry Mikheev

I do have the passwords for:

admin
kibanaserver
kibanaro
logstash
readall
snapshotrestore
wazuh-wui

[Dmitry Mikheev]

you can try to change it
https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

BUT I see that you are using logstash. I don't use it. Maybe it has some features...

[Andrehens Chicfici]

@Dmitry Mikheev

Yeah, I do have all the passwords. I just wanted which ones to use with your commands.

If I use them with my regular admin account, in both cases I get:

curl: (7) Failed to connect to 127.0.0.1 port 9200 after 0 ms: Connection refused

Cheers

chic

[hasitha.u...@wazuh.com]

Hello  Andrehens,

To resolving your issue, you need to update the password for the kibanaserver user, simply changing the <KIBANASERVER_PASSWORD> in the Wazuh Dashboard keystore with the old password might not work. Here's what to do:

Change the password.
If you use the tool in an all-in-one deployment, it automatically updates the passwords where necessary. If you use it in a distributed environment, depending on the user whose password you change, you may have to update the password on other components. See Changing the Passwords in a distributed environment for more details.  https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html#passwords-distributed

To change the password, use this command:
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>'
The password must have a length between 8 and 64 characters and contain at least one upper and lower case letter, a number and a symbol(.*+?-)

For distributed environments, update the password using the following command:
echo <kibanaserver>| /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearc
Refer to https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html#password-management

And also in the /etc/wazuh-dashboard/opensearch_dashboards.yml update the opensearch.hosts: https://localhost:9200 as your indexer IP or if its all in one use like this opensearch.hosts: https://127.0.0.1:9200

and then restart the dashboard by using systemctl restart wazuh-dashboard.

Let me know if this helps or if you need further assistance!

[Andrehens Chicfici]

@ hasitha.upekshitha

I changed the password like you recommended:

root@wazuh:~# /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p mypassword

11/09/2024 09:12:11 INFO: Updating the internal users.
11/09/2024 09:12:11 ERROR: The backup could not be created

My opensearch_dashboards.yml looks like

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://localhost:9200
opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true

server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem>
uiSettings.overrides.defaultRoute: /app/wz-home

When I restart wazuh-dashboard now I still get the message 'Wazuh dashboard server is not ready yet'

cheers

chic

[hasitha.u...@wazuh.com]

Hello Andrehens,

It seems that you haven't updated the value of opensearch.hosts from https://localhost:9200 to the IP address of the Wazuh indexer. You should replace it with something like https://127.0.0.1:9200 in opensearch_dashboards.yml.

After making this change, restart the Wazuh dashboard using the following command:
systemctl restart wazuh-dashboard

Once it's done, please let me know the outcome.

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

kay - I also changed the opensearch_dashboards.yml:

server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://999.999.999.999:9200

opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/wazuh-dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]

uiSettings.overrides.defaultRoute: /app/wz-home

Still: "Wazuh dashboard server is not ready yet"

 systemctl status wazuh-dashboard.service

 wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)

     Active: active (running) since Wed 2024-09-11 11:07:29 CEST; 10s ago
   Main PID: 3322092 (node)
      Tasks: 11 (limit: 57669)
     Memory: 257.6M
        CPU: 4.171s
     CGroup: /system.slice/wazuh-dashboard.service
             └─3322092 /usr/share/wazuh-dashboard/node/bin/node /usr/share/wazuh-dashboard/src/cli/dist

Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: [agentkeepalive:deprecated] options.freeSocketKeepAliveTimeout is deprecated, please use options.freeSocketTimeout instead
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: {"type":"log","@timestamp":"2024-09-11T09:07:33Z","tags":["info","savedobjects-service"],"pid":3322092,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: {"type":"log","@timestamp":"2024-09-11T09:07:33Z","tags":["error","opensearch","data"],"pid":3322092,"message":"[ConnectionError]: connect ECONNREFUSED  999.999.999.999 :9200"}
Sep 11 11:07:33 wazuh opensearch-dashboards[3322092]: {"type":"log","@timestamp":"2024-09-11T09:07:33Z","tags":["error","savedobjects-service"],"pid":3322092,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 11 11:07:35 wazuh opensearch-dashboards[3322092]: {"type":"log","@timestamp":"2024-09-11T09:07:35Z","tags":["error","opensearch","data"],"pid":3322092,"message":"[ConnectionError]: connect ECONNREFUSED 999.999.999.999 :9200"}
Sep 11 11:07:38 wazuh opensearch-dashboards[3322092]: {"type":"log","@timestamp":"2024-09-11T09:07:38Z","tags":["error","opensearch","data"],"pid":3322092,"message":"[ConnectionError]: connect ECONNREFUSED  999.999.999.999 :9200"}

[hasitha.u...@wazuh.com]

Hello Andrehens,

After reviewing your issue, I noticed that the certificate files in /etc/wazuh-dashboard/certs have incorrect permissions. Please update the permissions with the following command:

chmod -R 400 /etc/wazuh-dashboard/certs/*

Then, restart the Wazuh dashboard:

systemctl restart wazuh-dashboard

If the issue persists, let’s check if the indexer is running. Please provide the status of the Wazuh indexer:

systemctl status wazuh-indexer -l

If it’s not running, try restarting the indexer and share the logs:
systemctl restart wazuh-indexer
cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

Also, verify if there are any communication errors by running the below command.
filebeat test output.

Additionally, share the output of this command to check OpenSearch:

curl -k -u admin:<Password> https://127.0.0.1:9200

If you're using an all-in-one deployment, the password updates automatically. But in a distributed deployment, you’ll need to update the kibanaserver password in the Wazuh dashboard keystore by running this command:

echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

Let me know once you’ve followed these steps!

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

Hey @Hasita

I changed the permissions for the certs like you said:

I still get the ERROR: Connection Refused.

The wazuh-indexer is not running. 

Restarting immediately errrors out with:

"Failed with result 'exit-code'.

Failed to start wazuh-indexer."

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

Had the last errors on 06.09.

[2024-09-06-T03:41:26,197] [ERROR] [o.o.a.a.AlertIndices     ]  [node-1] info deleteOldIndices

[2024-09-06-T03:41:26,198] [ERROR] [o.o.a.a.AlertIndices     ]  [node-1] info deleteOldIndices

[2024-09-06-T05:01:37,826] [ERROR] [o.o.i.i.ManagedIndexRunner     ]  [node-1] Failed to save ManagedIndexMetaData for [index=wazuh-alerts-4.x-2024.08.22]

[2024-09-06-T05:01:37,855] [ERROR] [o.o.i.i.ManagedIndexRunner     ]  [node-1] Failed to update ManagedIndexMetaData after executing the Step : attempt_transition_step

[2024-09-06-T05:01:38,704] [WARN] [o.o.t.TransportService    ]  [node-1] Transport response handler not found of id [12170236]

[2024-09-06-T05:01:39,071] [ERROR] [o.o.j.s.u.LockService    ]   [node-1] Exception occured finding lock

[2024-09-06-T05:01:39,188] [ERROR] [o.o.j.s.u.LockService    ]   [node-1] Exception occured finding lock

[2024-09-06-T05:01:39,407] [ERROR] [o.o.j.s.u.LockService    ]   [node-1] Exception occured finding lock

[2024-09-06-T05:01:39,487] [ERROR] [o.o.b.OpenSearchUncaughtExceptionHandler]   [node-1] uncaught exception in thread [DefaultDispatcher-worker-5]

[2024-09-06-T05:01:39,487] [ERROR] [o.o.b.OpenSearchUncaughtExceptionHandler]   [node-1] uncaught exception in thread [DefaultDispatcher-worker-3]

[2024-09-06-T05:01:39,487] [ERROR] [o.o.b.OpenSearchUncaughtExceptionHandler]   [node-1] uncaught exception in thread [DefaultDispatcher-worker-2]

'filebeat test output' tells us:

elasticsearch: https://127.0.0.1:9200...

parse url... OK

connection...

parse host... OK

dns lookup... OK

adddresses: 127.0.0.1

dial up... ERROR dial tcp 127.0.0.1:9200: connect: connection refused

'curl -k -u admin:<PASSWORD> https://127.0.0.1:9200' gives:

curl:(7) Failed to connect to 127.0.0.1 port 9200 after 0 ms: Connection refused

[hasitha.u...@wazuh.com]

Hello  Andrehens,

We need to make sure the Wazuh indexer is up and running.
Restart the Wazuh Indexer:
systemctl restart wazuh-indexer

Then check its status:
systemctl status wazuh-indexer -l

Ensure the correct IP is used in the indexer and Filebeat configuration files.
Check the indexer configuration:
nano /etc/wazuh-indexer/opensearch.yml
Verify that the network.host has the correct indexer IP:
network.host: "indexer-IP"

Check Filebeat configuration:
nano /etc/filebeat/filebeat.yml
Confirm the correct indexer IP under output.elasticsearch.hosts:
output.elasticsearch.hosts:
    - <indexer-IP>:9200

Check IP in the Certificate: You can check the IP in the certificate using the following command:
openssl x509 -in /etc/wazuh-indexer/certs/wazuh-indexer.pem -text -noout

Restart Services: After verifying the IP, restart both services:
systemctl restart filebeat
systemctl restart wazuh-indexer

Test Filebeat Output: Ensure Filebeat is functioning properly by testing the output:
filebeat test output

Let me know how it goes!

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

systemctl restart wazuh-indexer

Job for wazuh-indexer.service failed because the control process exited with error code.
See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.

systemctl status wazuh-indexer -l

× wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/wazuh-indexer.service.d
             └─wazuh-indexer.conf
     Active: failed (Result: exit-code) since Thu 2024-09-19 08:25:14 CEST; 1min 6s ago
       Docs: https://documentation.wazuh.com
    Process: 1978278 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 1978278 (code=exited, status=1/FAILURE)
        CPU: 18.381s

Sep 19 08:25:13 wazuh systemd-entrypoint[1978278]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Sep 19 08:25:13 wazuh systemd-entrypoint[1978278]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Sep 19 08:25:13 wazuh systemd-entrypoint[1978278]:         at org.opensearch.cli.Command.main(Command.java:101)
Sep 19 08:25:13 wazuh systemd-entrypoint[1978278]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
Sep 19 08:25:13 wazuh systemd-entrypoint[1978278]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Sep 19 08:25:13 wazuh systemd-entrypoint[1978278]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Sep 19 08:25:14 wazuh systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Sep 19 08:25:14 wazuh systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Sep 19 08:25:14 wazuh systemd[1]: Failed to start wazuh-indexer.

Both filebeat.yml and opensearch.yml have the correct IP.

openssl x509 -in /etc/wazuh-indexer/certs/wazuh-indexer.pem -text -noout

Has 127.0.0.1 for the certificate. Can I just change the wazuh-indexer.pem?

systemctl restart wazuh-indexer

systemctl restart wazuh-indexer
Job for wazuh-indexer.service failed because the control process exited with error code.
See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.

filebeat test output
elasticsearch: https://XX.XXX.XXX.XXX:9200...

  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK

    addresses: XXX.XXX.XXX.XXX
    dial up... ERROR dial tcp  XX.XXX.XXX.XXX:9200 : connect: connection refused

Cheers chic.

[hasitha.u...@wazuh.com]

Hello Andrehens,

Due to Indexer is not up and running, we have to check the Wazuh-Indexer first. Please provide the output of these commands.

cat /etc/wazuh-indexer/opensearch.yml

ll /etc/wazuh-indexer/certs/

Let me know the update on this.

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

Hey hasitha

cat /etc/wazuh-indexer/opensearch.yml

network.host: "XXX.XXX.XXX.XXX"
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
cluster.name: "wazuh-cluster"

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

bootstrap.memory_lock: true

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=indexer,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

ll /etc/wazuh-indexer/certs/

total 28
dr-x------  2 wazuh-indexer wazuh-indexer 4096 Feb 20  2024 ./
drwxr-x--- 11 wazuh-indexer wazuh-indexer 4096 Sep 19 08:27 ../
-r--------  1 wazuh-indexer wazuh-indexer 1704 Feb 20  2024 admin-key.pem
-r--------  1 wazuh-indexer wazuh-indexer 1119 Feb 20  2024 admin.pem
-r--------  1 wazuh-indexer wazuh-indexer 1204 Feb 20  2024 root-ca.pem
-r--------  1 wazuh-indexer wazuh-indexer 1704 Feb 20  2024 wazuh-indexer-key.pem
-r--------  1 wazuh-indexer wazuh-indexer 1285 Feb 20  2024 wazuh-indexer.pem

Cheers

chic

[hasitha.u...@wazuh.com]

Hello Andrehens,

The configuration looks good to me.
Please restart the Wazuh indexer and share the indexer log.  
systemctl restart wazuh-indexer

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn”

Let me know the update to check further

Regards,

Hasitha Upekshitha

[Andrehens Chicfici]

systemctl restart wazuh-indexer
Job for wazuh-indexer.service failed because the control process exited with error code.

See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.

systemctl status wazuh-indexer.service

× wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/wazuh-indexer.service.d
             └─wazuh-indexer.conf

     Active: failed (Result: exit-code) since Fri 2024-09-20 11:10:47 CEST; 2min 1s ago
       Docs: https://documentation.wazuh.com
    Process: 2171793 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 2171793 (code=exited, status=1/FAILURE)
        CPU: 20.139s

Sep 20 11:10:46 wazuh systemd-entrypoint[2171793]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Sep 20 11:10:46 wazuh systemd-entrypoint[2171793]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Sep 20 11:10:46 wazuh systemd-entrypoint[2171793]:         at org.opensearch.cli.Command.main(Command.java:101)
Sep 20 11:10:46 wazuh systemd-entrypoint[2171793]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
Sep 20 11:10:46 wazuh systemd-entrypoint[2171793]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Sep 20 11:10:46 wazuh systemd-entrypoint[2171793]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Sep 20 11:10:47 wazuh systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Sep 20 11:10:47 wazuh systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Sep 20 11:10:47 wazuh systemd[1]: Failed to start wazuh-indexer.
Sep 20 11:10:47 wazuh systemd[1]: wazuh-indexer.service: Consumed 20.139s CPU time.

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn”

[2024-09-06T03:41:26,197][ERROR][o.o.a.a.AlertIndices     ] [node-1] info deleteOldIndices
[2024-09-06T03:41:26,198][ERROR][o.o.a.a.AlertIndices     ] [node-1] info deleteOldIndices
[2024-09-06T05:01:37,826][ERROR][o.o.i.i.ManagedIndexRunner] [node-1] Failed to save ManagedIndexMetaData for [index=wazuh-alerts-4.x-2024.08.22]
[2024-09-06T05:01:37,855][ERROR][o.o.i.i.ManagedIndexRunner] [node-1] Failed to update ManagedIndexMetaData after executing the Step : attempt_transition_step
[2024-09-06T05:01:38,704][WARN ][o.o.t.TransportService   ] [node-1] Transport response handler not found of id [12170236]
[2024-09-06T05:01:39,071][ERROR][o.o.j.s.u.LockService    ] [node-1] Exception occurred finding lock
[2024-09-06T05:01:39,188][ERROR][o.o.j.s.u.LockService    ] [node-1] Exception occurred finding lock
[2024-09-06T05:01:39,407][ERROR][o.o.j.s.u.LockService    ] [node-1] Exception occurred finding lock
[2024-09-06T05:01:39,487][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [DefaultDispatcher-worker-5]
[2024-09-06T05:01:39,487][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [DefaultDispatcher-worker-3]
[2024-09-06T05:01:39,487][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [DefaultDispatcher-worker-2]

Cheers

chic

[Andrehens Chicfici]

So it seems that the indexer is not starting or running at all. I also don't get any info about WHY it fails. Do you have any ideas how to troubleshoot further or at least reinstall it?

cheers

chic

[hasitha.u...@wazuh.com]

Hello Andrehens,

It seems to us the indexer was not upgraded properly.
While upgrading we need to make sure every component is up and running properly before moving to the next one.

Please upgrade Wazuh-Indexer again.
https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html#upgrading-the-wazuh-indexer

If you have not removed the ss4o index templates, disable shard allocation, and do a synced flush previously. If these steps aren’t done, the upgrade might fail.
You can follow this document to uninstall Wazuh indexer.
https://documentation.wazuh.com/current/installation-guide/uninstalling-wazuh/central-components.html#uninstall-the-wazuh-indexer

You can follow this document for installing the indexer. Make sure to skip Certificate creation, add the Wazuh repository part, and start from Installing the Wazuh indexer part.
https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#installing-the-wazuh-indexer

To do so please take a backup of cert files and your indexer configuration and use it in your new indexer.
Create a directory on other location.
For example:
mkdir /home/wazuh-indexer-backup
Then take a backup
mv /etc/wazuh-indexer/certs/* /home/wazuh-indexer-backup/
mv /etc/wazuh-indexer/opensearch.yml /home/wazuh-indexer-backup/

After install locate them to required paths.

Let me know the update on this.

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

Hey Hasitha.

when I try to delete the ss4o index templates I can't even access them:

curl -X DELETE "https://<WAZUH_INDEXER_IP_ADDRESS>:9200/_index_template/ss4o_*_template" -u <USERNAME>:<PASSWORD> -k

curl: (7) Failed to connect to XX.XXX.XX.XXX port 9200 after 0 ms: Connection refused

The password is correct. The IP is the one of my wazuh server. I have no ideas anymore...

cheers chic

[Andrehens Chicfici]

I purged the wazuh-indexer now and reinstalled it. Something changed.

 tail -n 99999 /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn|critical|fatal"

[2024-09-24T10:33:09,631][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2024-09-24T10:33:10,752][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.13.0]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
[2024-09-24T10:33:10,757][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.13.0]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
[2024-09-24T10:34:45,829][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2024-09-24T10:34:46,615][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.13.0]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
[2024-09-24T10:34:46,620][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.13.0.jar:2.13.0]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]

Cheers

chic

[hasitha.u...@wazuh.com]

Hello  Andrehens,

I hope you backed up your files before reinstalling. The log "Unable to read /etc/wazuh-indexer/certs/indexer.pem" indicates the certificate file can't be accessed. Please ensure the certificate files have been restored to /etc/wazuh-indexer/certs/.

If the files are present, check their permissions with:
ls -lh /etc/wazuh-indexer/certs

You can set the correct permissions using the following commands:
chmod 500 /etc/wazuh-indexer/certs
chmod 400 /etc/wazuh-indexer/certs/*
chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

Also, verify that the old opensearch.yml file has been restored to /etc/wazuh-indexer/.

Additionally, there's a HeapDumpOnOutOfMemoryError warning. To address this, make sure the JVM heap size is sufficient for your data. You may need to increase the JVM heap size in your indexer nodes. Keep these recommendations in mind:

Use no more than 50% of available RAM.
Do not exceed 32 GB of heap size.
First, check the memory on your indexer nodes by running:
free -h

Then, edit the /etc/wazuh-indexer/jvm.options file to adjust the JVM heap settings. For instance, if your server has 12GB of RAM, you can set the heap size to 6GB like this:
-Xms6g
-Xmx6g

After updating the heap size, restart the Wazuh indexer for the changes to take effect:
systemctl restart wazuh-indexer

For more details, you can refer to the official documentation:
https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking

Let me know how it goes!

Regards,
Hasitha Upekshitha

[Andrehens Chicfici]

ls -lh /etc/wazuh-indexer/certs
-r-------- 1 wazuh-indexer wazuh-indexer 1.7K Feb 20  2024 admin-key.pem
-r-------- 1 wazuh-indexer wazuh-indexer 1.1K Feb 20  2024 admin.pem
-r-------- 1 wazuh-indexer wazuh-indexer 1.2K Feb 20  2024 root-ca.pem
-r-------- 1 wazuh-indexer wazuh-indexer 1.7K Feb 20  2024 wazuh-indexer-key.pem

-r-------- 1 wazuh-indexer wazuh-indexer 1.3K Feb 20  2024 wazuh-indexer.pem

heap size is 48G. This should be enough....

cheers

chic

[Andrehens Chicfici]

I set up a new wazuh instance now. Anyway thanks for the help!

[hasitha.u...@wazuh.com]

Hello Andrehens,

I am glad to know that your Wazuh environment is back.

Let me know if you need further assistance.

Regards,
Hasitha Upekshitha