Upgrade 4.9 error WEB

[German DiCasas]

Hi team,

I upgraded my 4.8 to 4.9 and after do all the docs related to upgrade process the web interface not work. Shows "Wazuh dashboard server is not ready yet"

I have the wazuh-manager dashboard and indexer on same server. THe only error that I have over log ossec.lo is 2024/09/05 20:04:07 wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database. and some others related.

At the moment  to upgrade the indexer asked to overwrite some files like /etc/init.d/wazuh-indexer , /etc/wazuh-indexer/jvm.options and /etc/wazuh-dashboard/opensearch_dashboards.yml and select Y

Let me know what can be the issue.

journalctl -u wazuh-dashboard | grep -iE "err|warn"

.

.

}
Sep 05 19:59:20 wazuh-srv-a opensearch-dashboards[681]: {"type":"log","@timestamp":"2024-09-05T22:59:20Z","tags":["error","opensearch","data"],"pid":681,"message":"[ResponseError]: Response Error"}
Sep 05 19:59:21 wazuh-srv-a opensearch-dashboards[681]: {"type":"log","@timestamp":"2024-09-05T22:59:21Z","tags":["warning","savedobjects-service"],"pid":681,"message":"Unable to connect to OpenSearch. Error: Given the configuration, the ConnectionPool was not able to find a usable Connection for this request."}
Sep 05 20:00:40 wazuh-srv-a opensearch-dashboards[7641]: {"type":"log","@timestamp":"2024-09-05T23:00:40Z","tags":["error","opensearch","data"],"pid":7641,"message":"[ResponseError]: Response Error"}
Sep 05 20:00:40 wazuh-srv-a opensearch-dashboards[7641]: {"type":"log","@timestamp":"2024-09-05T23:00:40Z","tags":["error","savedobjects-service"],"pid":7641,"message":"Unable to retrieve version information from OpenSearch nodes."}
Sep 05 20:00:42 wazuh-srv-a opensearch-dashboards[7641]: {"type":"log","@timestamp":"2024-09-05T23:00:42Z","tags":["error","opensearch","data"],"pid":7641,"message":"[ResponseError]: Response Error"}

Regards

German

[German DiCasas]

Also 

/var/log/wazuh-indexer/wazuh-cluster.log
[2024-09-05T23:12:40,692][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.20.11:33456
[2024-09-05T23:12:43,195][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.20.11:33456
[2024-09-05T23:12:43,326][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"]) 

Regards

[Stuti Gupta]

Hi German Dicass

To solve this issue I recommend the following steps:

First, check the Wazuh Dashboard certificates using this command:
ls -lrt /etc/wazuh-dashboard/certs/

Ensure that the path and file names match in:
/etc/wazuh-dashboard/opensearch_dashboards.yml

You need to update the password for the kibanaserver user, simply changing the <KIBANASERVER_PASSWORD> in the Wazuh Dashboard keystore with the old password might not work. Here's what to do:

Change the password.
If you use the tool in an all-in-one deployment, it automatically updates the passwords where necessary. If you use it in a distributed environment, depending on the user whose password you change, you may have to update the password on other components. See Changing the Passwords in a distributed environment for more details.  https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html#passwords-distributed

To change the password, use this command:
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>'
The password must have a length between 8 and 64 characters and contain at least one upper and lower case letter, a number and a symbol(.*+?-)

For distributed environments, update the password using the following command:
echo <kibanaserver>| /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearc

Let me know if this helps or if you need further assistance!

Refer to https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html#password-management

[Franck Ehret]

Hi Stuti Gupta

I've the exact same symptoms/logs but I don't get the password change thing as I don't have any user/password (method certificate apparently and user/pass are commented out):

My installation is pretty old and I migrated to opensearch on the way.

Should I try the password change too?

Thanks and kind regards

[Stuti Gupta]

Hi  Franck Ehret

For you issue, please open another thread so we can track it better, which will also help other team members.

[Franck Ehret]

Hi,

I did open one in the meantime, thanks ;-)

[German DiCasas]

HI Stuti, thanks for the reply

I did the check and the certs are ok. Also I ejecuted this to change all users passwords  

curl -sO https://packages.wazuh.com/4.8/wazuh-passwords-tool.sh
bash wazuh-passwords-tool.sh --api --change-all --admin-user wazuh --admin-password <WAZUH_PASSWORD>

cat /etc/wazuh-dashboard/opensearch_dashboards.yml
server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://192.168.2.11:9200
opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home

ls -lrt /etc/wazuh-dashboard/certs/
total 12
-r-------- 1 wazuh-dashboard wazuh-dashboard 1204 Aug  8 18:44 root-ca.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1281 Aug  8 18:44 dashboard.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1704 Aug  8 18:44 dashboard-key.pem

Same problem for now.. 

[German DiCasas]

OK... until now

I did the related to https://github.com/wazuh/wazuh/issues/25605#issuecomment-2333884627 

 cp /etc/wazuh-dashboard/opensearch_dashboard.keystore /etc/wazuh-dashboard/opensearch_dashboard.keystore.back
 cp /usr/share/wazuh-dashboard/config/opensearch_dashboard.keystore /etc/wazuh-dashboard/

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p  XXX with the before password and work OK the web but not the password of the user for login, admin or my user too.. I did a reset all again and the issue come back again "Wazuh dashboard server is not ready yet".I did all this proccess again and  I tested too reset a single user and the same, the reset not work and admin password too. I think that is related but not sure how it works.

The fix over https://github.com/wazuh/wazuh/issues/25605#issuecomment-2333884627  say something about OSD_PATH_CONF="/etc/wazuh-dashboard" but I do not know where to change or add that...

Help

Regards

German

[Stuti Gupta]

Please confirm if this is an all-in-one deployment or a distributed deployment.
In the case of a distributed deployment, you'll need to update the passwords as well.

check the Wazuh Dashboard certificates using this command:
ls -lrt /etc/wazuh-dashboard/certs/

Ensure that the path and file names match in:
/etc/wazuh-dashboard/opensearch_dashboards.yml

Additionally, please provide the following logs:
Wazuh cluster logs : cat /var/log/wazuh-indexer/wazuh-cluster.log
Wazuh dashboard logs:   cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log

Refre to https://documentation.wazuh.com/current/user-manual/user-administration/password-management.html

Hope to hear from you soon 

[German DiCasas]

Hi Stuti,

The certs are ok and I reset the password too as doc. As before I reseted too the kibanaserver with the comand that you send me, and nothing..Let me know what else...Also, I have the wazuh indexer, manager, filebeat and dashboard all in same server.

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>'    (a new password and not the same that give me the the last of reset all apsswords...)I ejecuted this again and restarted the services indexer, manager , filebeat and dahboard... Now, not show the  issue Wazuh dashboard server is not ready yet ....  Only Black screen 

ls -lrt /etc/wazuh-dashboard/certs/
total 12
-r-------- 1 wazuh-dashboard wazuh-dashboard 1204 Aug  8 18:44 root-ca.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1281 Aug  8 18:44 dashboard.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1704 Aug  8 18:44 dashboard-key.pem

cat /etc/wazuh-dashboard/opensearch_dashboards.yml
server.host: 0.0.0.0
server.port: 443
opensearch.hosts: https://192.168.2.11:9200
opensearch.ssl.verificationMode: certificate
#opensearch.username:
#opensearch.password:
opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]
opensearch_security.multitenancy.enabled: false
opensearch_security.readonly_mode.roles: ["kibana_read_only"]
server.ssl.enabled: true
server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"
server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"
opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]
uiSettings.overrides.defaultRoute: /app/wz-home

Wazuh cluster logs : cat /var/log/wazuh-indexer/wazuh-cluster.log

[2024-09-09T12:47:36,139][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 192.168.2.11:54220
[2024-09-09T12:47:37,985][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:40,488][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:42,989][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:45,490][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:47,993][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:50,497][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:52,999][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:55,501][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:47:55,660][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 192.168.2.11:41714
[2024-09-09T12:47:58,002][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:00,505][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:03,006][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:05,509][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:08,008][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:09,045][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 192.168.2.11:54220
[2024-09-09T12:48:10,511][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:13,012][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:15,515][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:18,016][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:20,519][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:23,021][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:25,524][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:28,024][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:30,528][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:33,028][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:35,529][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:38,031][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:40,533][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:43,034][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:45,535][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:48,038][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:48,052][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 192.168.2.11:54220
[2024-09-09T12:48:50,542][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:53,039][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616
[2024-09-09T12:48:55,543][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.2.11:42616


cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log
{"date":"2024-08-30T16:19:35.116Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 192.168.2.11:55000"}
{"date":"2024-08-30T16:19:38.092Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 192.168.2.11:55000"}
{"date":"2024-08-30T16:19:41.132Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 192.168.2.11:55000"}
{"date":"2024-08-30T16:19:44.113Z","level":"error","location":"wazuh-api:makeRequest","message":"connect ECONNREFUSED 192.168.2.11:55000"}
{"date":"2024-08-30T23:52:31.145Z","level":"info","location":"initialize","message":"Wazuh dashboard index: .kibana"}
{"date":"2024-08-30T23:52:31.148Z","level":"info","location":"initialize","message":"App revision: 01"}
{"date":"2024-08-30T23:52:31.149Z","level":"info","location":"initialize","message":"Total RAM: 3917MB"}
{"data":{"message":"connect ECONNREFUSED 192.168.2.11:9200","stack":"ConnectionError: connect ECONNREFUSED 192.168.2.11:9200\n    at ClientRequest.onError (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Connection.js:126:16)\n    at ClientRequest.emit (node:events:513:28)\n    at ClientRequest.emit (node:domain:489:12)\n    at TLSSocket.socketErrorListener (node:_http_client:502:9)\n    at TLSSocket.emit (node:events:513:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"},"date":"2024-09-03T15:55:00.550Z","level":"info","location":"Cron-scheduler"}
{"data":{"message":"connect ECONNREFUSED 192.168.2.11:9200","stack":"ConnectionError: connect ECONNREFUSED 192.168.2.11:9200\n    at ClientRequest.onError (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Connection.js:126:16)\n    at ClientRequest.emit (node:events:513:28)\n    at ClientRequest.emit (node:domain:489:12)\n    at TLSSocket.socketErrorListener (node:_http_client:502:9)\n    at TLSSocket.emit (node:events:513:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"},"date":"2024-09-03T15:55:00.594Z","level":"info","location":"Cron-scheduler"}
{"date":"2024-09-03T16:00:00.560Z","level":"error","location":"monitoring:cronTask","message":"connect ECONNREFUSED 192.168.2.11:9200"}
{"data":{"message":"connect ECONNREFUSED 192.168.2.11:9200","stack":"ConnectionError: connect ECONNREFUSED 192.168.2.11:9200\n    at ClientRequest.onError (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Connection.js:126:16)\n    at ClientRequest.emit (node:events:513:28)\n    at ClientRequest.emit (node:domain:489:12)\n    at TLSSocket.socketErrorListener (node:_http_client:502:9)\n    at TLSSocket.emit (node:events:513:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"},"date":"2024-09-03T16:00:00.623Z","level":"info","location":"Cron-scheduler"}
{"data":{"message":"connect ECONNREFUSED 192.168.2.11:9200","stack":"ConnectionError: connect ECONNREFUSED 192.168.2.11:9200\n    at ClientRequest.onError (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Connection.js:126:16)\n    at ClientRequest.emit (node:events:513:28)\n    at ClientRequest.emit (node:domain:489:12)\n    at TLSSocket.socketErrorListener (node:_http_client:502:9)\n    at TLSSocket.emit (node:events:513:28)\n    at TLSSocket.emit (node:domain:489:12)\n    at emitErrorNT (node:internal/streams/destroy:151:8)\n    at emitErrorCloseNT (node:internal/streams/destroy:116:3)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21)"},"date":"2024-09-03T16:00:00.627Z","level":"info","location":"Cron-scheduler"}
{"date":"2024-09-03T16:06:01.557Z","level":"error","location":"wazuh-check-updates:getSavedObject","message":"No shard available for [get [.kibana][wazuh-check-updates-user-preferences:german]: routing [null]]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.kibana][wazuh-check-updates-user-preferences:german]: routing [null]]"}
{"date":"2024-09-03T16:06:01.558Z","level":"error","location":"wazuh-check-updates:getUserPreferences","message":"No shard available for [get [.kibana][wazuh-check-updates-user-preferences:german]: routing [null]]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.kibana][wazuh-check-updates-user-preferences:german]: routing [null]]"}
{"date":"2024-09-03T16:06:01.689Z","level":"error","location":"wazuh-check-updates:getSavedObject","message":"No shard available for [get [.kibana][wazuh-check-updates-available-updates:wazuh-check-updates-available-updates]: routing [null]]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.kibana][wazuh-check-updates-available-updates:wazuh-check-updates-available-updates]: routing [null]]"}
{"date":"2024-09-03T16:06:01.693Z","level":"error","location":"wazuh-check-updates:getUpdates","message":"No shard available for [get [.kibana][wazuh-check-updates-available-updates:wazuh-check-updates-available-updates]: routing [null]]: no_shard_available_action_exception: [no_shard_available_action_exception] Reason: No shard available for [get [.kibana][wazuh-check-updates-available-updates:wazuh-check-updates-available-updates]: routing [null]]"}

Let me know

[German DiCasas]

Stuti,

After do a reset all with bash wazuh-passwords-tool.sh --api --change-all --admin-user wazuh --admin-password XXXXX I get a password reset of each user but now with this "keystore not found"

ERROR: OpenSearch Dashboards keystore not found. Use 'create' command to create one.
09/09/2024 20:03:28 INFO: The password for user logstash is xxxxx 

.

.

.

The issue continue with  Wazuh dashboard server is not ready yet.

Regards,

German

[German DiCasas]

Stuti Gupta, any update o what can be the issue?

Regards,,

German

[Stuti Gupta]

Hi @german

You only need to change the kibanaserver password and in distributed you need to update the kibanaserver password in the Wazuh dashboard keystore Now in case you have rested all the passwords the follow these steps:

1. Backup your current keystore:
cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak
2. Backup the keystore from the new location:
cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak
3. Move the keystore to the correct location:
mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore
4. Restart the Wazuh dashboard.

For the error:
If the keystore has not been created, you can create it using the following command:
sudo -u wazuh-dashboard /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore create

Then reset the password:
All-in-one
To change only the kibanaserver password:  usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>'   
Run the script with the -a option to changel oll the indexer user:
wazuh-passwords-tool.sh -a

Distributed environment
To change only the kibanaserver password:  usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>'   
Then update the password in keystore:  echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

To change all the passwords in the distributed environment
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh --change-all

If you've set up a user other than admin for Filebeat, manually add the username and password using the following commands. Replace <CUSTOM_USERNAME> and <CUSTOM_PASSWORD> with your custom username and password.
echo <CUSTOM_USERNAME> | filebeat keystore add username --stdin --force
echo <CUSTOM_PASSWORD> | filebeat keystore add password --stdin --force

Restart Filebeat to apply the changes.
systemctl restart filebeat

On your Wazuh dashboard node, run the following command to update the kibanaserver password in the Wazuh dashboard keystore. Replace <KIBANASERVER_PASSWORD> with the random password generated in the first step.
echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

Update the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml configuration file with the new wazuh-wui password generated in the second step.
hosts:
  - default:
      url: https://127.0.0.1
      port: 55000
      username: wazuh-wui
      password: "<wazuh-wui-password>"
      run_as: false
Restart the Wazuh dashboard to apply the changes.
systemctl restart wazuh-dashboard

Hope this helps

[German DiCasas]

Thanks  for the reply.. I have an error on second step command. All in one form may case.

I created the file with : OK
sudo -u wazuh-dashboard /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore create  OK

1.cp /etc/wazuh-dashboard/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore.bak  OK
2. cp /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore.bak  OK
3. mv /usr/share/wazuh-dashboard/config/opensearch_dashboards.keystore /etc/wazuh-dashboard/opensearch_dashboards.keystore  OK
4. Restart the Wazuh dashboard. OK

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p 'Qx*x*a?Tmypasswordghjh(/&Kjhk''
12/09/2024 10:02:39 INFO: Updating the internal users.
12/09/2024 10:02:40 ERROR: The backup could not be created

 

I tested to do it with wazuh-passwords-tool.sh -a but the same error.. Also with sudo bash wazuh-passwords-tool.sh -a -au wazuh -ap password

I did the /usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh --change-all

12/09/2024 10:09:09 INFO: Updating the internal users.
12/09/2024 10:09:10 ERROR: The backup could not be created

I am using the defoult Filebeat user so the next steps after fix that are:

echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

and update  Update the /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml

What can be the problem and the solution of  ERROR: The backup could not be created? 

Regards,

German

[Stuti Gupta]

Hi German:

ERROR: The backup could not be created

It looks as if the securityadmin.sh script is not able to correctly create a backup for your passwords.
Try to restart the Wazuh indexer and try this again.
To get more information as to the nature of the error, could you please run the script on your host with the same arguments as now plus the argument `-v`? Please, send me the output of the script after removing any personal information that would be in it, as ip's or passwords.
Ex:
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>' -v
For wazuh-indexer logs, as you can see there is a file wazuh-cluster.log, located in /var/log/wazuh-indexer/

Let me know the update on the issue.

[German DiCasas]

Hi Stuti,

I cant restart wazuh-indexer, and some other errors after last change.. Let me know:

systemctl status wazuh-indexer
× wazuh-indexer.service - wazuh-indexer
     Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Fri 2024-09-13 21:13:15 -03; 1s ago
       Docs: https://documentation.wazuh.com
    Process: 682 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 682 (code=exited, status=1/FAILURE)
        CPU: 11.739s

Sep 13 21:13:15 wazuh-srv-a systemd-entrypoint[682]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Sep 13 21:13:15 wazuh-srv-a systemd-entrypoint[682]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Sep 13 21:13:15 wazuh-srv-a systemd-entrypoint[682]:         at org.opensearch.cli.Command.main(Command.java:101)
Sep 13 21:13:15 wazuh-srv-a systemd-entrypoint[682]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
Sep 13 21:13:15 wazuh-srv-a systemd-entrypoint[682]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Sep 13 21:13:15 wazuh-srv-a systemd-entrypoint[682]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Sep 13 21:13:15 wazuh-srv-a systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Sep 13 21:13:15 wazuh-srv-a systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Sep 13 21:13:15 wazuh-srv-a systemd[1]: Failed to start wazuh-indexer.service - wazuh-indexer.
Sep 13 21:13:15 wazuh-srv-a systemd[1]: wazuh-indexer.service: Consumed 11.739s CPU time.
systemctl restart wazuh-indexer
Job for wazuh-indexer.service failed because the control process exited with error code.
See "systemctl status wazuh-indexer.service" and "journalctl -xeu wazuh-indexer.service" for details.

journalctl -xeu wazuh-indexer.service

Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]: Sep 14, 2024 12:26:43 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]: WARNING: COMPAT locale provider will be removed in a future release
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:477)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/java.nio.file.Files.newDirectoryStream(Files.java:481)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at java.base/java.nio.file.Files.walkFileTree(Files.java:2820)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:227)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:146)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.cli.Command.main(Command.java:101)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Sep 13 21:26:43 wazuh-srv-a systemd-entrypoint[3945]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Sep 13 21:26:43 wazuh-srv-a systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit wazuh-indexer.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Sep 13 21:26:43 wazuh-srv-a systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit wazuh-indexer.service has entered the 'failed' state with result 'exit-code'.
Sep 13 21:26:43 wazuh-srv-a systemd[1]: Failed to start wazuh-indexer.service - wazuh-indexer.
░░ Subject: A start job for unit wazuh-indexer.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit wazuh-indexer.service has finished with a failure.
░░
░░ The job identifier is 1583 and the job result is failed.
Sep 13 21:26:43 wazuh-srv-a systemd[1]: wazuh-indexer.service: Consumed 9.099s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit wazuh-indexer.service completed and consumed the indicated resources.

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|war"

...

39,163][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:55:41,021][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.20.11:37004
[2024-09-12T11:55:43,524][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.20.11:37004
[2024-09-12T11:55:43,884][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for admin from 192.168.20.11:40784
[2024-09-12T11:55:44,165][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:55:46,027][WARN ][o.o.s.a.BackendRegistry  ] [node-1] Authentication finally failed for kibanaserver from 192.168.20.11:37004
[2024-09-12T11:55:49,167][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:55:54,012][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:55:59,014][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:56:04,089][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:56:09,017][WARN ][o.o.p.c.u.JsonConverter  ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-12T11:58:07,993][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms2g, -Xmx2g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=1073741824, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2024-09-12T11:59:23,116][ERROR][o.o.p.c.j.GCMetrics      ] [node-1] MX bean missing: G1 Concurrent GC
[2024-09-12T12:00:19,932][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-09-12T12:00:20,684][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2024-09-12T12:00:20,686][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.

filebeat test output
elasticsearch: https://192.168.20.11:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 192.168.20.11
    dial up... ERROR dial tcp 192.168.20.11:9200: connect: connection refused

/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p asdasdasdasd -v                            

13/09/2024 22:57:59 INFO: Updating the internal users.
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.168.20.11:9200
ERR: Seems there is no OpenSearch running on 192.168.20.11:9200 - Will exit
cp: cannot stat '/etc/wazuh-indexer/backup/internal_users.yml': No such file or directory
chmod: cannot access '/etc/wazuh-indexer/internalusers-backup/internal_users_20240913_225759.yml.bkp': No such file or directory
13/09/2024 22:58:05 INFO: A backup of the internal users has been saved in the /etc/wazuh-indexer/internalusers-backup folder.
cp: cannot stat '/etc/wazuh-indexer/backup/internal_users.yml': No such file or directory
13/09/2024 22:58:05 INFO: Generating password hash
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.168.20.11:9200
ERR: Seems there is no OpenSearch running on 192.168.20.11:9200 - Will exit

ERROR: OpenSearch Dashboards keystore not found. Use 'create' command to create one.

**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 192.168.20.11:9200
ERR: Seems there is no OpenSearch running on 192.168.20.11:9200 - Will exit
13/09/2024 22:58:17 WARNING: Password changed. Remember to update the password in the Wazuh dashboard, Wazuh server, and Filebeat nodes if necessary, and restart the services.

Regards

German

[Stuti Gupta]

Hello,

As expected the wazuh-indexer is not running that's the reason for the securityadmin.sh script is not able to correctly create a backup for your passwords.
the reason is this error:

son Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain:

This is a known problem that happens when upgrading to Opensearch version 2.13, which is what the indexer uses on 4.9.0:

https://github.com/opensearch-project/performance-analyzer/issues/644

To solve this, you will need to disable the performance analyzer plugin that comes built into Opensearch.
https://opensearch.org/docs/2.13/monitoring-your-cluster/pa/index/#disable-performance-analyzer

Hope this helps

[German DiCasas]

Stuti,

I did the related and same situation. This are the results:

curl -XPOST https://192.168.2.11:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": false}'
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

error with certificates: I did this too:

curl --cacert /etc/wazuh-indexer/certs/root-ca.pem --cert /etc/wazuh-indexer/certs/admin.pem  --key /etc/wazuh-indexer/certs/admin-key.pem -X POST https://192.168.2.11:9200/_plugins/_performanceanalyzer/rca/cluster/config -H 'Content-Type: application/json' -d '{"enabled": false}'
{"currentPerformanceAnalyzerClusterState":3,"shardsPerCollection":0,"batchMetricsRetentionPeriodMinutes":7}

 kill $(ps aux | grep -i 'PerformanceAnalyzerApp' | grep -v grep |                                                                                                     awk '{print $2}')
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]

curl -X POST 192.168.2.11:9200/_plugins/_performanceanalyzer/cluster/config -H 'Content-Type: application/json' -d '{"enabl^C": false}'
 bin/opensearch-plugin remove opensearch-performance-analyzer
bash: bin/opensearch-plugin: No such file or directory

After that same issue:

Wazuh dashboard server is not ready yet

I restarted the services and the same..

What can be the problem?

Regards,

German

[German DiCasas]

I did a bakup and reinstall  all again but to 4.9.0. Its working fine .. Not shure the Events over Threat Hunting, that  changed?.. similar to Discover.. it is correct? on the attached file, the information below, you cant select fields to filter like previous version 4.8.

My wazuh , all in one

wazuh-dashboard                      4.9.0-2
wazuh-indexer                        4.9.0-1                          
wazuh-manager                        4.9.0-1  

Regards,,