Migration to wazuh-indexer opensearch security not initialized

[gcstechnet]

I have a single node Wazuh-manager with a 3 node cluster of elasticsearch machines.  I followed the migration instructions and when running filebeat test output I get the following error

ERROR 503 Service Unavailable: OpenSearch Security not initialized.

I had security set up between the filebeat instance on the Wazuh-manager and the elasticsearch nodes.  I am not sure how to fix this issue with the new Wazuh-Indexer configuration.   Any help would be greatly appreciated.

Thank you in advance.

Phil

[José Fernández]

Hello Phil,

Could you test to perform a curl command to the indexer environment? probably is something misconfigured.

curl -k -u USER:PASS https://YOUR_INDEXER_IP:9200/_cluster/health?pretty

curl -k -u USER:PASS https://YOUR_INDEXER_IP:9200/_cat/indices

If such commands don't produce an output. Probably it's a symptom of a bad migration procedure.

Please attach here all the information that you have about the indexer status.

Execute:

journalctl -xe -u wazuh-indexer

I will wait for your reply. Thanks.

[gcstechnet]

First curl command

OpenSearch Security not initialized.

Second curl command

OpenSearch Security not initialized.

journalctl -xe -u wazuzh-indexer response

-- A stop job for unit wazuh-indexer.service has begun execution.
--
-- The job identifier is 2906.
May 19 14:35:50 lascluster21 systemd[1]: wazuh-indexer.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit wazuh-indexer.service has successfully entered the 'dead' state.
May 19 14:35:50 lascluster21 systemd[1]: Stopped Wazuh-indexer.
-- Subject: A stop job for unit wazuh-indexer.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit wazuh-indexer.service has finished.
--
-- The job identifier is 2906 and the job result is done.
May 19 14:35:50 lascluster21 systemd[1]: Starting Wazuh-indexer...
-- Subject: A start job for unit wazuh-indexer.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit wazuh-indexer.service has begun execution.
--
-- The job identifier is 2906.
May 19 14:36:06 lascluster21 systemd-entrypoint[159360]: WARNING: An illegal reflective access oper>
May 19 14:36:06 lascluster21 systemd-entrypoint[159360]: WARNING: Illegal reflective access by io.p>
May 19 14:36:06 lascluster21 systemd-entrypoint[159360]: WARNING: Please consider reporting this to>
May 19 14:36:06 lascluster21 systemd-entrypoint[159360]: WARNING: Use --illegal-access=warn to enab>
May 19 14:36:06 lascluster21 systemd-entrypoint[159360]: WARNING: All illegal access operations wil>
May 19 14:36:38 lascluster21 systemd[1]: Started Wazuh-indexer.
-- Subject: A start job for unit wazuh-indexer.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit wazuh-indexer.service has finished successfully.
--
-- The job identifier is 2906.

Thank you for your assistance.

Phil

[José Fernández]

You have a wrong security state, or something removed the security index.

Try to re-create the security index executing this command in the Indexer master node:

/usr/share/wazuh-indexer/bin/indexer-security-init.sh

Then, your master node has to initialize and reply to the curl command.

Could you share your indexer configuration file? It's inside /etc/wazuh-indexer/opensearch.yml. Hash all IPs and domains.

You have more information about cluster configuration at https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#cluster-initialization

[gcstechnet]

The indexer-security-init.sh script fixed the issues.  That command is not in the migration documentation.  The cluster is back up and healthy.  Thank you very much for your assistance with this issue.

Phil

[José Fernández]

I'm glad to know you solved the issue. We will review the migration documentation to include some optional steps as you did. Don't hesitate to ask us if you have any doubts.

Jose.

[Valerio Vinci]

Hi,

I'm having the same issue but the indexer-security-init.sh is not working...

[root@xxx ~]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Security Admin v7
Will connect to 127.0.0.1:9300
ERR: Seems there is no OpenSearch running on 127.0.0.1:9300 - Will exit

for more details -> https://groups.google.com/g/wazuh/c/UchSi0PnLTc/m/6GbpQkCyMwAJ

[José Fernández]

Hello Valeavin,

I have seen that Federico is helping you in this process. I will let him help you but. I will help in case of need. Thanks.

[Rai Ner]

Hello, 
I have a similar Issue with the Migration to wazuh-indexer, I hope you can point me to the right direction. 

"filebeat test output" receives as result: "ERROR 503 Service Unavailable: OpenSearch Security not initialized."

I already tried indexer-security-init.sh but received this error: 

# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
Security Admin v7

Will connect to 10.55.XXX.XX:9300 ... done
15:18:43.633 [opensearch[_client_][transport_worker][T#1]] ERROR org.opensearch.security.ssl.transport.SecuritySSLNettyTransport - Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

find below further details: 
###
journalctl -xe -u wazuh-indexer
Sep 05 15:21:51 WAZUH-ELASTIC01.test.net systemd[1]: Starting Wazuh-indexer...
-- Subject: Unit wazuh-indexer.service has begun start-up
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- Unit wazuh-indexer.service has begun starting up.
Sep 05 15:22:00 WAZUH-ELASTIC01.test.net systemd-entrypoint[19954]: WARNING: An illegal reflective access operation has occurred
Sep 05 15:22:00 WAZUH-ELASTIC01.test.net systemd-entrypoint[19954]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-detection/>
Sep 05 15:22:00 WAZUH-ELASTIC01.test.net systemd-entrypoint[19954]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
Sep 05 15:22:00 WAZUH-ELASTIC01.test.net systemd-entrypoint[19954]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Sep 05 15:22:00 WAZUH-ELASTIC01.test.net systemd-entrypoint[19954]: WARNING: All illegal access operations will be denied in a future release
Sep 05 15:22:32 WAZUH-ELASTIC01.test.net systemd[1]: Started Wazuh-indexer.
-- Subject: Unit wazuh-indexer.service has finished start-up
-- Defined-By: systemd
-- Support: https://support.oracle.com
--
-- Unit wazuh-indexer.service has finished starting up.
--
-- The start-up result is done.

###
# curl --noproxy '*' -k -u admin:XYZ https://10.55.XXX.XX:9200/_cluster/health?pretty
OpenSearch Security not initialized.
###

Many thanks for your Support

Rainer

[Rai Ner]

Hi, 

I managed to get one step further, but there is still an issue...

I recreated the certificates and verified the configuration, I executed securityadmin.sh manually, but the result is the same as with indexer-security-init.sh

WAZUH-ELASTIC01[wazuh-elastic01]:/etc/wazuh-indexer # /usr/share/wazuh-indexer/plugins/opensearch-security/tools/securityadmin.sh -nhnv -cacert /etc/wazuh-indexer/certs/root-ca.pem -cert /etc/wazuh-indexer/certs/admin.pem -key /etc/wazuh-indexer/certs/admin-key.pem -icl -h 10.55.XXXX.XXX -cd /usr/share/wazuh-indexer/plugins/opensearch-security/securityconfig
Security Admin v7
Will connect to 10.55.XXX.XXX:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Wazuh / opensearch Log: (same messages on all of the Nodes)

[2022-09-07T13:59:30,491][ERROR][o.o.s.a.BackendRegistry  ] [wazuh-elastic01] Not yet initialized (you may need to run securityadmin)
[2022-09-07T13:59:32,093][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-elastic01] master not discovered or elected yet, an election requires at least 2 nodes with ids from [MYtAKAw5SPeTZq_1Swe93A, rWyPopx6TiKYkq8k28YmoQ, GPIj8tx2Tz-pT17JW0iWaQ], have discovered [{wazuh-elastic01}{MYtAKAw5SPeTZq_1Swe93A}{c8c9cX4mQ5qCMUd3qQtIxQ}{10.55.XXX.41}{10.55.250.41:9300}{dimr}{shard_indexing_pressure_enabled=true}, {wazuh-elastic02}{qaywH0xQQRCcXj4JpRM4rw}{FpvB-xt-TCavwGakQo9JKA}{10.55.XXX.XX}{10.55.XXX.XX:9300}{dimr}{shard_indexing_pressure_enabled=true}, {wazuh-elastic03}{OVPOIMV6RzGDQgZRKQHBWQ}{V1RH5C0UTuStKgaW14DhYQ}{10.55.250.43}{10.55.250.43:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [10.55.XXXX.XXX:9300, 10.55.XXX.XXX:9300] from hosts providers and [{wazuh-elastic01}{MYtAKAw5SPeTZq_1Swe93A}{c8c9cX4mQ5qCMUd3qQtIxQ}{10.55.XXXX.41}{10.55.XXX.41:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 51, last-accepted version 10592476 in term 51
[2022-09-07T13:59:32,979][ERROR][o.o.s.a.BackendRegistry  ] [wazuh-elastic01] Not yet initialized (you may need to run securityadmin)

Any hints are appreciated. 

Thanks

Rainer