Wazuh cluster deployment

[Satwika sree]

Hello everyone,

I'm in need of assistance regarding the deployment process for a Wazuh cluster. Currently, I've initiated the setup by utilizing two single nodes, following the step-by-step instructions provided in the Wazuh documentation: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html. 

However, I've encountered an issue during the cluster installation phase as outlined in this section: https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#cluster-initialization.

The specific error message I'm encountering is related to "Contacting opensearch cluster 'opensearch' and waiting for YELLOW clusterstate ... Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. While I understand that this might not be an actual error and the system will continue attempting, I am seeking guidance on how to proceed. The root cause of the issue seems to be a java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE].

[Damian Nicastro]

Hello Satwika:

I hope you are fine.

Could you confirm that you are talking about a wazuh-indexer cluster? If that so, the recommendation is to install at least 3 nodes for a cluster in order to be able to have a replica of the indices distributed in the three nodes.

Please, let me know.

Thanks 

[Satwika sree]

Yes, I am taking wazuh -indexer cluster.

Actually, my plan is to install the Wazuh indexer, server, and dashboard on one host, and on the other host, I'll install another indexer and server. By combining these two hosts, I aim to establish a cluster setup. 

Could you please assist me in ensuring this configuration is correct?

[Satwika sree]

pls help me

[Damian Nicastro]

Hello Satwika:
I hope you are fine.

The recommendation for a small distributed deployment (less than 100 wazuh-agents) is to have one machine with the wazuh-indexer and wazuh-dashboard and another with wazuh-server and filebeat service.

If you have more than this, is convenient to have a wazuh-server cluster of at least 2 machines (with filebeat on each of them). Additionally, the installation of Load balancer (like nginx) is required to balance the wazuh-agents load between both wazuh-servers. On the wazuh-indexer, a cluster of three is required if you need to have replicas of your indices.

Please, let me know about your case.

Thanks

[Satwika sree]

Hi,

The content you provided is fine with me. 

However, I'm encountering errors during the initialization of the Wazuh indexer cluster. The error message is as follows:
In the "/usr/share/wazuh-indexer/bin" directory, when I run the "indexer-security-init.sh" script, I receive the following output:

ThinkCentre-M72e:/usr/share/wazuh-indexer/bin# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.6.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
  Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Additionally, in the "wazuh-indexer_cluster.log" file, I observed the following logs:

[2023-08-30T10:08:05,907][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
[2023-08-30T10:08:06,287][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Exception while retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2023-08-30T10:08:06,356][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [node-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors

Please provide assistance in resolving this issue.
 

[Damian Nicastro]

Hello Satwika:

I hope you are fine.

I would try first to check your certificates. 

Please, send me the /etc/wazuh-indexer/opensearch.yml file of each wazuh-indexer node to check the location and names of the certificates and also the result of:

# ls -l /etc/wazuh-indexer/certs

In all the nodes too.

Also verify the that the subject of the certificates are the same as configured in /etc/wazuh-indexer/opensearch.yml:

You should check the same in the wazuh-dashboard and filebeat service:

# cat /etc/wazuh-dashboard/opensearch_dashboards.yml
# ls -l /etc/wazuh-dashboard/certs

# cat /etc/filebeat/filebeat.yml

# ls -l /etc/filebeat/certs

Also verify the that the subject of all the certificates are the same as configured in /etc/wazuh-indexer/opensearch.yml:

# openssl x509 -in <cert> -noout -subject

And the certificates were issued by the same CA:

# openssl verify -CAfile /etc/wazuh-indexer/certs/root-ca.pem <cert>

If all this is fine, you can try running the securityadmin script to reissue the security in your cluster:

# /usr/share/wazuh-indexer/plugins/opendsearch-security/tools/securityadmin.sh -cd /usr/etc/wazuh-indexer/opensearch-security/ -nhnv -icl -cacert /etc/wazuh-indexer/certs/root-ca.pem -cert /etc/wazuh-indexer/certs/admin.pem -key /etc/wazuh-indexer/certs/admin-key.pem -h <wazuh-indexer_ip>

I hope this helps