Error on initializing Opendistro Cluster for distributed deployment

[angelo zinna]

Hi everyone,

I'm trying to reproduce the same tutorial of distributed deployment of Wazuh server, with 2 Wazuh servers and 2 Elasticsearch with Opendistro (https://documentation.wazuh.com/current/installation-guide/open-distro/distributed-deployment/step-by-step-installation/elasticsearch-cluster/elasticsearch-multi-node-cluster.html#elasticsearch-multi-node-cluster) into 4  t2.large EC2 instances  (2 CPU, 8 GB RAM).

However, I always receive an error when initializing the Opendistro cluster in step 3 of the tutorial with the command into the elastic-master:

[root@ip-172-31-20-158 certs]# export JAVA_HOME=/usr/share/elasticsearch/jdk/ && /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh -cd /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/ -icl -nhnv -cacert /etc/elasticsearch/certs/root-ca.pem -cert /etc/elasticsearch/certs/admin.pem -key /etc/elasticsearch/certs/admin-key.pem -h 172.31.20.158 --diagnose

where 172.31.20.158 is the elastic-master
The error is:

[root@ip certs]# Open Distro Security Admin v7 Will connect to 172.31.20.158:9300 ... done Connected as CN=admin,OU=Docu,O=Wazuh,L=California,C=US Elasticsearch Version: 7.10.2 Open Distro Security Version: 1.13.1.0 Diagnostic trace written to: /root/certs/securityadmin_diag_trace_2021-Dec-21_11-01-14.txt Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ... Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ... Root cause: MasterNotDiscoveredException[null] (org.elasticsearch.discovery.MasterNotDiscoveredException/org.elasticsearch.discovery.MasterNotDiscoveredException) * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates) * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file) * Add --accept-red-cluster to allow securityadmin to operate on a red cluster. PendingClusterTasksRequest: MasterNotDiscoveredException[null]         at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:230)         at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:335)         at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252)         at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:601)         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)         at java.lang.Thread.run(Thread.java:832) IndicesStatsRequest: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]         at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:190)         at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:70)         at org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:48)         at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction$AsyncAction.<init>(TransportBroadcastByNodeAction.java:258)         at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:236)         at org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecute(TransportBroadcastByNodeAction.java:76)         at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179)         at com.amazon.opendistroforelasticsearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:124)         at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)         at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply0(OpenDistroSecurityFilter.java:231)         at com.amazon.opendistroforelasticsearch.security.filter.OpenDistroSecurityFilter.apply(OpenDistroSecurityFilter.java:151)         at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177)         at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155)         at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)         at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:60)         at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceivedDecorate(OpenDistroSecuritySSLRequestHandler.java:182)         at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityRequestHandler.messageReceivedDecorate(OpenDistroSecurityRequestHandler.java:293)         at com.amazon.opendistroforelasticsearch.security.ssl.transport.OpenDistroSecuritySSLRequestHandler.messageReceived(OpenDistroSecuritySSLRequestHandler.java:142)         at com.amazon.opendistroforelasticsearch.security.OpenDistroSecurityPlugin$7$1.messageReceived(OpenDistroSecurityPlugin.java:639)         at com.amazon.opendistroforelasticsearch.indexmanagement.rollup.interceptor.RollupInterceptor$interceptHandler$1.messageReceived(RollupInterceptor.kt:124)         at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:72)         at org.elasticsearch.transport.InboundHandler.handleRequest(InboundHandler.java:207)         at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:107)         at org.elasticsearch

Troubleshooting process that I checked are:

• Port 9200/TCP and 9300/TCP are open (so Elasticsearch is running) and reachable by every host in the subnet
• ICMP rule per the firewall is open for all the subnet
• IP and node name are correct
• The 2 Elasticsearch host have the same hostname of their node name
  
  

Thanks in advance for the answer;
AZ

[Daniel Folch]

Hello,

This error usually appears because the master nodes in your Elasticsearch cluster have not passed the bootstrapping. In this stage the all the master nodes share their configuration to determine which are master eligibles and which are the initial master node.

This usually fails when the nodes can’t reach each other or because of a configuration error.

Please check that all the master nodes are running correctly and that the certificates are correct. Also, verify that all the nodes that have the tag node.master: true are included in cluster.initial_master_nodes: and their IPs in discovery.seed_hosts:.

If possible can you share the elasticsearch.yml configuration file, please remember to hide any sensitive information.

Regards,
Daniel F

​

[angelo zinna]

Hi Daniel,

Thanks for the answer, the Elasticsearch configuration file is attached in this new mail.
In the meantime I am trying to reproduce the changes you suggested since:
- of the two Elasticsearch hosts I have indicated only one as an eligible master (and not both)
I will let you know what the result will be.
Thanks again,

AZ

[Seetha Ram]

Dear AZ, did you have a resolution for this? I am facing the same issue and couldn't fix it.