Cannot initialize wazuh-indexer cluster

[Robert A]

Hey, 

I'm installing 3-node wazuh-indexer cluster with step-by-step documentation (https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html) and cannot initialize the cluster using indexer-security-init.sh (last step on installation).

I'm getting an error (full output):

Device "wazuhindexer1.ourdomian.name" does not exist.
Security Admin v7
ERR: Parsing failed.  Reason: Missing argument for option: h
usage: securityadmin.sh [-arc] [-backup <folder>] [-cacert <file>] [-cd
       <directory>] [-cert <file>] [-cn <clustername>] [-dci] [-dg] [-dra]
       [-ec <cipers>] [-ep <protocols>] [-er <number of replicas>] [-era]
       [-esa] [-f <file>] [-ff] [-h <host>] [-i <indexname>] [-icl] [-key
       <file>] [-keypass <password>] [-ks <file>] [-ksalias <alias>]
       [-kspass <password>] [-kst <type>] [-migrate <folder>] [-mo
       <folder>] [-nhnv] [-noopenssl] [-nrhn] [-p <port>] [-prompt] [-r]
       [-rev] [-rl] [-si] [-sniff] [-t <file-type>] [-ts <file>] [-tsalias
       <alias>] [-tspass <password>] [-tst <type>] [-us <number of
       replicas>] [-vc <version>] [-w]
 -arc,--accept-red-cluster                      Also operate on a red
                                                cluster. If not specified
                                                the cluster state has to
                                                be at least yellow.
 -backup <folder>                               Backup configuration to
                                                folder
 -cacert <file>                                 Path to trusted cacert
                                                (PEM format)
 -cd,--configdir <directory>                    Directory for config files
 -cert <file>                                   Path to admin certificate
                                                in PEM format
 -cn,--clustername <clustername>                Clustername (do not use
                                                together with -icl)
 -dci,--delete-config-index                     Delete
                                                '.opendistro_security'
                                                config index and exit.
 -dg,--diagnose                                 Log diagnostic trace into
                                                a file
 -dra,--disable-replica-autoexpand              Disable replica auto
                                                expand and exit
 -ec,--enabled-ciphers <cipers>                 Comma separated list of
                                                enabled TLS ciphers
 -ep,--enabled-protocols <protocols>            Comma separated list of
                                                enabled TLS protocols
 -er,--explicit-replicas <number of replicas>   Set explicit number of
                                                replicas or autoexpand
                                                expression for
                                                .opendistro_security index
 -era,--enable-replica-autoexpand               Enable replica auto expand
                                                and exit
 -esa,--enable-shard-allocation                 Enable all shard
                                                allocation and exit.
 -f,--file <file>                               file
 -ff,--fail-fast                                fail-fast if something
                                                goes wrong
 -h,--hostname <host>                           OpenSearch host (default:
                                                localhost)
 -i,--index <indexname>                         The index OpenSearch
                                                Security uses to store the
                                                configuration
 -icl,--ignore-clustername                      Ignore clustername (do not
                                                use together with -cn)
 -key <file>                                    Path to the key of admin
                                                certificate
 -keypass <password>                            Password of the key of
                                                admin certificate
                                                (optional)
 -ks,--keystore <file>                          Path to keystore
                                                (JKS/PKCS12 format
 -ksalias,--keystore-alias <alias>              Keystore alias
 -kspass,--keystore-password <password>         Keystore password
 -kst,--keystore-type <type>                    JKS or PKCS12, if not
                                                given we use the file
                                                extension to dectect the
                                                type
 -migrate <folder>                              Migrate and use folder to
                                                store migrated files
 -mo,--migrate-offline <folder>                 Migrate and use folder to
                                                store migrated files
 -nhnv,--disable-host-name-verification         Disable hostname
                                                verification
 -noopenssl,--no-openssl                        Do not use OpenSSL even if
                                                available (default: use it
                                                if available)
 -nrhn,--disable-resolve-hostname               Disable DNS lookup of
                                                hostnames
 -p,--port <port>                               OpenSearch transport port
                                                (default: 9300)
 -prompt,--prompt-for-password                  Prompt for password if not
                                                supplied
 -r,--retrieve                                  retrieve current config
 -rev,--resolve-env-vars                        Resolve/Substitute env
                                                vars in config with their
                                                value before uploading
 -rl,--reload                                   Reload the configuration
                                                on all nodes, flush all
                                                Security caches and exit
 -si,--show-info                                Show system and license
                                                info
 -sniff,--enable-sniffing                       Enable
                                                client.transport.sniff
 -t,--type <file-type>                          file-type
 -ts,--truststore <file>                        Path to truststore
                                                (JKS/PKCS12 format)
 -tsalias,--truststore-alias <alias>            Truststore alias
 -tspass,--truststore-password <password>       Truststore password
 -tst,--truststore-type <type>                  JKS or PKCS12, if not
                                                given we use the file
                                                extension to dectect the
                                                type
 -us,--update_settings <number of replicas>     Update the number of
                                                Security index replicas,
                                                reload configuration on
                                                all nodes and exit
 -vc,--validate-configs <version>               Validate config for
                                                version 6 or 7 (default 7)
 -w,--whoami                                    Show information about the
                                                used admin certificate

I've configured and deployed certificates and /etc/wazuh-indexer/opensearch.yml accordingly to documentation from link above. DNS names of nodes are visible and available in network where nodes are installed. Here I redacted real names but in configs they look the same just different names.

Full opensearch.yml config from first node, rest are the same just with next numbers (2 and 3):

network.host: "wazuhindexer1.ourdomain.name"
node.name: "wazuhindexer1"
cluster.initial_master_nodes:
- "wazuhindexer1"
- "wazuhindexer2"
- "wazuhindexer3"
cluster.name: "wazuh-cluster"
discovery.seed_hosts:
- "wazuhindexer1"
- "wazuhindexer2"
- "wazuhindexer3"
node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false

plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=wazuhindexer1.ourdomain.name,OU=Wazuh,O=Wazuh,L=California,C=US"
- "CN=wazuhindexer2.ourdomain.name,OU=Wazuh,O=Wazuh,L=California,C=US"
- "CN=wazuhindexer3.ourdomain.name,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

Now, network.host: "wazuhindexer1.ourdomain.name" is configured this way accordingly to documentation:

Our config.yml to create certificates:

nodes:
  # Wazuh indexer nodes
  indexer:
    - name: wazuhindexer1
      ip: wazuhindexer1.ourdomain.name
    - name: wazuhindexer2
      ip: wazuhindexer2.ourdomain.name
    - name: wazuhindexer3
      ip: wazuhindexer3.ourdomain.name

  # Wazuh server nodes
  # Use node_type only with more than one Wazuh manager
  server:
    - name: wazuhmaster1
      ip: wazuhmaster1.ourdomain.name
    node_type: master
    - name: wazuhworker1
      ip: wazuhworker1.ourdomain.name
    node_type: worker

  # Wazuh dashboard nodes
  dashboard:
    - name: wazuhdashboard1
      ip: wazuhdashboard1.ourdomain.name

What we've tried to overcome the problem:

1. Looked into indexer-security-init.sh code and I think it does not work with network.host configured as a DNS but not 100% sure;

2. Changed network.host to: 

a) hostname of indexers - didn't help, 

b) adres 0.0.0.0 - didn't help either, here we fall into this: https://github.com/wazuh/wazuh/issues/12940

c) interface name - didn't help - same as above, output:

Security Admin v7
Will connect to xxx.xxx.xxx.xxx:9300 ... done
Connected as CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US
OpenSearch Version: 1.2.4
OpenSearch Security Version: 1.2.4.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: null. This is not an error, will keep on trying ...
  Root cause: MasterNotDiscoveredException[null] (org.opensearch.discovery.MasterNotDiscoveredException/org.opensearch.discovery.MasterNotDiscoveredException)
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

Configuration per node:

OS version: Debian 11

Platform: Hardware, Dell M610

I'd appreciate any help to initialize indexer cluster properly.

Regards, 

Robert

[Carlos Ezequiel Bordon]

Hi Robert, you have to run the indexer-security-init.sh with the parameter -ho, example:

/usr/share/wazuh-indexer/bin/indexer-security-init.sh -ho wazuhindexer1.domain.name

This must be done when we use DNS instead of IP in the configuration, also validate that the indexer configuration is correct, you can validate in the logs of the different nodes to verify that there are no errors:

less /var/log/wazuh-indexer/wazuh-cluster.log

If the nodes were successfully linked, you should be able to validate messages similar to these:

[2022-07-06T12:17:36,819][INFO ][o.o.a.c.ADClusterEventListener] [wazuhindexer1] Cluster node changed, node removed: false, node added: true
[2022-07-06T12:17:36,820][INFO ][o.o.a.c.HashRing ] [wazuhindexer1] Node added: [TYENRM78RgOW4_zLZhAKLg]
[2022-07-06T12:17:36,834][INFO ][o.o.a.c.HashRing ] [wazuhindexer1] Add data node to AD version hash ring: TYENRM78RgOW4_zLZhAKLg
[2022-07-06T12:17:36,835][INFO ][o.o.a.c.HashRing ] [wazuhindexer1] All nodes with known AD version: {TYENRM78RgOW4_zLZhAKLg=ADNodeInfo{version=1.2.4, isEligibleDataNode=true}, ju-TCSXNQcOZp7Fda9aMcQ=ADNodeInfo{ version=1.2.4, isEligibleDataNode=true}, EN7wfGXXRriOgAGAxmiZVA=ADNodeInfo{version=1.2.4, isEligibleDataNode=true}}

[Robert A]

Ok, eventually we managed to resolve the issue. I'd like to leave my comment about documentation and process of deploying wazuh-indexer with certificates.

1. When submitting hosts names in config.yml for certs I'd have to use DNS name in the name not only in ip, eg.:

nodes:
  # Wazuh indexer nodes
  indexer:

    - name: wazuhindexer1.domain.name
      ip: wazuhindexer1.domain.name
    - name: wazuhindexer2.domain.name
      ip: wazuhindexer2.domain.name
    - name: wazuhindexer3.domain.name
      ip: wazuhindexer3.domain.name

Otherwise certificates for nodes then would have wrong CN name in certificate and nodes wouldn't talk to each other properly (SSL handshake failure). I think tool for generating cert should be ready for this.

2. Configuration of opensearch.yml worked properly only when I put FQDN of nodes in 

network.host:
node.name: 
cluster.initial_master_nodes:
discovery.seed_hosts:

3. Initialization of cluster was possible with additional option of indexer-security-init.sh:

/usr/share/wazuh-indexer/bin/indexer-security-init.sh -ho wazuhindexer1.domain.name

Of course 1 line after executing was: Device "wazuhindexer1.ourdomian.name" does not exist.

But eventually whole cluster initialized properly and now we can see pretty green ;)

{
  "cluster_name" : "wazuh-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "discovered_master" : true,
  "active_primary_shards" : 1,
  "active_shards" : 3,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

Regards,

Robert.

[Carlos Ezequiel Bordon]

I was carrying out tests trying to replicate the environment that you mentioned in the first message and I confirm that it is not necessary for the names of the nodes to be the DNS names, what is necessary is that the data found in the config.yml and the opensearch.yml of each node match.
I share my configuration so that you can validate it:

---------------------------------------------------------------------------------

config.yml

nodes:
  # Wazuh indexer nodes
  indexer:
    - name: wazuhindexer1

      ip: wazuhindexer1.wazuh.com
    - name: wazuhindexer2
      ip: wazuhindexer2.wazuh.com
    - name: wazuhindexer3
      ip: wazuhindexer3.wazuh.com

---------------------------------------------------------------------------------

opensearch.yml:

network.host: "wazuhindexer1.wazuh.com"

node.name: "wazuhindexer1"
cluster.initial_master_nodes:
- "wazuhindexer1"
- "wazuhindexer2"
- "wazuhindexer3"
cluster.name: "wazuh-cluster"
discovery.seed_hosts:

  - "wazuhindexer1.wazuh.com"
  - "wazuhindexer2.wazuh.com"
  - "wazuhindexer3.wazuh.com"
node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexerplugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem

plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: falseplugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:

- "CN=wazuhindexer1,OU=Wazuh,O=Wazuh,L=California,C=US"
- "CN=wazuhindexer2,OU=Wazuh,O=Wazuh,L=California,C=US"
- "CN=wazuhindexer3,OU=Wazuh,O=Wazuh,L=California,C=US"

plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomaly-detection-state", ".opendistro-reports-*", ".opendistro-notifications-*", ".opendistro-notebooks", ".opensearch-observability", ".opendistro-asynchronous-search-response*", ".replication-metadata-store"]### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

---------------------------------------------------------------------------------

Request:

curl -k -u admin:admin https://wazuhindexer1.wazuh.com:9200
{
  "name" : "wazuhindexer1",
  "cluster_name" : "wazuh-cluster",
  "cluster_uuid" : "ZazPQ1JFT2GUiF511qejmQ",
  "version" : {
    "number" : "7.10.2",
    "build_type" : "rpm",
    "build_hash" : "e505b10357c03ae8d26d675172402f2f2144ef0f",
    "build_date" : "2022-01-14T03:38:06.881862Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

curl -k -u admin:admin https://wazuhindexer1.wazuh.com:9200/_cat/nodes?v
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.56.252           46          41   0    0.15    0.06     0.06 dimr      -      wazuhindexer3
192.168.56.253           43          57   0    0.00    0.03     0.06 dimr      -      wazuhindexer2
192.168.56.254           29          23   0    0.00    0.01     0.05 dimr      *      wazuhindexer1

[Robert A]

Ok, I managed to straighten things out. Thanks for your input. Do you think adding more info to documentation about -ho parameter and maybe examples of proper configuration of openserach.yml & config.yml would be possible? 

When deploying only with DNS names without IPs I think that might be helpful for future generations ;)

[Carlos Ezequiel Bordon]

Robert, I have created the following issue to contemplate this case in our documentation, thank you very much for your participation in the community

https://github.com/wazuh/wazuh-documentation/issues/5428

[Robert A]

Thank you :)

Cheers