Nodes are not joining after enabling searchguard

[gan...@customerlabs.co]

2 master and data nodes are joining in cluster if i disable the search guard. not connecting if i enable search guard.

Elasticsearch Version: 6.3.0

Search Guard Version: 6.3.0.

CentOS 7

java version "1.8.0_171"

Java(TM) SE Runtime Environment (build 1.8.0_171-b11)

Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)

Certificates are generated using Search Guard TLS Tool

Node1 config:

-----------------------------------------------------------------------------

cluster.name: cles

node.name: cl-esnode-1

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.6

discovery.zen.ping.unicast.hosts: ["31.239.124.150", "31.238.130.20"]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node1.pem

searchguard.ssl.transport.pemkey_filepath: node1.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

- CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

searchguard.authcz.admin_dn:

- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

searchguard.enterprise_modules_enabled: false

Node2 config:

---------------------------------------------------------------------

cluster.name: cles

node.name: cl-esnode-2

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

network.host: 10.240.0.9

discovery.zen.ping.unicast.hosts: ["31.239.124.150", "31.238.130.20"]

discovery.zen.minimum_master_nodes: 2

action.destructive_requires_name: true

xpack.security.enabled: false

searchguard.ssl.transport.pemcert_filepath: node2.pem

searchguard.ssl.transport.pemkey_filepath: node2.key

searchguard.ssl.transport.pemtrustedcas_filepath: root-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.resolve_hostname: false

searchguard.ssl.http.enabled: false

searchguard.nodes_dn:

- CN=node2.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

searchguard.authcz.admin_dn:

- CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

searchguard.enterprise_modules_enabled: false

[Jochen Kressin]

Please attach your Elasticsearch logfiles:

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version

* Installed and used enterprise modules, if any

* JVM version and operating system version

* Search Guard configuration files

* Elasticsearch log messages on debug level

* Other installed Elasticsearch or Kibana plugins, if any

[gan...@customerlabs.co]

I can not attach my logs here. Its being deleted automatically. Am i missing something?

thanks

[gan...@customerlabs.co]

Please find my partial es log

[2018-07-05T10:47:49,676][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin

[2018-07-05T10:47:50,119][INFO ][o.e.d.DiscoveryModule    ] [cl-esnode-1] using discovery type [zen]

[2018-07-05T10:47:51,229][INFO ][c.f.s.SearchGuardPlugin  ] 0 Search Guard modules loaded so far: []

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node               ] [cl-esnode-1] initialized

[2018-07-05T10:47:51,230][INFO ][o.e.n.Node               ] [cl-esnode-1] starting ...

[2018-07-05T10:47:51,388][INFO ][o.e.t.TransportService   ] [cl-esnode-1] publish_address {10.240.0.6:9300}, bound_addresses {10.240.0.6:9300}

[2018-07-05T10:47:51,444][INFO ][o.e.b.BootstrapChecks    ] [cl-esnode-1] bound or publishing to a non-loopback address, enforcing bootstrap checks

[2018-07-05T10:47:51,461][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ...

[2018-07-05T10:47:51,469][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [cl-esnode-1] no known master node, scheduling a retry

[2018-07-05T10:47:54,497][WARN ][o.e.d.z.ZenDiscovery     ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:47:57,501][WARN ][o.e.d.z.ZenDiscovery     ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:00,503][WARN ][o.e.d.z.ZenDiscovery     ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:03,506][WARN ][o.e.d.z.ZenDiscovery     ] [cl-esnode-1] not enough master nodes discovered during pinging (found [[Candidate{node={cl-esnode-1}{eo

PblikNReu7vQ4FIXOoig}{lQ4gjWt_QWqZMUWN_duEvQ}{10.240.0.6}{10.240.0.6:9300}{ml.machine_memory=7673548800, xpack.installed=true, ml.max_open_jobs=20, ml.enabled=true}

, clusterStateVersion=-1}]], but needed [2]), pinging again

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

On Thursday, July 5, 2018 at 3:06:57 PM UTC+5:30, Jochen Kressin wrote:

[Jochen Kressin]

Don't know about the deleted messages, we never delete anything here. Strange.

You see the TLS error here:

[2018-07-05T10:48:05,297][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [cl-esnode-1] SSL Problem Received fatal alert: certificate_unknown

javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

This means that the certificate could not be validated against the root CA. How were the certificates generated? 

If you have problems uploading the complete logs please send them to in...@search-guard.com

[gan...@customerlabs.co]

Certificates are generated using sgtlstool

Followed this steps: https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

./sgtlstool.sh -c ../config/tlsconfig.yml -ca -crt

Generated certificates on both nodes using same configuration

------tlsconnfig.yml------

###

### Self-generated certificate authority

###

#

# If you want to create a new certificate authority, you must specify its parameters here.

# You can skip this section if you only want to create CSRs

#

ca:

   root:

      dn: CN=root.ca.example.com,OU=CA,O=Example Com\, Inc.,DC=example,DC=com

      keysize: 2048

      pkPassword: changeit

      validityDays: 3650

      file: root-ca.pem

nodes:

  - name: node1

    dn: CN=node1.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

    dns: node1.example.com

    ip: 10.240.0.6

- name: node2

    dn: CN=node2.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

    dns: node2.example.com

    ip: 10.240.0.9

clients:

  - name: spock

    dn: CN=spock.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

  - name: kirk

    dn: CN=kirk.example.com,OU=Ops,O=Example Com\, Inc.,DC=example,DC=com

    admin: true

[gan...@customerlabs.co]

TLS error only happens if both nodes are running. No errors if i stop either of a node. Certificates are working fine on both the nodes independently but not with

discovery.zen.ping.unicast.hosts: ["31.239.124.150", "31.238.130.20"]

I have sent a logs to in...@search-guard.com as i can't attach here

Thanks

[Oumeyma JELLALI]

hello

i have a problem , please help me

Plugin [search-guard-6] was built for Elasticsearch version 6.2.4 but version 6.3.0 is running

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/85aac7b7-ef8b-4af8-b048-fff6ecc79b1d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Jochen Kressin]

Please do not hijack threads. This thread is about TLS errors.

I think the error message is quite clear. You have installed Search Guard 6.2.4 on Elasticsearch 6.3.0. Please read the documentation for installation instructions:

https://docs.search-guard.com/latest/search-guard-installation

"Replace the version number in the examples above with the exact version number that matches your Elasticsearch installation. A plugin built for Elasticsearch 6.3.0 will not run on Elasticsearch 6.2.4 and vice versa."

[gan...@customerlabs.co]

Hello Team,

Any finds on my issue please?

I think this is my case updated in the docs

https://docs.search-guard.com/latest/troubleshooting-tls#checking-the-ip-addresses-of-the-certificate

but i don't find any solution to fix this.

This issue holds my cluster moving in to production. Help on this would really much appreciated.

Thanks