Searchguard nodes not joining cluster?

[Sami Yessou]

Hello,
I'm trying to allow my other nodes join the Elasticsearch cluster protected with searchguard but when node-2 tries to join the cluster i get this errors(instead of my local domain there is [mylocaldomain]):

[2016-05-07 16:38:59,695][DEBUG][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [node-2] disconnecting from [{#zen_unicast_1#}{192.168.0.21}{host-192-168-0-21.[mylocaldomain].local/192.168.0.21:9300}], channel closed event
[2016-05-07 16:39:01,174][DEBUG][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [node-2] connected to node [{#zen_unicast_1#}{192.168.0.21}{host-192-168-0-21.[mylocaldomain].local/192.168.0.21:9300}]
[2016-05-07 16:39:01,193][WARN ][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [node-2] exception caught on transport layer [[id: 0x32507a86, /192.168.0.31:50464 => host-192-168-0-21.[mylocaldomain].local/192.168.0.21:9300]], closing connection
javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431)
    at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
    at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:919)
    at sun.security.ssl.Handshaker$1.run(Handshaker.java:916)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
    at org.jboss.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1392)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1255)
    ... 18 more
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching host-192-168-0-21.[mylocaldomain].local found.
    at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:204)
    at sun.security.util.HostnameChecker.match(HostnameChecker.java:95)
    at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:455)
    at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:436)
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:252)
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)
    ... 26 more
[2016-05-07 16:39:01,196][DEBUG][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [node-2] disconnecting from [{#zen_unicast_1#}{192.168.0.21}{host-192-168-0-21.[mylocaldomain].local/192.168.0.21:9300}], channel closed event

And on the node 1 logs i see:

[2016-05-07 16:39:01,137][WARN ][com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] [node-1] exception caught on transport layer [[id: 0x2467bc1a, /192.168.0.31:50464 => /192.168.0.21:9300]], closing connection
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
    at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
    at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
    at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
    at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
    at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
    at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
    at org.jboss.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1218)
    at org.jboss.netty.handler.ssl.SslHandler.decode(SslHandler.java:852)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:425)
    at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

I have generated the certificates with example-pki-scripts/example.sh on node-1 then i copied the node-2-keystore.jks and trustore.jks on node-2 sgconfig , executed the sgadmin.sh to commit(no errors) the configuration and then restarted ES.
I am missing something to make the nodes joining?i have tried to edit /etc/hosts if something changes for host-192-168-0-21.[mylocaldomain].local found but nothing.
The searchguard configuration on ES(node-2) is this:

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/node-2-keystore.jks

searchguard.ssl.transport.truststore_filepath: /usr/share/elasticsearch/plugins/search-guard-2/sgconfig/truststore.jks

logger.com.floragunn.searchguard.ssl: DEBUG

discovery.zen.ping.unicast.hosts: ["192.168.0.21"]
security.manager.enabled: false

searchguard.authcz.admin_dn:
  - "CN=node-2.example.com, OU=SSL, O=Test, L=Test, C=DE"
                                                       

Now both ES nodes are working but out of the cluster.

[SG]

try

searchguard.ssl.transport.enforce_hostname_verification: true

or

include host-192-168-0-21.[mylocaldomain].local in the san list in your certificates
(https://github.com/floragunncom/search-guard-ssl/blob/master/example-pki-scripts/gen_node_cert.sh line 35 and 49)

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/fa060dec-c82f-409e-91d1-f93ed44a2b0e%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[SG]

sorry, meant

searchguard.ssl.transport.enforce_hostname_verification: false

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/563CF595-567B-4720-8B57-A925CBCADF9A%40search-guard.com.

[Sami Yessou]

Thank you, now other nodes joined the cluster after i disabled hostname verification :), disabling dns verification won't be a security issue due to the truststore certificate verification?
My last question is, If i set this options on a node:
node.master: false
node.data: false
From the elasticsearch documentation i see that a client node can be used as a loadbalancer, do you know if SG have some performance impact when multiple requests come at the same time from different hosts?

[SG]

SG should have no significant performance impact

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c0ba9968-2e39-4bf1-a4c8-879bfb2f93d7%40googlegroups.com.