search-guard-2 not working with ES 2.3.1

[Guillaume Perréal]

Hello there,

I have succesfully set up a cluster of 3 nodes (elise1-elise3) with search-guard-ssl, but when it comes to search-guard-2, something fails with the certificates.

The server gives me this error at start (host names have been removed/changed in this dump):

com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1t  3 May 2016 available
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,238][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1t  3 May 2016 available
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,239][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[setting dump removed]
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,241][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Effective settings:
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,346][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Transport keystore subject DN no. 0 [removed for privacy]
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,347][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Transport keystore subject DN no. 1 CN=TERENA SSL CA 3, O=TERENA, L=Amsterdam, ST=Noord-Holland, C=NL
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,348][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Transport keystore subject DN no. 2 CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,357][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTP client auth mode OPTIONAL
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,366][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTP keystore subject DN no. 0 [removed for privacy]
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,367][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTP keystore subject DN no. 1 CN=TERENA SSL CA 3, O=TERENA, L=Amsterdam, ST=Noord-Holland, C=NL
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,367][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTP keystore subject DN no. 2 CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=US
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,784][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,785][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
juin 14 10:50:24 elise1 elasticsearch[999]: [2016-06-14 10:50:24,785][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
juin 14 10:50:25 elise1 elasticsearch[999]: [2016-06-14 10:50:25,106][INFO ][http                     ] [elise1] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by [search-guard-ssl]
juin 14 10:50:25 elise1 elasticsearch[999]: [2016-06-14 10:50:25,314][INFO ][transport                ] [elise1] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service, overridden by [search-guard2]
juin 14 10:50:25 elise1 elasticsearch[999]: [2016-06-14 10:50:25,315][INFO ][transport                ] [elise1] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
juin 14 10:50:27 elise1 elasticsearch[999]: [2016-06-14 10:50:27,727][INFO ][node                     ] [elise1] initialized
juin 14 10:50:27 elise1 elasticsearch[999]: [2016-06-14 10:50:27,731][INFO ][node                     ] [elise1] starting ...
juin 14 10:50:27 elise1 elasticsearch[999]: [2016-06-14 10:50:27,830][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [elise1] publish_address {10.69.192.153:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {10.69.192.153:9300}
juin 14 10:50:27 elise1 elasticsearch[999]: [2016-06-14 10:50:27,838][INFO ][discovery                ] [elise1] elise.example.com/dhIpumhbQg6i_D45-qvOUw
juin 14 10:50:27 elise1 elasticsearch[999]: [2016-06-14 10:50:27,863][DEBUG][action.admin.cluster.health] [elise1] no known master node, scheduling a retry
juin 14 10:50:31 elise1 elasticsearch[999]: [2016-06-14 10:50:31,530][INFO ][cluster.service          ] [elise1] detected_master {elise2}{Cjv2-hc1T-qKpMZ75-Vnww}{10.69.192.154}{10.69.192.154:9300}, added {{elise2}{Cjv2-hc1T-qKpMZ75-Vnww}{10.69.192.154}{10.69.192.154:9300},{elise3}{HEBJdqcQTRmJNvGPiijo7g}{10.69.192.155}{10.69.192.155:9300},}, reason: zen-disco-receive(from master [{elise2}{Cjv2-hc1T-qKpMZ75-Vnww}{10.69.192.154}{10.69.192.154:9300}])
juin 14 10:50:31 elise1 elasticsearch[999]: Exception in thread "Thread-4" ElasticsearchSecurityException[No SSL client certificates found. Search Guards needs the Search Guard SSL plugin to be installed]
juin 14 10:50:31 elise1 elasticsearch[999]: at com.floragunn.searchguard.transport.SearchGuardTransportService.messageReceivedDecorate(SearchGuardTransportService.java:204)
juin 14 10:50:31 elise1 elasticsearch[999]: at com.floragunn.searchguard.ssl.transport.SearchGuardSSLTransportService$Interceptor.messageReceived(SearchGuardSSLTransportService.java:85)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:245)
juin 14 10:50:31 elise1 elasticsearch[999]: at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.handleRequest(SearchGuardMessageChannelHandler.java:57)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:114)
juin 14 10:50:31 elise1 elasticsearch[999]: at com.floragunn.searchguard.ssl.transport.SearchGuardMessageChannelHandler.messageReceived(SearchGuardMessageChannelHandler.java:45)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
juin 14 10:50:31 elise1 elasticsearch[999]: at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
juin 14 10:50:31 elise1 elasticsearch[999]: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
juin 14 10:50:31 elise1 elasticsearch[999]: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
juin 14 10:50:31 elise1 elasticsearch[999]: at java.lang.Thread.run(Thread.java:745)
juin 14 10:50:31 elise1 elasticsearch[999]: [2016-06-14 10:50:31,977][INFO ][http                     ] [elise1] publish_address {10.69.192.153:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.69.192.153:9200}
juin 14 10:50:31 elise1 elasticsearch[999]: [2016-06-14 10:50:31,980][INFO ][node                     ] [elise1] started

And when trying to use the sgadmin tool, I get :

java -cp '/usr/share/elasticsearch/plugins/search-guard-ssl/*:/usr/share/elasticsearch/plugins/search-guard-2/*:/usr/share/elasticsearch/lib/*' com.floragunn.searchguard.tools.SearchGuardAdmin -ks /etc/elasticsearch/ssl/sgadmin.jks -kspass SOMEPASS -ts /etc/elasticsearch/ssl/truststore.jks -tspass SOMEPASS -cd /etc/elasticsearch/search-guard -h elise1.lyon.cemagref.fr -p 9300 -cn MYCLUSTERNAME
Connect to elise1.example.com:9300
Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{10.69.192.153}{elise1.example.com/10.69.192.153:9300}]]
    at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:290)
    at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:207)
    at org.elasticsearch.client.transport.support.TransportProxyClient.execute(TransportProxyClient.java:55)
    at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:288)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:359)
    at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:348)
    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:848)
    at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:868)
    at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:137)

The plugins seem to be correctly installed.

/usr/share/elasticsearch/bin/plugin list
Installed plugins in /usr/share/elasticsearch/plugins:
    - search-guard-2
    - head
    - search-guard-ssl

All the certificates (one per node for transport, one for http, and one for the sgadmin user) have been signed by the same CA.

Any idea about these errors ?

Regards,
Guillaume Perréal.

[SG]

Can you please upgrade to ES 2.3.3 and SG 2.3.3.0-rc1 and SG SSL 2.3.3.11 and try again?

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/779874c3-ce9c-4ea7-8a29-d4a061e21f08%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Guillaume Perréal]

2.3.3 gave a more precise error. It turns out it was an issue with the node certificates. This is now fixed.