Updating transport certificate and getting bad_certificate

321 views
Skip to first unread message

tim haak

unread,
Feb 16, 2018, 10:21:37 AM2/16/18
to Search Guard Community Forum
Hello,

We currently are using an open ssl self signed certificate for transport communication. Elasticsearch is set to use the java keystore and truststore. I have imported our internal CA cert into the truststore on each node in the cluster. Then I have generated a new keypair and CSR "note, I tried with the current and new keystore". I get the signed certificate from our internal CA and import into the keystore on 1 node. I restart Elasticssearch and I get bad_certificate. I have ensured the new certificate is valid, has both clientAuth and serverAuth extensions, and has a SAN entry for OID 1.2.3.4.5.5. The only difference I can see between the current working cert and new cert is the signature hash algorithm sha256 vs sha1. Could this be the issue? Is there documentation on how to renew or change a certificate?

* Search Guard and Elasticsearch version: 5.2.2-15
* Installed and used enterprise modules, if any: NA
* JVM version and operating system version: 1.8.0_131 and Debian 3.16.39-1
* Search Guard configuration files: sg_action_groups.yml sg_config.yml sg_internal_users.yml sg_roles_mapping.yml sg_roles.yml
* Elasticsearch log messages on debug level: true
* Other installed Elasticsearch or Kibana plugins, if any: repository and ec2 discovery

Thanks,
Tim Haak

Jochen Kressin

unread,
Feb 19, 2018, 6:04:17 AM2/19/18
to Search Guard Community Forum
Hi - could you please send the full elasticsearch logs on debug level when this exception happens?

Thanks.

Tim Haak

unread,
Feb 20, 2018, 9:13:18 AM2/20/18
to search...@googlegroups.com
Hi Jochen,

Here is the stacktrace. Looks like Java does not like the cert. 

[2018-02-09T21:44:11,103][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [hostname] SSL Problem Received fatal alert: bad_certificate
javax.net.ssl.SSLException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
        at io.netty.handler.ssl.SslHandler$SslEngineType$2.unwrap(SslHandler.java:218) ~[netty-handler-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1028) ~[netty-handler-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:950) ~[netty-handler-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248) ~[netty-codec-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:527) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:481) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441) [netty-transport-4.1.7.Final.jar:4.1.7.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.7.Final.jar:4.1.7.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]


I also verified the certificate chain with openssl verify on the node.

Thanks,
Tim

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/kk-JFp1WADY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/15ed86d7-2300-472b-b0e8-5701a1ec3bf5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Tim Haak

unread,
Feb 22, 2018, 11:24:36 AM2/22/18
to search...@googlegroups.com
[2018-02-22T16:12:00,914][INFO ][o.e.n.Node               ] [hostname] initializing ...
[2018-02-22T16:12:01,191][INFO ][o.e.e.NodeEnvironment    ] [hostname] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [44.3gb], net total_space [49gb], spins? [unknown], types [rootfs]
[2018-02-22T16:12:01,203][INFO ][o.e.e.NodeEnvironment    ] [hostname] heap size [1.8gb], compressed ordinary object pointers [true]
[2018-02-22T16:12:01,433][INFO ][o.e.n.Node               ] [hostname] node name [hostname], node ID [b8rQMLcATUK__qHcs84u7w]
[2018-02-22T16:12:01,445][INFO ][o.e.n.Node               ] [hostname] version[5.2.2], pid[25039], build[f9d9b74/2017-02-24T17:26:45.835Z], OS[Linux/3.16.0-4-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2018-02-22T16:12:06,523][INFO ][c.f.s.SearchGuardPlugin  ] Clustername: clustername
[2018-02-22T16:12:06,631][INFO ][c.f.s.SearchGuardPlugin  ] Node [hostname] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2018-02-22T16:12:06,632][INFO ][c.f.s.SearchGuardPlugin  ] FLS/DLS module not available
[2018-02-22T16:12:06,702][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL
[2018-02-22T16:12:06,704][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.version: 1.8.0_131
[2018-02-22T16:12:06,704][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vendor: Oracle Corporation
[2018-02-22T16:12:06,705][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.version: 1.8
[2018-02-22T16:12:06,706][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.vendor: Oracle Corporation
[2018-02-22T16:12:06,706][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.name: Java Virtual Machine Specification
[2018-02-22T16:12:06,707][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.name: OpenJDK 64-Bit Server VM
[2018-02-22T16:12:06,707][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.vendor: Oracle Corporation
[2018-02-22T16:12:06,707][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.version: 1.8
[2018-02-22T16:12:06,708][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.vendor: Oracle Corporation
[2018-02-22T16:12:06,708][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.name: Java Platform API Specification
[2018-02-22T16:12:06,709][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.name: Linux
[2018-02-22T16:12:06,709][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.arch: amd64
[2018-02-22T16:12:06,710][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.version: 3.16.0-4-amd64
[2018-02-22T16:12:08,110][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]
[2018-02-22T16:12:08,360][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]
[2018-02-22T16:12:08,416][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2018-02-22T16:12:08,457][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the firs one: 20180222
[2018-02-22T16:12:08,473][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias 20180222 contains a root certificate
[2018-02-22T16:12:08,849][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2018-02-22T16:12:08,849][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2018-02-22T16:12:08,849][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:null with ciphers []
[2018-02-22T16:12:08,849][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2018-02-22T16:12:08,850][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2018-02-22T16:12:11,004][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [aggs-matrix-stats]
[2018-02-22T16:12:11,023][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [ingest-common]
[2018-02-22T16:12:11,023][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [lang-expression]
[2018-02-22T16:12:11,024][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [lang-groovy]
[2018-02-22T16:12:11,024][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [lang-mustache]
[2018-02-22T16:12:11,025][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [lang-painless]
[2018-02-22T16:12:11,025][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [percolator]
[2018-02-22T16:12:11,025][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [reindex]
[2018-02-22T16:12:11,026][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [transport-netty3]
[2018-02-22T16:12:11,039][INFO ][o.e.p.PluginsService     ] [hostname] loaded module [transport-netty4]
[2018-02-22T16:12:11,040][INFO ][o.e.p.PluginsService     ] [hostname] loaded plugin [discovery-ec2]
[2018-02-22T16:12:11,042][INFO ][o.e.p.PluginsService     ] [hostname] loaded plugin [repository-s3]
[2018-02-22T16:12:11,042][INFO ][o.e.p.PluginsService     ] [hostname] loaded plugin [search-guard-5]
[2018-02-22T16:12:14,514][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: org.apache.tomcat.jni.SSL
[2018-02-22T16:12:14,514][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.version: 1.8.0_131
[2018-02-22T16:12:14,515][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vendor: Oracle Corporation
[2018-02-22T16:12:14,516][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.version: 1.8
[2018-02-22T16:12:14,516][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.vendor: Oracle Corporation
[2018-02-22T16:12:14,516][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.name: Java Virtual Machine Specification
[2018-02-22T16:12:14,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.name: OpenJDK 64-Bit Server VM
[2018-02-22T16:12:14,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.vendor: Oracle Corporation
[2018-02-22T16:12:14,518][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.version: 1.8
[2018-02-22T16:12:14,518][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.vendor: Oracle Corporation
[2018-02-22T16:12:14,527][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.name: Java Platform API Specification
[2018-02-22T16:12:14,527][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.name: Linux
[2018-02-22T16:12:14,528][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.arch: amd64
[2018-02-22T16:12:14,528][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.version: 3.16.0-4-amd64
[2018-02-22T16:12:14,599][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]
[2018-02-22T16:12:14,739][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]
[2018-02-22T16:12:14,741][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2018-02-22T16:12:14,743][INFO ][c.f.s.s.u.SSLCertificateHelper] No alias given, use the firs one: 20180222
[2018-02-22T16:12:14,743][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias 20180222 contains a root certificate
[2018-02-22T16:12:14,777][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2018-02-22T16:12:14,809][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2018-02-22T16:12:14,809][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:null with ciphers []
[2018-02-22T16:12:14,809][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2018-02-22T16:12:14,809][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2018-02-22T16:12:17,522][DEBUG][o.e.a.ActionModule       ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin
[2018-02-22T16:12:17,676][INFO ][c.f.s.SearchGuardPlugin  ] FLS/DLS valve not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.DlsFlsValveImpl
[2018-02-22T16:12:17,677][INFO ][c.f.s.SearchGuardPlugin  ] Auditlog not available due to java.lang.ClassNotFoundException: com.floragunn.searchguard.auditlog.impl.AuditLogImpl
[2018-02-22T16:12:17,695][INFO ][c.f.s.SearchGuardPlugin  ] Privileges interceptor not bound (noop) due to java.lang.ClassNotFoundException: com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl
[2018-02-22T16:12:22,408][INFO ][o.e.n.Node               ] [hostname] initialized
[2018-02-22T16:12:22,422][INFO ][o.e.n.Node               ] [hostname] starting ...
[2018-02-22T16:12:22,649][WARN ][i.n.u.i.MacAddressUtil   ] Failed to find a usable hardware address from the network interfaces; using random bytes: 6b:45:a3:14:56:1e:5d:b4
[2018-02-22T16:12:22,870][INFO ][o.e.t.TransportService   ] [hostname] publish_address {ipaddress:9300}, bound_addresses {[::]:9300}
[2018-02-22T16:12:22,901][INFO ][o.e.b.BootstrapChecks    ] [hostname] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2018-02-22T16:12:22,961][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ...
[2018-02-22T16:12:22,967][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [hostname] no known master node, scheduling a retry
[2018-02-22T16:12:27,670][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [hostname] SSL Problem Received fatal alert: bad_certificate
[2018-02-22T16:12:27,954][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [hostname] SSL Problem Received fatal alert: bad_certificate
[2018-02-22T16:12:28,418][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [hostname] SSL Problem Received fatal alert: bad_certificate
[2018-02-22T16:12:28,806][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [hostname] SSL Problem Received fatal alert: bad_certificate
[2018-02-22T16:12:29,215][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [hostname] SSL Problem Received fatal alert: bad_certificate
[2018-02-22T16:12:29,369][INFO ][o.e.c.s.ClusterService   ] [hostname] new_master {hostname}{b8rQMLcATUK__qHcs84u7w}{Wm0EEWbJQq2Nza2lWwAtRQ}{ipaddress}{ipaddress:9300}{aws_availability_zone=us-east-1e}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2018-02-22T16:12:29,553][INFO ][o.e.h.HttpServer         ] [hostname] publish_address {ipaddress:9200}, bound_addresses {[::]:9200}
[2018-02-22T16:12:29,571][INFO ][o.e.n.Node               ] [hostname] started
[2018-02-22T16:12:33,450][ERROR][c.f.s.a.BackendRegistry  ] Not yet initialized (you may need to run sgadmin)
[2018-02-22T16:12:33,628][ERROR][c.f.s.a.BackendRegistry  ] Not yet initialized (you may need to run sgadmin)
[2018-02-22T16:12:34,040][INFO ][o.e.g.GatewayService     ] [hostname] recovered [14] indices into cluster_state
[2018-02-22T16:12:37,549][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'hostname' initialized


Thanks,
Tim

Jochen Kressin

unread,
Feb 23, 2018, 12:35:25 PM2/23/18
to Search Guard Community Forum
Sorry for the delay, but at the moment I'm at a loss here. The signature hash algorithm should not be a problem IMHO. "bad certificate" usually means the certificate cannot be validated against the certificate in the truststore. Are you sure both certificates, means the working and the non-working one, have been signed by the same CA? Maybe your PKI uses intermediate certificates and they have changed?

You can verify that for example by looking at the Authority Key Identifier field of your certificates. If they are signed by the same CA, they should be equal.
...

Tim Haak

unread,
Feb 23, 2018, 1:18:10 PM2/23/18
to search...@googlegroups.com
Hi Jochen,

No worries on the delay. The certs are not signed by the same CA. I am trying to use a new internal CA. I have ensured both CA certs are in the truststore. Is it not possible to have more than 1 CA cert in the truststore? I wanted to make sure both CAs are in the truststore so nodes with the older certificate will still be able to communicate with each other. Is this the wrong way to go about swapping out the certificate on each node? I am trying to avoid a complete cluster shutdown and preform a rolling certificate update. 

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/kk-JFp1WADY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.

Jochen Kressin

unread,
Feb 26, 2018, 1:49:57 PM2/26/18
to search...@googlegroups.com
Hi Tim,

now I understand what you are trying to do, sorry, I was misled somehow. Yes, the rollover scenario is generally correct, and I've tested it again today. I'm not 100% sure what you did on the nodes in the cluster but here are the steps:

Let's say you have ca_1 in the truststore which signed all node certificates node1_1, node2_1, node 3_1 and so on. You now want to swap ca_1 for ca_2.

You would import ca_2 in your existing truststore, so it contains both ca_1 and ca_2. You need to replace the truststore on all nodes in your cluster now. That's the first rolling restart. Now all your nodes trust both ca_1 and ca_2.

Now you can start changing the node certificates. You take down node1 and replace the existing node1_1 certificate with node1_2, which is signed by ca_2. This node can now join the cluster because all other nodes trust ca_2. This is the second rolling restart.

If you now want to eliminate the unused ca_1 from all nodes after all certs have been swapped, remove it from the truststores on all nodes. This is the third rolling restart.

In other words, make sure you change the truststores on all nodes first, and then start changing the node certificates.

Hope that helps.
...

Tim Haak

unread,
Feb 27, 2018, 1:11:16 PM2/27/18
to search...@googlegroups.com
Thanks Jochen!

On Mon, Feb 26, 2018 at 1:49 PM, Jochen Kressin <jkre...@floragunn.com> wrote:
Hi Tim,

now I understand what you are trying to do, sorry, I was misled somehow. Yes, the rollover scenario is generally correct, and I've tested it again today. I'm not 100% sure what you did on the nodes in the cluster but here are the steps:

Let's say you want have ca_1 in the truststore which signed all node certificates node1_1, node2_1, node 3_1 and so on. You now want to swap ca_1 for ca_2.

You would now import ca_2 in your truststore, so it contains both ca_1 and ca_2. You need to replace the truststore on all nodes in your cluster now. That's the first rolling restart. Now all your nodes trust both ca_1 and ca_2.

Now you can start to change the node certificates. You take down node1 and replace the existing node1_1 certificate with node1_2, which is signed by ca_2. This node can now join the cluster, because all other nodes trust ca_2. This is the second rolling restart.

If you now want to eliminate the unused ca_1 from all nodes, remove it from the truststores on all nodes. This is the third rolling restart.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/kk-JFp1WADY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages