UnicastZenPing failing due to a "bad header". Any hints as to how I can locate this header and fix?
268 views
Skip to first unread message
Steve Haertel
Jul 12, 2017, 10:11:28 AM7/12/17
to Search Guard
Has anyone encountered this problem before?
<pre>
[2017-07-12T09:28:26,518][WARN ][o.e.d.z.ZenDiscovery ] [client_hostname.domain] not enough master nodes discovered during pinging (found [[]], but needed [1]), pinging again
[2017-07-12T09:28:29,184][WARN ][o.e.d.z.UnicastZenPing ] [client_hostname.domain] [3] failed send ping to {#zen_unicast_hostname.domain:9301_0#}{79tDJstFSJ-enaTA4DiWLw}{hostname.domain}{XXX.XXX.XXX.XXX:9301}
java.lang.IllegalStateException: handshake failed with {#zen_unicast_hostname.domain:9301_0#}{79tDJstFSJ-enaTA4DiWLw}{hostname.domain}{XXX.XXX.XXX.XXX:9301}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:386) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:353) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:401) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:508) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.2.jar:5.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
Caused by: org.elasticsearch.transport.RemoteTransportException: [master_hostname.domain][XXX.XXX.XXX.XXX:9301][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchException: bad header found
at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:158) ~[?:?]
at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:139) ~[?:?]
at com.floragunn.searchguard.SearchGuardPlugin$2$1.messageReceived(SearchGuardPlugin.java:338) ~[?:?]
</pre>
================= elasticsearch.yml ================= searchguard.ssl.transport.enabled: true searchguard.ssl.transport.keystore_filepath: /PATH/serverKeyStore.jks searchguard.ssl.transport.truststore_filepath: /PATH/serverTrustStore.jks searchguard.ssl.http.enabled: true searchguard.ssl.http.keystore_filepath: /PATH/serverKeyStore.jks searchguard.ssl.http.truststore_filepath: /PATH/serverTrustStore.jks searchguard.ssl.transport.enforce_hostname_verification: false searchguard.ssl.transport.keystore_password: XXX searchguard.ssl.transport.truststore_password: XXX searchguard.ssl.http.keystore_password: XXX searchguard.ssl.http.truststore_password: XXX searchguard.ssl.transport.keystore_alias: srvalias searchguard.ssl.transport.truststore_alias: srvalias searchguard.ssl.http.keystore_alias: srvalias searchguard.ssl.http.truststore_alias: srvalias searchguard.ssl.http.enabled_protocols: - "TLSv1" - "TLSv1.1" - "TLSv1.2"
======================= Env details ======================= search-guard-5 elasticsearch-5.4.2 OpenJDK Red Hat Enterprise Linux Server release 7.3 (Maipo) 1-node
Steve Haertel
Jul 12, 2017, 10:44:25 AM7/12/17
to Search Guard
Update:
The above error was from my client log. After looking at the master log, I see this error:
The above error was from my client log. After looking at the master log, I see this error:
[2017-07-12T10:37:57,673][ERROR][c.f.s.t.SearchGuardRequestHandler] Error validating headers
[2017-07-12T10:37:58,359][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [master_hostname.domain] SSL Problem null cert chain
javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1431) ~[?:?]
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_102]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
Caused by: javax.net.ssl.SSLHandshakeException: null cert chain
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:292) ~[?:?]
at sun.security.ssl.ServerHandshaker.clientCertificate(ServerHandshaker.java:1865) ~[?:?]
at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:230) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:919) ~[?:?]
at sun.security.ssl.Handshaker$1.run(Handshaker.java:916) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_102]
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369) ~[?:?]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1295) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1208) ~[?:?]
... 18 more
Steve Haertel
Jul 12, 2017, 11:05:09 AM7/12/17
to Search Guard
Update again...
I was looking at the source code, seeing where the error message about the bad headers was called, and I see this:
| try { | |
| HeaderHelper.checkSGHeader(request); | |
| } catch (Exception e) { | |
| auditLog.logBadHeaders(request); | |
| log.error("Error validating headers "+e, e); | |
| transportChannel.sendResponse(ExceptionsHelper.convertToElastic(e)); | |
| return; | |
| } |
I then searched for that "checkSGHeader" method, and I saw that it thows an exception if there is a "_sg_" at the start of the header. I also noticed that it does an audit of a "logBadHeaders", which, I see on this page, https://github.com/floragunncom/search-guard-docs/blob/master/auditlogging.md, --->
- BAD_HEADERS—an attempt was made to spoof a request to Elasticsearch with Search Guard internal headers.
Ok. Now I believe I can see what is happening -- the next step is to try to figure out why this is happening, and think of a way to fix it. Anybody know?
SG
Jul 12, 2017, 11:27:36 AM7/12/17
to search...@googlegroups.com
"javax.net.ssl.SSLHandshakeException: null cert chain" points to a SSL/cert misconfiguration and is likely the root cause for your problem.
I strongly recommend to use https://floragunn.com/tls-certificate-generator/ to get it up and running and then switch to your own certificates.
Especially if you are not familiar with SSL/TLS you spend a lot of time getting this to work before you can explore the functionality and features.
Alternatively use https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2c4ad824-c74f-4b18-924b-d4cf577c9953%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
I strongly recommend to use https://floragunn.com/tls-certificate-generator/ to get it up and running and then switch to your own certificates.
Especially if you are not familiar with SSL/TLS you spend a lot of time getting this to work before you can explore the functionality and features.
Alternatively use https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/2c4ad824-c74f-4b18-924b-d4cf577c9953%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Steve Haertel
Jul 12, 2017, 2:18:28 PM7/12/17
to Search Guard
Hmm yeah. Actually this may sound weird, but exploring the function/features isn't my goal. Getting my certificates to do the same thing as the generated ones (which I did get!) is my primary objective here. I have to figure out just what the heck are the differences between my keystore certs and the generated ones....
Steve Haertel
Jul 12, 2017, 3:41:15 PM7/12/17
to Search Guard
We can narrow down what the problem is at least...
I went ahead and used the generated certificates in place of my own, but I ended up with the same error. So at least we know the certificates themselves are not the problem. Probably some config problem?
Anyone, help? :(
======================================
logger.com.floragunn.searchguard.ssl: DEBUG
searchguard.ssl.http.clientauth_mode: REQUIRE
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.transport.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.http.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_password: kpwd
searchguard.ssl.transport.truststore_password: tpwd
searchguard.ssl.http.keystore_password: kpwd
searchguard.ssl.http.truststore_password: tpwd
searchguard.ssl.transport.keystore_alias: cn=hostname.domain
searchguard.ssl.transport.truststore_alias: root-ca-chain
searchguard.ssl.http.keystore_alias: cn=hostname.domain
searchguard.ssl.http.truststore_alias: root-ca-chain
searchguard.ssl.transport.enabled_protocols:
- "TLSv1.2"
searchguard.ssl.http.enabled_protocols:
- "TLSv1.2"
==============================================
[2017-07-12T15:32:22,922][INFO ][o.e.n.Node ] [master_hostname.domain] initializing ...
[2017-07-12T15:32:23,437][INFO ][o.e.e.NodeEnvironment ] [master_hostname.domain] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [6.3gb], net total_space [16.9gb], spins? [unknown], types [rootfs]
[2017-07-12T15:32:23,437][INFO ][o.e.e.NodeEnvironment ] [master_hostname.domain] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-12T15:32:23,520][INFO ][o.e.n.Node ] [master_hostname.domain] node name [master_hostname.domain], node ID [jH94B8oWQTiKrNfu-3eQuA]
[2017-07-12T15:32:23,520][INFO ][o.e.n.Node ] [master_hostname.domain] version[5.4.2], pid[6068], build[929b078/2017-06-15T02:29:28.122Z], OS[Linux/3.10.0-514.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_102/25.102-b14]
[2017-07-12T15:32:23,521][INFO ][o.e.n.Node ] [master_hostname.domain] JVM arguments [-Xms2g, -Xmx2g, -Djavax.net.ssl.trustStore=/PATH/serverTrustStore.jks, -Djavax.net.ssl.trustAnchors=/PATH/serverTrustStore.jks, -Djavax.net.ssl.keyStore=/PATH/serverKeyStore.jks, -Djavax.net.debug=ssl:handshake, -Dlog4j2.disable.jmx=true, -Djava.security.policy=/PATH/plugin-security.policy, -Des.path.home=/PATH]
[2017-07-12T15:32:34,739][INFO ][c.f.s.SearchGuardPlugin ] Clustername: elk-CwS2.2.1-0705
[2017-07-12T15:32:34,989][INFO ][c.f.s.SearchGuardPlugin ] Node [master_hostname.domain] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2017-07-12T15:32:35,101][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.version: 1.8.0_102
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vendor: Oracle Corporation
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.version: 1.8
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.vendor: Oracle Corporation
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.specification.name: Java Virtual Machine Specification
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.name: OpenJDK 64-Bit Server VM
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.vm.vendor: Oracle Corporation
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.version: 1.8
[2017-07-12T15:32:35,102][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.vendor: Oracle Corporation
[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] java.specification.name: Java Platform API Specification
[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.name: Linux
[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.arch: amd64
[2017-07-12T15:32:35,103][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] os.version: 3.10.0-514.el7.x86_64
[2017-07-12T15:32:36,835][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for https [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]
[2017-07-12T15:32:36,875][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] JVM supports the following 82 ciphers for transport [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDH_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDH_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA, SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV, TLS_DH_anon_WITH_AES_256_GCM_SHA384, TLS_DH_anon_WITH_AES_128_GCM_SHA256, TLS_DH_anon_WITH_AES_256_CBC_SHA256, TLS_ECDH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_256_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA256, TLS_ECDH_anon_WITH_AES_128_CBC_SHA, TLS_DH_anon_WITH_AES_128_CBC_SHA, TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA, SSL_DH_anon_WITH_3DES_EDE_CBC_SHA, SSL_RSA_WITH_DES_CBC_SHA, SSL_DHE_RSA_WITH_DES_CBC_SHA, SSL_DHE_DSS_WITH_DES_CBC_SHA, SSL_DH_anon_WITH_DES_CBC_SHA, SSL_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA, SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA, SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA, TLS_RSA_WITH_NULL_SHA256, TLS_ECDHE_ECDSA_WITH_NULL_SHA, TLS_ECDHE_RSA_WITH_NULL_SHA, SSL_RSA_WITH_NULL_SHA, TLS_ECDH_ECDSA_WITH_NULL_SHA, TLS_ECDH_RSA_WITH_NULL_SHA, TLS_ECDH_anon_WITH_NULL_SHA, SSL_RSA_WITH_NULL_MD5, TLS_KRB5_WITH_3DES_EDE_CBC_SHA, TLS_KRB5_WITH_3DES_EDE_CBC_MD5, TLS_KRB5_WITH_DES_CBC_SHA, TLS_KRB5_WITH_DES_CBC_MD5, TLS_KRB5_EXPORT_WITH_DES_CBC_40_SHA, TLS_KRB5_EXPORT_WITH_DES_CBC_40_MD5]
[2017-07-12T15:32:36,877][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /PATH/integration/elk/conf/elasticsearch/, from there the key- and truststore files are resolved relatively
[2017-07-12T15:32:36,885][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2017-07-12T15:32:36,885][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true
[2017-07-12T15:32:36,886][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3
[2017-07-12T15:32:36,886][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:36,888][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 -> false
[2017-07-12T15:32:36,889][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 -> true
[2017-07-12T15:32:36,889][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:36,889][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias cn=hostname.domain contains a root certificate
[2017-07-12T15:32:36,890][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2017-07-12T15:32:36,890][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true
[2017-07-12T15:32:36,891][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3
[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 -> false
[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 -> true
[2017-07-12T15:32:36,983][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:36,996][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2017-07-12T15:32:36,996][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: is a certificate entry?true/is a key entry?false
[2017-07-12T15:32:36,996][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: single cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 -> true
[2017-07-12T15:32:37,170][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] HTTPS client auth mode REQUIRE
[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true
[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3
[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:37,234][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 -> false
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 -> true
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:37,235][WARN ][c.f.s.s.u.SSLCertificateHelper] Certificate chain for alias cn=hostname.domain contains a root certificate
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: is a certificate entry?false/is a key entry?true
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: chain len 3
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:37,235][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Signing CA, OU=ORGANIZATION Canada Signing CA, O=ORGANIZATION Canada of type 0 -> false
[2017-07-12T15:32:37,236][DEBUG][c.f.s.s.u.SSLCertificateHelper] cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 -> true
[2017-07-12T15:32:37,236][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias cn=hostname.domain: single cert CN=hostname.domain of type -1 -> false
[2017-07-12T15:32:37,237][DEBUG][c.f.s.s.u.SSLCertificateHelper] Keystore has 1 entries/aliases
[2017-07-12T15:32:37,237][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: is a certificate entry?true/is a key entry?false
[2017-07-12T15:32:37,237][DEBUG][c.f.s.s.u.SSLCertificateHelper] Alias root-ca-chain: single cert CN=ORGANIZATION Canada Root CA, OU=ORGANIZATION Canada Root CA, O=ORGANIZATION Canada of type 2147483647 -> true
[2017-07-12T15:32:37,245][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportClientProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2017-07-12T15:32:37,245][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransportServerProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2017-07-12T15:32:37,246][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTPProvider:JDK with ciphers [TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256]
[2017-07-12T15:32:37,246][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslTransport protocols [TLSv1.2]
[2017-07-12T15:32:37,246][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] sslHTTP protocols [TLSv1.2]
[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [aggs-matrix-stats]
[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [ingest-common]
[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-expression]
[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-groovy]
[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-mustache]
[2017-07-12T15:32:37,249][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [lang-painless]
[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [percolator]
[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [reindex]
[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [transport-netty3]
[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded module [transport-netty4]
[2017-07-12T15:32:37,250][INFO ][o.e.p.PluginsService ] [master_hostname.domain] loaded plugin [search-guard-5]
[2017-07-12T15:32:47,316][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin
[2017-07-12T15:32:47,651][INFO ][c.f.s.a.BackendRegistry ] Register EgoAuthenticationBackend()
[2017-07-12T15:32:47,930][INFO ][o.e.d.DiscoveryModule ] [master_hostname.domain] using discovery type [zen]
[2017-07-12T15:32:52,995][INFO ][o.e.n.Node ] [master_hostname.domain] initialized
[2017-07-12T15:32:52,995][INFO ][o.e.n.Node ] [master_hostname.domain] starting ...
[2017-07-12T15:32:53,414][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] using profile[default], worker_count[4], port[9301], bind_host[null], publish_host[null], compress[false], connect_timeout[30s], connections_per_node[0/3/6/1/1], receive_predictor[64kb->64kb]
[2017-07-12T15:32:53,420][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] binding server bootstrap to: [XXX.XXX.XXX.XXX]
[2017-07-12T15:32:54,196][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] Bound profile [default] to address {XXX.XXX.XXX.XXX:9301}
[2017-07-12T15:32:54,197][INFO ][o.e.t.TransportService ] [master_hostname.domain] publish_address {XXX.XXX.XXX.XXX:9301}, bound_addresses {XXX.XXX.XXX.XXX:9301}
[2017-07-12T15:32:54,204][INFO ][o.e.b.BootstrapChecks ] [master_hostname.domain] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-07-12T15:32:57,992][INFO ][o.e.c.s.ClusterService ] [master_hostname.domain] new_master {master_hostname.domain}{jH94B8oWQTiKrNfu-3eQuA}{aJMQWztPRlmVg5LnNlAvkg}{hostname.domain}{XXX.XXX.XXX.XXX:9301}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-07-12T15:32:58,592][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [master_hostname.domain] publish_address {127.0.0.1:9201}, bound_addresses {XXX.XXX.XXX.XXX:9201}
[2017-07-12T15:32:58,646][INFO ][o.e.n.Node ] [master_hostname.domain] started
[2017-07-12T15:33:01,290][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] connected to node [{data_hostname.domain}{2y1AAE8SSB-ZiIeHJNRvTg}{EgseokcITtKt6fEt52FrtA}{hostname.domain}{XXX.XXX.XXX.XXX:9302}]
[2017-07-12T15:33:02,284][INFO ][o.e.g.GatewayService ] [master_hostname.domain] recovered [15] indices into cluster_state
[2017-07-12T15:33:03,195][INFO ][o.e.c.s.ClusterService ] [master_hostname.domain] added {{data_hostname.domain}{2y1AAE8SSB-ZiIeHJNRvTg}{EgseokcITtKt6fEt52FrtA}{hostname.domain}{XXX.XXX.XXX.XXX:9302},}, reason: zen-disco-node-join[{data_hostname.domain}{2y1AAE8SSB-ZiIeHJNRvTg}{EgseokcITtKt6fEt52FrtA}{hostname.domain}{XXX.XXX.XXX.XXX:9302}]
[2017-07-12T15:33:03,212][DEBUG][c.f.s.s.t.SearchGuardSSLNettyTransport] [master_hostname.domain] connected to node [{client_hostname.domain}{svr4h2-EQVWWKJdek19CQg}{PjFrAfHxSeOXMtlK07F1dQ}{hostname.domain}{XXX.XXX.XXX.XXX:9300}]
[2017-07-12T15:33:03,502][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [master_hostname.domain] SSL Problem null cert chain
Steve Haertel
Jul 12, 2017, 3:42:51 PM7/12/17
to Search Guard
(I should have specified in my previous post -- I used the generated trust store, and node keystore and renamed the files to the same name/location that I had before when I was trying my own key/trust stores.)
SG
Jul 13, 2017, 5:18:08 AM7/13/17
to search...@googlegroups.com
try skipping the ks/ts aliases:
searchguard.ssl.http.clientauth_mode: REQUIRE
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.transport.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.http.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_password: kpwd
searchguard.ssl.transport.truststore_password: tpwd
searchguard.ssl.http.keystore_password: kpwd
searchguard.ssl.http.truststore_password: tpwd
###searchguard.ssl.transport.keystore_alias: cn=hostname.domain #use defaults
###searchguard.ssl.transport.truststore_alias: root-ca-chain #use defaults
###searchguard.ssl.http.keystore_alias: cn=hostname.domain #use defaults
###searchguard.ssl.http.truststore_alias: root-ca-chain #use defaults
searchguard.ssl.http.clientauth_mode: REQUIRE
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.transport.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: stores/serverKeyStore.jks
searchguard.ssl.http.truststore_filepath: stores/serverTrustStore.jks
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.keystore_password: kpwd
searchguard.ssl.transport.truststore_password: tpwd
searchguard.ssl.http.keystore_password: kpwd
searchguard.ssl.http.truststore_password: tpwd
###searchguard.ssl.transport.truststore_alias: root-ca-chain #use defaults
###searchguard.ssl.http.keystore_alias: cn=hostname.domain #use defaults
###searchguard.ssl.http.truststore_alias: root-ca-chain #use defaults
searchguard.ssl.transport.enabled_protocols:
- "TLSv1.2"
searchguard.ssl.http.enabled_protocols:
- "TLSv1.2"
- "TLSv1.2"
searchguard.ssl.http.enabled_protocols:
- "TLSv1.2"
> Am 12.07.2017 um 21:42 schrieb Steve Haertel <steveh...@gmail.com>:
>
> (I should have specified in my previous post -- I used the generated trust store, and node keystore and renamed the files to the same name/location that I had before when I was trying my own key/trust stores.)
>
>
> (I should have specified in my previous post -- I used the generated trust store, and node keystore and renamed the files to the same name/location that I had before when I was trying my own key/trust stores.)
>
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d367905b-a5d5-4429-bfe0-1e0c007ab156%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
Steve Haertel
Jul 13, 2017, 9:54:26 AM7/13/17
to Search Guard
SOLVED.
I had an idea last night and tested it out today.
My problem wasn't the certificates from the elasticsearch.yml. Those were all OK. They were OK as soon as I got the es master to show "started"
The reason why these null cert errors were happening after that point is because I have a whole application cluster set up and some OTHER services try to query the master after it starts up. It was these other services that weren't modified to handle ssl things yet that seemed to be causing the errors to show up in the master log.
If anyone from elasticsearch/searchguard development is reading this (or heck, literally any software developer), it just goes to show you why log messages are super important. So many times through this whole process I was getting error messages like "alias not found" or "cert not found" but nowhere did it actually tell me where it was looking when the error happened. (keystore? truststore? who's making the call? )
SG
Jul 13, 2017, 2:27:20 PM7/13/17
to search...@googlegroups.com
thx for pointing this out.
We will add this to our feature backlog and try to improve the error messages in future versions
We will add this to our feature backlog and try to improve the error messages in future versions
> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3989f4e4-b3a3-4217-82dd-56ff67c25e91%40googlegroups.com.
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages