javax.net.ssl.SSLException: Received fatal alert: certificate_unknown

[Fabien Wernli]

Using ES 5.6.6 and SG 18.

I tried adding a new node to the cluster, and am having the following issue on startup:

[2018-02-14T13:38:59,887][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [node0210] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
  at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
  at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
  at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
  at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
  at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
  at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
  at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
  at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_161]
  at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:255) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1162) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084) ~[netty-handler-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.13.Final.jar:4.1.13.Final]
  at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.13.Final.jar:4.1.13.Final]
  at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

I checked all the steps at http://docs.search-guard.com/latest/troubleshooting-tls and didn't find anything different when comparing the node's certificate to a working one's.

Any idea on what I could try next?

[Jochen Kressin]

Hm, strange, but I guess there has to be a difference somewhere ;) In order to check whether the general SG config is correct on this new node, have you tried to run it with a working certificate from another node? (You probably need to disable hostname verification if you use another certificate).

* Are you using the same truststore / trusted CA on all nodes? 

* Are you sure the certificate chain is complete? Maybe the working certificates include an intermediate cert and the non-working does not?

* Are you sure the certificate is not expired?

[Fabien Wernli]

I just tried disabling hostname verification on the new node, and tried using the new and another working certificate to no avail.
The same truststore is being used on all nodes (when comparing the output of `keytool -list -v -keystore truststore.jks` only the "Alias name" and "Creation date" differ on all nodes).
The certificate is valid until 2020.

Here's the stack trace:

[Fabien Wernli]

On the cluster's master node I just saw the following message when trying to start the new node:

Caused by: java.security.cert.CertificateException: No subject alternative names matching IP address 1.2.3.4 found

Upon re-inspection of the keystore, I saw that only the IPv6 address was stored. I missed it the first time as we have a in-house policy where the string value of the IPv4 address is contained in the IPv6 address:

IPAddress: 2001:db8:0:1:1.2.3.4

Sorry for the noise!

[Jochen Kressin]

No problem and thanks for the update, glad you found it :)