New to SSL and Search-Guard - help!

[Ross Coundon]

Hi 

I've installed search-guard for Elasticsearch and Kibana 6.2.3 using the sample configuration for TLS.  I now would like to migrate to using my own keys and certificates.

I'm new to the details of how to manage CSRs, keys and certs.  As such, the documentation has me rather baffled so would really appreciate some help.

I have a: 
Wildcard certificate: star_mydomain_com.crt

CA bundle: star_mydomain_com.ca-bundle (from comodo - they've apparently formatted this as PEM)

A private key: mydomain.key

And a root certificate: root-ca.pem

Of course I also have the original mydomain.csr

As well as applying these, I'm also confused about the admin_dn, and what it means to have separate node vs admin certificates, how they're specified and what the significance of this block is

searchguard.authcz.admin_dn:

- CN=kirk,OU=client,O=client,L=test, C=de

I've extracted the values from my csr using:
openssl req -noout -text -in mydomain.csr 

Is it simply a case of replacing the CN, C, L, O values with what I've extracted?  (I don't have a value for OU)

I appreciate it's a lot to ask but can somebody help with how I apply these to my cluster?  And, once that's done, do I need to specify this root-ca.pem within the logstash output to Elasticsearch (sitting on a separate server) or is something else required?

Any help, gratefully received - if you're close by, there's a beer in it for you!

Thanks

Ross

[Ross Coundon]

My original post may look like I haven't tried anything at all so here's what I've done.

To see more about what's going on I've added -Djavax.net.debug=all to sgadmin.sh

From this, I see about 2600 lines of output dealing with adding certs, presumably from the java certificate store, such as:

adding as trusted cert:

  Subject: CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK

  Issuer:  CN=Hongkong Post Root CA 1, O=Hongkong Post, C=HK

  Algorithm: RSA; Serial number: 0x3e8

  Valid from Thu May 15 05:13:14 UTC 2003 until Mon May 15 04:52:29 UTC 2023

Then I see something specific to what I'm trying:

***

found key for : key

chain [0] = [

[

  Version: V3

  Subject: CN=*.mydomain.com, OU=PositiveSSL Wildcard, OU=Domain Control Validated

  Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11

  Key:  Sun RSA public key, 2048 bits

  modulus: [I've removed this section]

  public exponent: 65537

  Validity: [From: Thu Apr 12 00:00:00 UTC 2018,

               To: Fri Apr 12 23:59:59 UTC 2019]

  Issuer: CN=COMODO RSA Domain Validation Secure Server CA, O=COMODO CA Limited, L=Salford, ST=Greater Manchester, C=GB

  SerialNumber: [    402c86b8 bb9b2d08 4e88561c e7cf4278]

Certificate Extensions: 10

[1]: ObjectId: 1.3.6.1.4.1.11129.2.4.2 Criticality=false

Extension unknown: DER encoded OCTET string =

I don't know if the highlighted text is a problem here.  The processing carries on though and repeats the same error message.

There's then a section that begins:

trigger seeding of SecureRandom

Amongst other things, I see that example.com is explicitly referenced which seems odd.

[Raw read]: length = 2712

0000: 02 00 00 51 03 03 5A D0   B2 BF D1 99 89 5B AD FA  ...Q..Z......[..

0010: B7 27 10 8E 66 F2 8D 04   C4 7C E3 D7 F4 04 C0 8B  .'..f...........

0020: 19 1F A2 C9 FA 74 20 5A   D0 B2 BF DF 8E 35 E7 DA  .....t Z.....5..

0030: 7E 06 30 5C BC 12 CE D4   05 3B AF FF 8F A5 D1 69  ..0\.....;.....i

0040: 7C 68 36 3F B5 48 0B C0   28 00 00 09 FF 01 00 01  .h6?.H..(.......

0050: 00 00 17 00 00 0B 00 08   34 00 08 31 00 04 20 30  ........4..1.. 0

0060: 82 04 1C 30 82 03 04 A0   03 02 01 02 02 01 01 30  ...0...........0

0070: 0D 06 09 2A 86 48 86 F7   0D 01 01 05 05 00 30 81  ...*.H........0.

0080: 95 31 13 30 11 06 0A 09   92 26 89 93 F2 2C 64 01  .1.0.....&...,d.

0090: 19 16 03 63 6F 6D 31 17   30 15 06 0A 09 92 26 89  ...com1.0.....&.

00A0: 93 F2 2C 64 01 19 16 07   65 78 61 6D 70 6C 65 31  ..,d....example1

00B0: 19 30 17 06 03 55 04 0A   0C 10 45 78 61 6D 70 6C  .0...U....Exampl

00C0: 65 20 43 6F 6D 20 49 6E   63 2E 31 24 30 22 06 03  e Com Inc.1$0"..

00D0: 55 04 0B 0C 1B 45 78 61   6D 70 6C 65 20 43 6F 6D  U....Example Com

00E0: 20 49 6E 63 2E 20 53 69   67 6E 69 6E 67 20 43 41   Inc. Signing CA

00F0: 31 24 30 22 06 03 55 04   03 0C 1B 45 78 61 6D 70  1$0"..U....Examp

0100: 6C 65 20 43 6F 6D 20 49   6E 63 2E 20 53 69 67 6E  le Com Inc. Sign

0110: 69 6E 67 20 43 41 30 1E   17 0D 31 36 30 35 30 34  ing CA0...160504

0120: 32 30 34 35 32 38 5A 17   0D 31 38 30 35 30 34 32  204528Z..1805042

0130: 30 34 35 32 38 5A 30 56   31 0B 30 09 06 03 55 04  04528Z0V1.0...U.

0140: 06 13 02 44 45 31 0D 30   0B 06 03 55 04 07 13 04  ...DE1.0...U....

0150: 54 65 73 74 31 0D 30 0B   06 03 55 04 0A 13 04 54  Test1.0...U....T

0160: 65 73 74 31 0C 30 0A 06   03 55 04 0B 13 03 53 53  est1.0...U....SS

0170: 4C 31 1B 30 19 06 03 55   04 03 13 12 6E 6F 64 65  L1.0...U....node

0180: 2D 30 2E 65 78 61 6D 70   6C 65 2E 63 6F 6D 30 82  -0.example.com0.

0190: 01 22 30 0D 06 09 2A 86   48 86 F7 0D 01 01 01 05  ."0...*.H.......

and

Key:  Sun RSA public key, 2048 bits

  modulus: 19579262089439182306813910512089210784193831404875226993250345175654787298297043918518918240912236540761250810413971000032753030384026416020067376495649430546864673120750373144604100523339925957091775635880977601302176033955846992036269861042526135657206863363789746440075873750482961699045597279616237270673297853254775005903575825666880844997212279579569199005223033233844719209309190994757415731305102297022933204960478013857951072912495917762377372497359604793744954899314149955717930975558549653990412023000074495076291653345376101271231271235495926296019374371818640692274376333587077509759755839367493162914301

  public exponent: 65537

  Validity: [From: Wed May 04 20:45:28 UTC 2016,

               To: Fri May 04 20:45:28 UTC 2018]

  Issuer: CN=Example Com Inc. Signing CA, OU=Example Com Inc. Signing CA, O=Example Com Inc., DC=example, DC=com

  SerialNumber: [    01]

Then finally, around line 3377, I see:

elasticsearch[_client_][transport_client_boss][T#1], fatal error: 46: General SSLEngine problem

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]

elasticsearch[_client_][transport_client_boss][T#1], SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown

elasticsearch[_client_][transport_client_boss][T#1], WRITE: TLSv1.2 Alert, length = 2

elasticsearch[_client_][transport_client_boss][T#1], fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem

[Raw write]: length = 7

0000: 15 03 03 00 02 02 2E                               .......

elasticsearch[_client_][transport_client_boss][T#1], called closeOutbound()

elasticsearch[_client_][transport_client_boss][T#1], closeOutboundInternal()

elasticsearch[_client_][transport_client_boss][T#1], called closeInbound()

elasticsearch[_client_][transport_client_boss][T#1], fatal: engine already closed.  Rethrowing javax.net.ssl.SSLException: Inbound closed before receiving peer's close_notify: possible truncation attack?

Unable to check whether cluster is sane: None of the configured nodes are available: [{#transport#-1}{xL0RkWiMTr-bETA1T7K2xw}{localhost}{127.0.0.1:9300}]

13:38:07.636 [elasticsearch[_client_][transport_client_boss][T#1]] ERROR com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport - SSL Problem General SSLEngine problem

javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1529) ~[?:1.8.0_162]

at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:535) ~[?:1.8.0_162]

at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:813) ~[?:1.8.0_162]

at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:1.8.0_162]

at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_162]

at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:281) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1215) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]

at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]

at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]

Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:1.8.0_162]

at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:330) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:322) ~[?:1.8.0_162]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1614) ~[?:1.8.0_162]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:1.8.0_162]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:1.8.0_162]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

... 19 more

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:1.8.0_162]

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:1.8.0_162]

at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_162]

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_162]

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_162]

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_162]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ~[?:1.8.0_162]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:1.8.0_162]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:1.8.0_162]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

... 19 more

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:1.8.0_162]

at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:1.8.0_162]

at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_162]

at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:1.8.0_162]

at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:1.8.0_162]

at sun.security.validator.Validator.validate(Validator.java:260) ~[?:1.8.0_162]

at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:1.8.0_162]

at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281) ~[?:1.8.0_162]

at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136) ~[?:1.8.0_162]

at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1601) ~[?:1.8.0_162]

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1052) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:992) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$1.run(Handshaker.java:989) ~[?:1.8.0_162]

at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_162]

at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1467) ~[?:1.8.0_162]

at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1364) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1272) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]

... 19 more

elasticsearch[_client_][transport_client_boss][T#1], called closeOutbound()

elasticsearch[_client_][transport_client_boss][T#1], closeOutboundInternal()

elasticsearch[_client_][transport_client_boss][T#1], called closeInbound()

elasticsearch[_client_][transport_client_boss][T#1], closeInboundInternal()

ERR: Cannot connect to Elasticsearch. Please refer to elasticsearch logfile for more information

I can however connect to my elasticsearch cluster using ReST without a problem, data is being persisted there.

I thought this may be a network bind problem so I've set:

network.bind_host: 0.0.0.0

network.host: 0.0.0.0

transport.host: 0.0.0.0

I haven't changed the port for internal comms so it should still be 9300.

I currently have the elasticsearch.yml set up with the demo configuration for search-guard, I have tried entering own certificates/key in the place of the demo ones but elasticsearch won't start.  This whole trace was as a result of running sgadmin.sh which, I assume, sets the values necessary in elasticsearch.yml rather than me needing to specify them manually, so my values are currently commented out in elasticsearch.yml.

[Search Guard]

Would it be sufficient for your to only apply your own certificates to the http part (port 9200) of elasticsearch?

Or do you have also people from "outside" which connects to your elasticsearch via the transport protocol (port 9300).

If http is enough i recommend to use your own certs for http part and the generated certificates for the inter cluster communication/transport protocol.

If you need your own certs everywhere you have to create another certificate which you can use as a client certificate (the searchguard.authcz.admin_dn thing). Maybe also this can help you https://docs.search-guard.com/latest/offline-tls-tool#tls-tool

[Ross Coundon]

I think it would be, I just have logstash communicating from a remote server on the HTTP part to 9200.

So, if I use my own certs for the http part, I'll take a look at the offline tool, thanks.  I'm still not really clear on how I secure the 9200 communication though with my proper CA certs for my domain.  Should it be sufficient to set the arguments to sgadmin.sh to specify the certs or do I need to set the values in some configuration files first?

Thanks!

[Search Guard]

sgadmin is not related to http/port 9200

In general you have two ways to connect to elasticsearch: via http/s (port 9200) and via the transport protocol (port 9300).

Most clients like logstash, filebeat etc use the http/s way but sgadmin use the transport way.

So i recommend you configure http with you own certificates (you do not need an admin cert or something here).

An for all transport level communication on port 9300 (including sgadmin) you use the provided certs from the online or offline certificate generator/tls tool

The reason why sgadmin needs an admin certificate (which is in fact a ssl client certificate) is because of security reasons.

Normally ssl is used only to proof the servers authority to the client (for example when you do homebanking) but sgadmin use ssl in two way

mode where the client also authenticates to the server and therefore the client needs also a certificate.

Hope this helps

[lmjo...@gmail.com]

Don't know if this will help but two things that were holding me up back when I was trying to get things going was that the certificate needed to have both authServer and authClient (decided by your CA conf when they sign the certificate) as it will present itself as a client to other nodes as well as as a server to accept communication from other nodes; and that it needs to include the device certificate in the same PEM with my certificate on top and the device certificate below that one.