Cannot initialize SearchGuard

[vale...@servergeek.at]

Hi guys,

I have a 7 node ES cluster and I installed SG on it.

For some reason I cannot initialize the SG plugin. Any idea why? 

I'm using generated (online by SG) certificates and those seems to be OK. 

When I try to initialize SG I do it using the following command:

sgadmin.sh -cacert root-ca.pem -cert sgadmin.crt.pem -key sgadmin.key.pem -keypass blabla -nhnv -cd ../sgconfig/ -h 1.2.3.4 -p 9300 -cn MyCluster

My SG configuration block present in ES is the following:

####### SEARCH GUARD #######

searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.transport.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.transport.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.crtfull.pem
searchguard.ssl.http.pemkey_filepath: certificates/search-guard-certificates/node-certificates/CN=avl2923t.it.internal.key.pem
searchguard.ssl.http.pemkey_password: e4aae4f746361c10e3aa
searchguard.ssl.http.pemtrustedcas_filepath: certificates/search-guard-certificates/chain-ca.pem
searchguard.nodes_dn:
   - '*'
searchguard.authcz.admin_dn:
   - CN=sgadmin

######## End Search Guard Configuration ########

* Search Guard and Elasticsearch version

SG -> 5-5.5.2-16, ES -> 5.5.2

* Installed and used enterprise modules, if any

-> none

* JVM version and operating system version 

-> openjdk version "1.8.0_144"

* Search Guard configuration files

-> more exactly?

* Elasticsearch log messages on debug level

-> 

ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
        at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
        at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
        at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
        at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
        at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
        at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
        at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1544)
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
        at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1501)
        at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1385)
        at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1267)
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078)
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at java.lang.Thread.run(Thread.java:748)

* Other installed Elasticsearch or Kibana plugins, if any

-> none

[Jochen Kressin]

This usually happens why you try to use a node certificate as an admin certificate. We improved the error messages in SG6 when this happens, in SG5 it's not so obvious ;)

So can you please check if the certificate you use in the sgadmin call:

sgadmin.crt.pem

Is also a node certificate, means has the OID set?

[Valentin Fischer]

Hi Jochen,

The certificate has the following properties:

Certificate:

    Data:

        Version: 3 (0x2)

        Serial Number: 1 (0x1)

    Signature Algorithm: sha256WithRSAEncryption

        Issuer: O=SaS, OU=SaS Signing CA, CN=SaS Signing CA

        Validity

            Not Before: Jan 23 07:47:48 2018 GMT

            Not After : Jan 23 07:47:48 2020 GMT

        Subject: CN=sgadmin

        Subject Public Key Info:

            Public Key Algorithm: rsaEncryption

                Public-Key: (2048 bit)

...

        X509v3 extensions:

            X509v3 Key Usage: critical

                Digital Signature, Key Encipherment

            X509v3 Basic Constraints:

                CA:FALSE

            X509v3 Extended Key Usage:

                TLS Web Server Authentication, TLS Web Client Authentication

            X509v3 Subject Key Identifier:

                ...

            X509v3 Authority Key Identifier:

                keyid:...

...

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.
To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jochen Kressin]

Can you please also post the SAN section of the key? The OID field in question is in that section. Thx!

[Valentin Fischer]

Hmmm... no idea how to get that. How do I get/see it ?

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/90c70109-9415-44d9-8bb3-c95234166c07%40googlegroups.com.

[Jochen Kressin]

It's in the X509v3 extensions part, I think you just cutted it off. Should be something like:

       X509v3 extensions:

           X509v3 Key Usage: critical

               Digital Signature, Key Encipherment

           X509v3 Basic Constraints:

               CA:FALSE

           X509v3 Extended Key Usage:

               TLS Web Server Authentication, TLS Web Client Authentication

           X509v3 Subject Key Identifier:

               7D:A1:DE:12:4D:AE:D6:79:9D:CF:A8:57:7E:30:08:8B:BA:8E:59:D8

           X509v3 Authority Key Identifier:

               keyid:35:03:23:13:30:30:21:1F:8F:BD:F3:DF:5E:C1:B0:A9:20:88:2C:B0

            X509v3 Subject Alternative Name:

               DNS:node-0.example.com, DNS:localhost, IP Address:127.0.0.1, Registered ID:1.2.3.4.5.5

If you see the "Registered ID" part here, it means this is a node certificate.

[vale...@servergeek.at]

Hi,

There is no SAN part in it.

X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Key Identifier:

                72:E6:02:00:BF:2A:F0:E4:BB:18:EB:E7:5E:DC:ED:9F:A8:FD:BC:67
            X509v3 Authority Key Identifier:
                keyid:E9:4A:18:64:74:6D:C6:EF:46:FA:C1:BB:53:62:98:B2:C8:6C:75:4A

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/3a8548ea-14d4-42f4-b0be-d6a2e2389c62%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/search-guard/ruyB5QVFAds/unsubscribe.

To unsubscribe from this group and all its topics, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

[vale...@servergeek.at]

Any updates on this ? 

I want to have a running cluster and for some reason I cannot initialize it.

[Jochen Kressin]

Ah, sorry, it's obvious but somehow I totally overlooked it:

It's this part of the config that causes problems:

searchguard.nodes_dn:
   - '*'

If you just use a wildcard here it means that Search Guard will treat every certificate, including the admin one, as node certificate. And that's why sgadmin chokes with the error message you posted. You need to make sure that the pattern you are using here does not match the DN of the admin certificate. Depending on the DNs of your node certificates that might be something like:

CN=*.it.internal

[vale...@servergeek.at]

Hi Jochen,

Thank you for the info! I have applied the change and also switched to "official" certificates and I have one BIG issue...

[2018-01-30T10:05:13,481][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
javax.net.ssl.SSLHandshakeException: error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:955) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:914) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:978) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1021) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:205) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]

Any ideas why is it throwing this error? The certificate and key are OK

(openssl x509 -noout -modulus -in node.pem | openssl md5 ; openssl rsa -noout -modulus -in node.key | openssl md5) | uniq
(stdin)= a5ab322f1cd213600e8ac367a471e06f

This is starting to drive me crazy...

Another question would be if the Registered ID:1.2.3.4.5.5 is really needed in the SAN.

Thank you!!!!!!!

[vale...@servergeek.at]

My current ES settings are the following:

####### SEARCH GUARD #######


searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.pemcert_filepath: certificates/node.pem
searchguard.ssl.transport.pemkey_filepath: certificates/node.key
searchguard.ssl.transport.pemkey_password: blabla
searchguard.ssl.transport.pemtrustedcas_filepath: certificates/chain.pem
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.ssl.http.pemcert_filepath: certificates/node.pem
searchguard.ssl.http.pemkey_filepath: certificates/node.key
searchguard.ssl.http.pemkey_password: blabla
searchguard.ssl.http.pemtrustedcas_filepath: certificates/chain.pem
searchguard.nodes_dn:
   - CN=*.it.internal
searchguard.authcz.admin_dn:
   - CN=admin

[Jochen Kressin]

I know, TLS can be tricky from time to time ...

First regarding the OID. Background of all of this is that Search Guard needs to reliably identify traffic between the nodes in your cluster (inter-node traffic). Inter-node traffic has elevated privileges, so we need to make sure only trusted nodes talk to each other. For this we use TLS. This also shields from the attack vector where an attacker would start a node, let if join your cluster, and then sniffs traffic.

When a node joins the cluster or sends a request to another node, we check if sends a node certificate. We identify a certificate as node certificate by:

* Checking if it has the OID in the SAN part OR

* Checking the DN against the configures list of DNs in searchguard.nodes_dn

The first approach is more flexible, since it does not require you to make changes in elasticsearch.yml should something with the DNs change. However, not all PKIs are able to add an OID, that's why we offer the second approach, listing the DNs of the node certificates in elasticsearch.yml. So if you use this second approach, you don't need to add the OID.

Regarding the exception: According to the stack trace this exception happens on the REST layer (port 9200). It complains that it cannot find a client certificate in the request. What did you do in order for this error to show up? Maybe you pointed sgadmin to the REST port (9200) instead of the transport port (9300)?

[Valentin Fischer]

Hi,

Thanks for the reply! 

The certificate has the OID:
            X509v3 Key Usage: critical
                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Subject Key Identifier:
                B5:D9:1F:4C:01:2E:E1:84:FC:84:B6:F7:72:9F:1B:F0:19:09:D2:BB
            X509v3 Subject Alternative Name:
                DNS:avl2923t.it.internal, Registered ID:1.2.3.4.5.5
            X509v3 Authority Key Identifier:
                keyid:4E:6F:0C:C1:18:62:1C:2B:A4:E2:7B:C9:A3:D0:5F:1E:57:4A:F8:41

The exception is happening when I start ES:

[2018-01-30T14:21:50,500][ERROR][c.f.s.s.t.SearchGuardSSLNettyTransport] [ES_MASTER1] SSL Problem error:140890C7:SSL routines:ssl3_get_client_certificate:peer did not return a certificate

The exception is raised every 2 seconds or so.

Any ideas ?

To unsubscribe from this group and all its topics, send an email to search-guard+unsubscribe@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/6961a039-a75b-4f7a-b5c8-51cbed8c6ffa%40googlegroups.com.

[Valentin Fischer]

You're suggestion about something else talking with the node opened my eyes! 

I did a tcpdump and seen that an older node was trying to talk with this node....

Stopped it and this node is starting "clean". 

Now I'll add another one to see if they can talk to each other.

Thanks for the replies!