sgadmin client certificate authentication not working when using provider signed server certificate

[Andreas Freudenreich]

* Search Guard and Elasticsearch version: searchguard 5.1.1 r11

* Installed and used enterprise modules, if any: DLSFLS, LDAP

* JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4

* Search Guard configuration files: don't think they are relevant here

* Elasticsearch log messages on debug level: see below

* Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,

Following scenarios - the first works, the second doesn't anymore.

Scenario 1: data nodes in the cluster make regular rest calls to a coordinator node to retrieve or change cluster state. The data nodes have a client certificate for sgadmin set up (self-signed) . The coordinator node has a self-signed http/rest server certificate in a java keystore and a truststore with the root CA.

Running following works fine:

[root@ela-hdatahis01 certs]# curl -v -k --cert /etc/pki/tls/certs/sgadmin-admin.crtfull.pem --key /etc/pki/tls/p rivate/sgadmin-admin.key.pem --request GET "https://kib-webhis01.ourdomain:9200/_cluster/health?pretty"

* About to connect() to kib-webhis01.ourdomain port 9200 (#0)

*   Trying 10.93.37.176...

* Connected to kib-webhis01.[root@ela-hdatahis01 certs]# curl -v -k --cert /etc/pki/tls/certs/sgadmin-admin.crtfull.pem --key /etc/pki/tls/p rivate/sgadmin-admin.key.pem --request GET "https://kib-webhis01.ourdomain:9200/_cluster/health?pretty"

* About to connect() to kib-webhis01.ourdomain port 9200 (#0)

*   Trying 10.93.37.176...

* Connected to kib-webhis01.ourdomain (10.93.37.176) port 9200 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* skipping SSL peer certificate verification

* NSS: client certificate from file

*       subject: CN=sgadmin,OU=IT,O=OurComp,L=OurLoc,C=OurCountry

*       start date: Feb 28 00:42:26 2017 GMT

*       expire date: Feb 27 00:42:26 2047 GMT

*       common name: sgadmin

*       issuer: CN=OurComp_ElasticStack Signing CA,OU=OurComp_ElasticStack Signing CA,O=OurComp_ElasticStack,DC=it,DC=ubc,DC=ca

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

*       subject: CN=kib-webhis01.ourdomain,OU=IT,O=OurComp,L=OurLoc,C=OurCountry

*       start date: Nov 09 05:28:10 2017 GMT

*       expire date: Nov 08 05:28:10 2047 GMT

*       common name: kib-webhis01.ourdomain

*       issuer: CN=OurComp_ElasticStack Signing CA,OU=OurComp_ElasticStack Signing CA,O=OurComp_ElasticStack,DC=it,DC=ubc,DC=ca

> GET /_cluster/health?pretty HTTP/1.1 (10.93.37.176) port 9200 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* skipping SSL peer certificate verification

* NSS: client certificate from file

*       subject: CN=sgadmin,OU=IT,O=OurComp,L=OurLoc,C=OurCountry

*       start date: Feb 28 00:42:26 2017 GMT

*       expire date: Feb 27 00:42:26 2047 GMT

*       common name: sgadmin

*       issuer: CN=OurComp_ElasticStack Signing CA,OU=OurComp_ElasticStack Signing CA,O=OurComp_ElasticStack,DC=it,DC=ubc,DC=ca

* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

* Server certificate:

*       subject: CN=kib-webhis01.ourdomain,OU=IT,O=OurComp,L=OurLoc,C=OurCountry

*       start date: Nov 09 05:28:10 2017 GMT

*       expire date: Nov 08 05:28:10 2047 GMT

*       common name: kib-webhis01.ourdomain

*       issuer: CN=OurComp_ElasticStack Signing CA,OU=OurComp_ElasticStack Signing CA,O=OurComp_ElasticStack,DC=it,DC=ubc,DC=ca

> GET /_cluster/health?pretty HTTP/1.1

...

Scenario 2: is exactly the same as scenario 1, but the server side certificate has been replaced by a gandi wildcard certificate. The truststore remains the same.

[root@ela-hdatahis01 certs]# curl -v -k --cert /etc/pki/tls/certs/sgadmin-admin.crtfull.pem --key /etc/pki/tls/private/sgadmin-admin.key.pem --request GET "https://kib-webhis01.ourdomain:9200/_cluster/health?pretty"

* About to connect() to kib-webhis01.ourdomain port 9200 (#0)

*   Trying 10.93.37.176...

* Connected to kib-webhis01.ourdomain (10.93.37.176) port 9200 (#0)

* Initializing NSS with certpath: sql:/etc/pki/nssdb

* skipping SSL peer certificate verification

* NSS: client certificate from file

*       subject: CN=sgadmin,OU=IT,O=OurComp,L=OurLoc,C=OurCountry

*       start date: Feb 28 00:42:26 2017 GMT

*       expire date: Feb 27 00:42:26 2047 GMT

*       common name: sgadmin

*       issuer: CN=OurComp_ElasticStack Signing CA,OU=OurComp_ElasticStack Signing CA,O=OurComp_ElasticStack,DC=it,DC=ubc,DC=ca

* NSS error -12188 (SSL_ERROR_INTERNAL_ERROR_ALERT)

* Peer reports it experienced an internal error.

* Closing connection 0

curl: (35) Peer reports it experienced an internal error.

It seems the client certificate is failing? Which I wouldn't expect as the truststore on the server had not been touched.

On the server side the error is following:

[2017-11-09T11:18:42,888][WARN ][c.f.s.h.SearchGuardHttpServerTransport] [kib-webhis01] caught exception while handling client http traffic, closing connection [id: 0x5aea6dbe, L:0.0.0.0/0.0.0.0:9200 ! R:/10.93.37.179:45278]

io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: General OpenSslEngine problem

...

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The relevant TLS settings in elasticsearch.yml on the server side (kib-webhis01) - the only change between scenario 1 and 2 is the content of http.keystore_filepath:

# SearchGuard SSL http/REST settings

#searchguard.ssl.http.enabled: false

searchguard.ssl.http.enabled: true

searchguard.ssl.http.keystore_filepath: kib-webhis01.ourdomain-http-keystore.jks

searchguard.ssl.http.keystore_password: 123

searchguard.ssl.http.truststore_filepath: truststore.jks

searchguard.ssl.http.truststore_password: changeit

searchguard.ssl.http.clientauth_mode: OPTIONAL

# SearchGuard common settings:

#searchguard.config_index_name: searchguard

#searchguard.cert.oid: '1.2.3.4.5.5'

searchguard.authcz.admin_dn:

  - CN=sgadmin,OU=IT,O=OurComp,L=OurLoc,C=OurCountry

# SearchGuard SSL transport settings

#searchguard.ssl.transport.enabled: true

searchguard.ssl.transport.keystore_filepath: kib-webhis01.ourdomain-transport-keystore.jks

searchguard.ssl.transport.keystore_password: 123

searchguard.ssl.transport.truststore_filepath: truststore.jks

searchguard.ssl.transport.truststore_password: changeit

searchguard.ssl.transport.enforce_hostname_verification: false

logger.com.floragunn.searchguard.ssl: DEBUG

Keytool outputs:

TRUSTSTORE: root-ca-chain is the own CA

[root@kib-webhis01 elasticsearch]# keytool -list -keystore truststore.jks

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 3 entries

ead-cert, Aug 29, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49

root-ca-chain, Jan 17, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 94:21:9A:32:36:2B:55:70:7D:85:36:24:2D:75:06:A5:D8:FC:2A:C6

usertrustrsaca, Nov 9, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 2B:8F:1B:57:33:0D:BB:A2:D0:7A:6C:51:F7:0E:E9:0D:DA:B9:AD:8E

KEYSTORE self-signed: contains privkey and certificate + signing/root CA certificate

[root@kib-webhis01 elasticsearch]# keytool -list -keystore kib-webhis01.ourdomain-http-keystore.jks

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

kib-webhis01.ourdomain, Nov 8, 2017, PrivateKeyEntry,

Certificate fingerprint (SHA1): 61:9C:84:17:7D:57:48:F9:05:E9:C3:4C:51:2B:11:90:03:DC:A2:AB

KEYSTORE gandi-signed: contains privkey and wildcard certificate + gandi intermediate certificates

[root@kib-webhis01 elasticsearch]# keytool -list -keystore kib-webhis01.ourdomain-http-keystore.jks.gandi

Enter keystore password:

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

star.ourdomain, Nov 8, 2017, PrivateKeyEntry,

Certificate fingerprint (SHA1): 77:9E:B8:FD:0D:CB:CF:D0:74:16:65:A7:8E:28:7A:61:6A:C1:AE:12

[SG]

Can you please try connecting with wget or openssl instead of curl? We saw a lot of strange thing with curl and NSS.

openssl s_client -verify_return_error -debug -connect kib-webhis01.ourdomain:9200 -CAfile ... -key ... -cert ...
wget --no-check-certificate --certificate=... --private-key=... https://kib-webhis01.ourdomain:9200

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/926d7c70-0821-4688-a65b-58e72a50a9af%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> <ES_log.txt>

[Andreas Freudenreich]

For both scenarios I get the same errors with wget:

[root@ela-hdatahis01 certs]# wget --no-check-certificate --certificate=/etc/pki/tls/certs/sgadmin-admin.crtfull. pem --private-key=/etc/pki/tls/private/sgadmin-admin.key.pem https://kib-webhis01.ourdomain:9200/_cluste r/health?pretty

--2017-11-09 13:28:19--  https://kib-webhis01.ourdomain:9200/_cluster/health?pretty

Resolving kib-webhis01.ourdomain (kib-webhis01.ourdomain)... 10.93.37.176

Connecting to kib-webhis01.ourdomain (kib-webhis01.ourdomain)|10.93.37.176|:9200... connected.

OpenSSL: error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

Unable to establish SSL connection.

I have attached the openssl outputs - also seem to have the same write error and result in the error message on the server (unable to find valid certification path).

Thanks for any hints,

Andreas

[Andreas Freudenreich]

Thanks for your help - with these commands and using a s_server for testing I was able to solve the issue.

The truststore on the server side needed to include all intermediate certificates relevant for the client certificate authentication. Having only the root CAs in the truststore and putting the intermediate certs (or the whole chain) into the client certificate did not work (except for one curl command - strange enough).

Does this sound correct? If yes, I think the documentation should be updated accordingly as it only talks about root CAs in the truststore: https://github.com/floragunncom/search-guard-docs/blob/master/tls_configuration.md

Thanks again,

Andreas

[Jochen Kressin]

I find it a bit odd that the intermediate certs need to be included in the truststore in this case. Usually you only have to place the root CA there. So thanks for reporting, we will investigate and update the docs eventually.

[Andreas Freudenreich]

Maybe it is because of the issue described here (under "Design Problems"): https://blogs.msdn.microsoft.com/kaushal/2015/05/27/client-certificate-authentication-part-1/