keytool says keystore valid, but search guard says it's invalid

[hokie...@gmail.com]

Hi Everyone,

I am getting the following error with the ES 2.3.5/searchguard 2.3.5.11:

Exception in thread "main" ElasticsearchSecurityException[Error while initializing transport SSL layer: java.io.IOException: Invalid keystore format]; nested: IOException[Invalid keystore format];
Likely root cause: java.io.IOException: Invalid keystore format

However, when I inspect the keystore via keytool, I don't get the invalid keystore format error. Specifically, keytool -v -list -keystore es.keystore.jks returns the expected info, no error

Consequently, it appears the keystore is indeed valid but somewhere in the Search Guard-Java stack an error is occurring.

Any suggestions would definitely be appreciated.

Thanks

--John

[Search Guard]

ES 2.3.5 is end of life, so we will not support it any longer. You should upgrade to ES 5.6 because ES 2.4 will become also EOL in Feb 2018.

See https://github.com/floragunncom/search-guard-docs/blob/master/eol.md

Can you check if you have the same issue with ES 2.4.6 and SG 14 or ES 5.6.4 and SG 16

Please also post you elasticsearch.yml and complete stacktraces/logfiles. If its only a test keystore (containing no real sensitive data) you can also mail them so we can try to reproduce the error.

[hokie...@gmail.com]

Cool, gotcha, okay. I upgraded to ES 5.6.4 and am using search guard-5-5.6.4-16/search guard-ssl-5.6.4-23 and I am seeing the same error.

In terms of configuration, I am using the default elasticsearch.yml and passing the following command-line configs:

searchguard.ssl.transport.enabled=true
searchguard.ssl.transport.keystore_filepath=*******
searchguard.ssl.transport.keystore_password=*******
searchguard.ssl.transport.truststore_filepath=*******
searchguard.ssl.transport.truststore_password=*******
searchguard.ssl.transport.alias=* (wildcard cert)
searchguard.ssl.transport.enforce_hostname_verification=false
searchguard.ssl.transport.resolve_hostname=false
searchguard.ssl.http.enabled=true
searchguard.ssl.http.keystore_filepath=*******
searchguard.ssl.http.keystore_password=*******
searchguard.ssl.http.truststore_filepath=*******
searchguard.ssl.http.truststore_password=*******

Again, since keytool does not return a keystore format error, I find this stack trace puzzling. Gonna look at the source code now to see if there is some config param I am missing. Please let me know if you see anything in my config params you think is incorrect and/or if I am missing any required params.

Thanks

--John

[SG]

what happens if you remove "searchguard.ssl.transport.alias" from the config?
An alias has nothing to do with wildcards, so i assume this is the problem here

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/d368bd0f-bed8-49b0-bab1-be5f16693078%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[hokie...@gmail.com]

Ah, thats a great thought. Unfortunately, that still did not fix the issue, same error message.

[hokie...@gmail.com]

Also, just curious, do I need both the searchguard and searchguard-ssl plugins? I am trying to get the encryption to work first, and then we are interested in exploring the document and field-level security.

--John

[SG]

With SG 5.x you need just the SG Plugin (which includes SSL already), with 2.x you have to install both plugins.

Please send complete logfiles and stacktraces

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/a86085d5-ff2a-4f78-b8c0-12a396efbd7c%40googlegroups.com.

[hokie...@gmail.com]

Okay, I just created test keystore and truststore files and now I can start up ES.

I turned on DEBUG and, when I compare the stack traces, I *think* I've found the problem.

With the successful launch using my test certificates, I have this:

SSLCertificateHelper] Alias localhost: single cert CN=elasticsearch, OU=hokiegeek2-es, O=hokiegeek2, C=[**] of type -1 -> true

With the unsuccessful launch, it appears to hit a wildcard cert in a chain of three certs:
SSLCertificateHelper] Alias _: single cert CN=*.[**************], OU=[*****], OU=[*****], OU=[******], O=[*****], C=[**] of type -1 -> false
ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]

Do I need to specify another one of the certs in another config for this to work?

--John

[hokie...@gmail.com]

Okay, figured it out. The keystore/truststore combo was not working. Switched to pem/crt/cacerts and now ES is starting up fine. Next problem is sgadmin failing.

--John

[Jochen Kressin]

For PEM it's important to check whether your trust chain is complete when you use intermediate / signing CAs. It's not important where you place the intermediate certs, but the chain must be complete. For example, if the certificate on ES configured in  searchguard.ssl.transport.pemtrustedcas_filepath contains only the root CA, the certificate you use with sgadmin and the -cert switch has to contain the admin certificate and also the intermediates, otherwise the chain is not complete.

Likewise, if searchguard.ssl.transport.pemtrustedcas_filepath contains the root CA and the intermediates, the -cert certificate in sgadmin only needs to contain the actual admin certificate.