File does not contain valid private key: /etc/elasticsearch/node1.key

[Li Cui]

Hello there.

Yesterday I setup the Serch-guard on ELK cluster after going through so many issues with the keys and certificates.

It was due to that the only pkcs8 keys were supported.. I'm on the last Search Guard version, Version: 6.x-23.

Anyway, it worked yesterday I was able to send logs from the client via filebeat and the data showing on Kibana.

I shut down the cluster and shut down the AWS instances (Elassticsearch and kibana on one node, logstash on another node).

Today, I started the AWS instances and started the cluster, checked that all ELK were up and running, but when I used the same credential(the default admin)

But I wasn't able to log on saying "The Search Guard license information could not be loaded. Please contact your system administrator."

Here is my sg_config.yml:

searchguard:

  dynamic:

    authc:

      clientcert_auth_domain:

        http_enabled: true

        order: 1

        http_authenticator:

          type: clientcert

          config:

            username_attribute: cn #optional, if omitted DN becomes username

          challenge: false

        authentication_backend:

          type: noop

========================

Elasticsearch started fine and I can access on :https://<elasticnode IP>:9200 and got:

{

  "name" : "node1",

  "cluster_name" : "hls-test-elk",

  "cluster_uuid" : "A_9Ls_hiSzKM34WZ7Rjxnw",

  "version" : {

    "number" : "6.3.2",

    "build_flavor" : "default",

    "build_type" : "rpm",

    "build_hash" : "053779d",

    "build_date" : "2018-07-20T05:20:23.451332Z",

    "build_snapshot" : false,

    "lucene_version" : "7.3.1",

    "minimum_wire_compatibility_version" : "5.6.0",

    "minimum_index_compatibility_version" : "5.0.0"

  },

  "tagline" : "You Know, for Search"

}

============================

./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elastic.IP> -cn hls-test-elk

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elastic.IP>:9300 ... done

01:26:01.556 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

Trace:

java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]

        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:701)

        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:114)

        at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:107)

        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:132)

        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:269)

        at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:886)

        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:441)

        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:123)

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)

        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)

        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)

        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:692)

        ... 7 more

Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: IllegalArgumentException[File does not contain valid private key: /etc/elasticsearch/node1.key]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : DER input, Integer tag error]; nested: InvalidKeyException[IOException : DER input, Integer tag error];

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)

        at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:193)

        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:197)

        ... 12 more

Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /etc/elasticsearch/node1.key

        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)

        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)

        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)

        ... 15 more

Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked

        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1045)

        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:1014)

        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)

        ... 18 more

Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : DER input, Integer tag error

        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)

        at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)

        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1043)

        ... 20 more

Caused by: java.security.InvalidKeyException: IOException : DER input, Integer tag error

        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)

        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)

        at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)

        at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)

        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)

        ... 22 more

==============================

I've changed the node1.key using "openssl pkcs8 topk8  -in node1.key -out node1-pkcs8.key" and didn't work. the sg-internal-users.yml has never been changed.

Please help and let us know what's next....

Thank you very much in advance

LI

[Jochen Kressin]

Maybe you ran into this Java issue here:

https://github.com/floragunncom/search-guard/issues/524

tl;dr: OpenSSL changed the default PKCS#5, v1.5 format to PKCS#5, v2 in OpenSSL 1.1.0. The JDK has a long outstanding issue with PKCS#5, v2. In short, PKCS#5, v2 does not work.

Can you try to force OpenSSL to generate PKCS#5, v1.5 by using:

openssl pkcs8 -v1 "PBE-SHA1-3DES" -topk8 ...

[Li Cui]

Thank you.... 

I did: openssl pkcs8 -topk8 -v1 PBE-SHA1-3DES -in node1.key -out node1.key

Still got the same result:

[ro...@elk.com search-guard-6]# ./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elastcIP> -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elastcIP>:9300 ... done

03:47:55.215 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

I can start the elasticsearch no problem, but can not initialize  the cluster...

This is very urgent,.. need to demo this...

Please help

Li

--
You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/60d4864c-7d03-4697-acb5-3cb12b7a08e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jochen Kressin]

So this seems strange to me. You write that it worked at some point, but then after restart you got the exception complaining about the key? Something is odd here, because if the key stayed the same, you should not see this exception. I guess I want to say that the key does not suddenly become invalid. So something else must be going on here.

First thing to do would be to check the contents of the key:

openssl rsa -in config/node1.key -check

Second would be the question how you exactly generated the key? If you have a PKCS#12 for example and want to convert it to PKCS#5, v1.5 you would use something like:

openssl pkcs8 -v1 "PBE-SHA1-3DES" -in "node1.key.pkcs12" -topk8 -out "node1.key" -passin "pass:..." -passout "pass:..."

If your key does not have any password of course omit the passin and passout parameters.

Third thing to check is whether your key has a password set. Depending on how you generated the key, you might have header in your node1.key file like:

 -----BEGIN ENCRYPTED PRIVATE KEY-----

or

-----BEGIN PRIVATE KEY-----

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

[Jochen Kressin]

@Li Cui, any news on that one?

[Li Cui]

Kressin,

Oh, Let me explain the one used to work a bit...

I setup one a different AWS environment, it worked last weekend, I was able to log on to Kibana with default admin/<password>, then I  copied all configuration files shut it down and. Next day, I started ELK and all the sudden, using the same credential I wasn't able to log on to Kibana and more. I tried to reset the password and when tried to initializing  elasticsearch, I got the issue.  Then I moved to our testing AWS environment, setup ELK with SG with the same steps and same configuration except the IPs... etc... I still get the same issue.

I could not figure what happened on that original one when all the sudden I wasn't able to log on using the same credential which used to be working.

I'm thinking there might be something I didn't do it right in the configuration in the configuration files? Please take a look and let me know if you see anything

Thanks in advance

Li

On Tue, Aug 28, 2018 at 3:27 PM Li Cui <lcu...@gmail.com> wrote:

Kressin,

Thank you very much for your help. By reading your suggestions, I did the following and got the same result:

1. [ec2-...@mycompany.com tools]$ ./sgtlstool.sh -c /home/ec2-user/config/tlsconfig.yml -ca -crt

WARNING: JAVA_HOME not set, will use /usr/bin/java

Root certificate has been sucessfully created.

The passwords of the private key files have been auto generated. You can find the passwords in root-ca.readme.

Created 2 node certificates.

Passwords for the private keys of the node certificates have been auto-generated. The passwords are stored in the config snippet files.

Created 1 client certificates.

Passwords for the private keys of the client certificates have been auto-generated. The passwords are stored in the file "client-certificates.readme"

2.[ec2-...@mycompany.com out]$ openssl rsa -in node1.key -check

Enter pass phrase for node1.key:

RSA key ok

writing RSA key

-----BEGIN RSA PRIVATE KEY-----

MIIEogIBAAKCAQEAmnD5vaOnsJdxfYGutvrkQ74vht/jvQWIxfm10fiZd0FCaQN+

LzyfizUW2ayLjs/F2H/GbZSvjtfoIatwdD/AEJgqvLzVJj4YAYeVAcqTk9iYmMQM

rOFaJQ3jbSc7XkB4ewGguuJO+oP3Sduh5PcRdCiTEZTMWxnCVOknv5k1vZq7e8Sq

ad+KKTzLRB1LfIKlu80lEbOQmjhIGoUKUhO6H60NLjAuE5vGgxRm0Xuhe4SG4DKh

H73cyylriOfyIyb92sg3CC1X71xqgjvVIPhuDOlJPEeBwTXyqnhhoftZotfQaN4D

SYMrJA397+l3wwWPxZ+OJ8YLSyid26dhmdzS/wIDAQABAoH/WY7fvP41ZOcc+jz8

Hz+QRk/SBhr/YCmBWBeeMvZeDhqyWa9q8+vYHKQ+xxzVTqFTq5JUvzhUd2yQlLYm

8lMxf7Ut+YdHjgMxUpgAbUPRbsEN+O4C4dOe2l606Zfve0wlpTIBezgUMgocnOeC

wtpw5fBw+mDQoqEIdMGmAyjLamviqX1W2y1/CeVTN5RvAkR25MhnYNlzkR0//4cV

M2zV3AXytfS8fHU59kBlDZmc5DFC3FBZTO79l2NVI25E9SBjgNAsa7QJSsYAHL3j

8S+EInuP+xt55kyQzDCe/1DxRAc3tcxMnxiercUyFRMEIeaz2cp+PfQW+dYRlS0u

Y1uJAoGBANarS6c4fhq2gtKjISZlxNjRgz4gp+wFGeBkHzzv8vSJsClJwnA1Z6V5

3lFJc0e5XtwWhwkCoxu3jCXmOt0YmCNyUWz/F/N++cZZZ+ECk+kZ/ahZddqiUI5W

bgrsgp2xnMDm/tKro0peEyDR1ONHX/DoUEwZGtO4qzkIjVzMTxYXAoGBALgtJUWi

2Att1hATly7QMnTZKJa2EbJS1AdRBI60aJh5OiTyWAMSz8Ozm65qjBuQsrzs8Xiv

/f4X+gYXVkimuWiImuWkTE1XII/D4Gce6Naw2jO/OIadTn7mmhVLoLmXhyysGA9r

X0KnhaTTrXk2vBqLgD7ksLdeuHgD2z+hDyNZAoGBALGs15IRRPaRJxh2IoLoapPW

XR035CSKWE+MLvq900YpDCAwR3meCipBAwwekgbEW8on7x47C4tuUNYxh3jl2U5F

WTWwsT89UjikWUE4Y/nkJDmJyezquWgGC1CJTtZ8YjSbPEBcBWwRDx3z7pMqyTLY

5VUG1lh0mWs8M9yh0beDAoGAP/4XrDlnnZj9Prq/1dkrHuRxMOR0A6S53Qzua3MN

wKSmaj32DKNVoybd2d1VbyJ4qCIfyJJFyyqjichEB5w+Ya7NOys6A8lZG/qwFsRr

nGT4e/BeAI/ZvUf8k5eicbnPktr8xSbGg49XZSzTYoF9HDAKUE//gscbiz+zx0j7

zIkCgYEAkhMaSadi/7m0nb20l9VuTOOs2RbLE4xgWOWU/wLEB6zPPLgJqS7paWr7

bDJuSA5bYKyf/hbN2wsZwc79RyJJ3a8b7F6H8J9ZtvkZUbG3s/hrGNDSu9mOJxWJ

vbePZ9s95qBmpJL4gFC1os+lPazr6Y1DqU/vhSVdEzIKd97VCIg=

-----END RSA PRIVATE KEY-----

3.[ec2-...@mycompany.com tools]$ ./sgtlsdiag.sh -ca /home/ec2-user/tools/out/root-ca.pem -crt /home/ec2-user/tools/out/node1.pem

WARNING: JAVA_HOME not set, will use /usr/bin/java

========================================================================

/home/ec2-user/tools/out/node1.pem

------------------------------------------------------------------------

Certificate 1

------------------------------------------------------------------------

            SHA1 FPR: AA4CEF301E3A418066984FF54D043E3DB9D6EE3E

             MD5 FPR: EEAE1AC4B2FF4D22FA73CC492A9AF10D

Subject DN [RFC2253]: CN=node1.hlx.dxc.com,OU=Ops,O=Example Com\, Inc.,DC=hls.dxc,DC=com

       Serial Number: 1535485513377

 Issuer DN [RFC2253]: CN=root.ca.hls.dxc.com,OU=CA,O=Example Com\, Inc.,DC=hls.dxc,DC=com

          Not Before: Tue Aug 28 19:45:14 UTC 2018

           Not After: Fri Aug 25 19:45:14 UTC 2028

           Key Usage: digitalSignature nonRepudiation keyEncipherment

 Signature Algorithm: SHA256WITHRSA

             Version: 3

  Extended Key Usage: id_kp_serverAuth id_kp_clientAuth

  Basic Constraints: -1

                SAN:

                  dNSName: node1.hls.dxc.com

                  iPAddress: <elasticIP>

------------------------------------------------------------------------

Trust anchor:

DC=com,DC=hls.dxc,O=Example Com\, Inc.,OU=CA,CN=root.ca.hls.dxc.com

4. Replaced the new keys and certificates in elasticsearch.yml with the new ones and also  node1_elasticsearch_config_snippet.yml 

5. Restarted elasticsearch node. 

6. Went to https:// <elasticIP>:9200

   Got:Search Guard not initialized (SG11). See http://docs.search-guard.com/v6/sgadmin

7.[ro...@mycompany.com tools]# /sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elasticIP> -cn elk-tsg

bash: /sgadmin.sh: No such file or directory

[ro...@mycompany.com tools]# ./sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elasticIP> -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elasticIP>:9300 ... done

20:00:36.045 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

==================================

I attached the tlsconfig.yml I used to generate the key/certs and also the elasticsearch.yml.

Please help and let me know how we should do to fix this.

Thanks a lot

Li

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/60d4864c-7d03-4697-acb5-3cb12b7a08e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/a5f04bf6-8717-4298-9bc8-46b69b1ea59f%40googlegroups.com.

[Li Cui]

Kressin,

Thank you for your help... I fixed the problem. I regenerated the keys as you indicated above. Replaced the keys/certs and the node..snippet.yml in elasticsearch.yml.

and ran sgadmin.sh to initialize the SG, it worked:

 [root@mycompany search-guard-6]# ./tools/sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/beats.pem -key /etc/elasticsearch/beats.key -keypass TUDJ7lxhdtMt -nhnv -h 10.100.35.182 -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elasticIP>:9300 ... done

Elasticsearch Version: 6.3.2

Search Guard Version: 6.3.2-23.0

Connected as CN=beats.xxxx.com,OU=Ops,O=xxxx Com\, Inc.,DC=xxxx,DC=com

Contacting elasticsearch cluster 'elk-tsg' and wait for YELLOW clusterstate ...

Clustername: elk-tsg

Clusterstate: YELLOW

Number of nodes: 1

Number of data nodes: 1

searchguard index already exists, so we do not need to create one.

Populate config from /usr/share/elasticsearch/plugins/search-guard-6/sgconfig

Will update 'sg/config' with sgconfig/sg_config.yml

   SUCC: Configuration for 'config' created or updated

Will update 'sg/roles' with sgconfig/sg_roles.yml

   SUCC: Configuration for 'roles' created or updated

Will update 'sg/rolesmapping' with sgconfig/sg_roles_mapping.yml

   SUCC: Configuration for 'rolesmapping' created or updated

Will update 'sg/internalusers' with sgconfig/sg_internal_users.yml

   SUCC: Configuration for 'internalusers' created or updated

Will update 'sg/actiongroups' with sgconfig/sg_action_groups.yml

   SUCC: Configuration for 'actiongroups' created or updated

Done with success

Thank you very much, 

Please mark this case closed.

[Li Cui]

[ro...@mycompany.com tools]# ./sgadmin.sh -cd sgconfig/ -cacert /etc/elasticsearch/root-ca.pem -cert /etc/elasticsearch/node1.pem -key /etc/elasticsearch/node1.key -nhnv -h <elasticIP> -cn elk-tsg

WARNING: JAVA_HOME not set, will use /bin/java

Search Guard Admin v6

Will connect to <elasticIP>:9300 ... done

20:00:36.045 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.

==================================

I attached the tlsconfig.yml I used to generate the key/certs and also the elasticsearch.yml.

Please help and let me know how we should do to fix this.

Thanks a lot

Li

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/60d4864c-7d03-4697-acb5-3cb12b7a08e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/a5f04bf6-8717-4298-9bc8-46b69b1ea59f%40googlegroups.com.

[Li Cui]

Hi, this one has been fixed... thank you very  much, please check my last update... you may mark this as closed.

Thank you again

Li

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/60d4864c-7d03-4697-acb5-3cb12b7a08e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.

To post to this group, send email to search...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/759655d2-4f7a-4b86-8b13-a75426610bdd%40googlegroups.com.