Cannot retrieve cluster state due to: No user found for cluster:monitor/health.

[Fabio Rodrigues]

Been trying to upgrade to ES 6.1.1 and SG6

Was working fine on SG5 and ES5, upgraded to 6, cluster is fine and shows as green (at one point at least), before i deleted the searchguard index to try and start fresh my old users would login in kibana but no sgadmin...

* Search Guard and Elasticsearch version

ES 6.1.1

SG6 6.1.1-20.1

* Installed and used enterprise modules, if any

none

* JVM version and operating system version

Ubuntu 16.04

openjdk version "1.8.0_151"

OpenJDK Runtime Environment (build 1.8.0_151-8u151-b12-0ubuntu0.16.04.2-b12)

OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

* Search Guard configuration files

tried with blank (the initial files) and my old ones, error is the same

* Other installed Elasticsearch or Kibana plugins, if any

none

sudo ./sgadmin.sh -cd ../sgconfig/original/ -ks sgadmin.jks -kspass password -ksalias client -ts truststore.jks -tspass password -icl -nhnv -h 10.10.x.x

WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6

Will connect to 10.10.x.x:9300 ... done

Connected as ---------------------------------------------------------

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: No user found for cluster:monitor/health. This is not an error, will keep on trying ...

  Root cause: ElasticsearchSecurityException[No user found for cluster:monitor/health] (org.elasticsearch.ElasticsearchSecurityException/org.elasticsearch.ElasticsearchSecurityException)

   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)

   * Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in elasticsearch.yml

   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

[SG]

can you pls post the output of

sudo ./sgadmin.sh -cd ../sgconfig/original/ -ks sgadmin.jks -kspass password -ksalias client -ts truststore.jks -tspass password -icl -nhnv -h 10.10.x.x -w

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/8a764afc-7b60-44c1-840c-462568cec8bc%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Fabio Rodrigues]

These are the same certificates I created for SG5.

Search Guard Admin v6

Will connect to 10.10.x.x:9300 ... done

Connected as EMAILADDRESS=c...@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK

{

  "whoami" : {

    "dn" : "EMAILADDRESS=c...@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK",

    "is_admin" : true,

    "is_authenticated" : true,

    "is_node_certificate_request" : true

  }

}

[SG]

SG6 is more strict about certificates than SG5.

Semms that you admin certificate is also a node certificate. Thats no longer permitted.
Did you encode the OID into the admin cert or do you use the nodes_dn property in elasticsearch.yml?

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/74026a2a-0b52-4e4e-8058-4bd789df6018%40googlegroups.com.

[Fabio Rodrigues]

I do remember encoding the OID into the admin cert and in elasticsearch i only have the 

searchguard.authcz.admin_dn: 

  - EMAILADDRESS=c...@company.com,CN=sgadmin,OU=company,O=company Ltd,L=London,ST=England,C=UK

[SG]

hen you need to recreate this certificate without an OID

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/ea90e9f7-355c-4f9b-9efa-dfcb0a15163a%40googlegroups.com.

[Fabio Rodrigues]

I see, ill give that a try, its nice that one can use the PEMs straight up tho! 

Ill report back when i try. 

[Fabio Rodrigues]

was just looking at my cmds file, and the ID i added was this

subjectAltName = RID:1.2.3.4.5.5

as per openssl.conf

[ es_server_cert ]

# Extensions for server certificates (`man x509v3_config`).

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

subjectAltName = RID:1.2.3.4.5.5

authorityKeyIdentifier = keyid,issuer:always

keyUsage = critical, digitalSignature, keyEncipherment

extendedKeyUsage = serverAuth,clientAuth

so indeed it seems i used the server (node) config for the sgadmin, tho i did have this note on my file:

# sgadmin # after hours of trying, it turns out sgadmin also NEEDS "TLS Server Auth" -.- go figure

which was me trying to use, but never worked....

[ es_usr_cert ]

# Extensions for client certificates (`man x509v3_config`).

basicConstraints = CA:FALSE

subjectKeyIdentifier = hash

authorityKeyIdentifier = keyid,issuer

keyUsage = critical, digitalSignature, keyEncipherment

extendedKeyUsage = clientAuth

what say you? use the client config or the server one without the RID, its been a while, cant remember what the RID was for :(

[SG]

see https://github.com/floragunncom/search-guard-ssl/tree/master/example-pki-scripts and http://docs.search-guard.com/latest/generating-tls-certificates
You can also generate certificates here: https://floragunn.com/tls-certificate-generator/

The sgadmin certificate must not have the RID and extendedKeyUsage = serverAuth,clientAuth, the dn of this certificate must be registered in elasticsearch.yml
The node certificates need to have the RID and extendedKeyUsage = serverAuth,clientAuth
Normal client certificate must not have the RID and extendedKeyUsage = clientAuth

Hope this helps

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/089631a4-816f-4c24-b05e-7a5a589c1d4a%40googlegroups.com.

[Jochen Kressin]

I will add this to the documentation backlog, seems we are not clear enough here.

> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard+unsubscribe@googlegroups.com.

[Fabio Rodrigues]

i did have a look for changes that might break this, but didnt find any on this :O

also recreated as above but now i get this....

i have tried encrypted PEM and encrypted PEM, with password and no password, the keys are RSA and have verified with openssl.

running sgadmin as per http://docs.search-guard.com/latest/sgadmin-examples

sudo ./sgadmin.sh -cd ../sgconfig/original/ -cacert sgadmin-full.cert.pem -cert sgadmin.cert.pem -key sgadmin.NOkey.pem -icl -nhnv -h 10.10.x.x


WARNING: JAVA_HOME not set, will use /usr/bin/java

Search Guard Admin v6
Will connect to 10.10.x.x:9300 ... done

13:23:52.062 [main] ERROR com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore - Your keystore or PEM does not contain a key. If you sepcified a key password try removing it. If you not sepcified a key password maybe you one because the key is password protected. Maybe you just confused keys and certificates.
ERR: An unexpected IllegalStateException occured: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
Trace:
java.lang.IllegalStateException: failed to load plugin class [com.floragunn.searchguard.SearchGuardPlugin]
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452)
        at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:105)
        at org.elasticsearch.client.transport.TransportClient.newPluginService(TransportClient.java:103)
        at org.elasticsearch.client.transport.TransportClient.buildTemplate(TransportClient.java:128)
        at org.elasticsearch.client.transport.TransportClient.<init>(TransportClient.java:251)
        at com.floragunn.searchguard.tools.SearchGuardAdmin$TransportClientImpl.<init>(SearchGuardAdmin.java:823)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:403)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:120)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443)
        ... 7 more
Caused by: ElasticsearchSecurityException[Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: sgadmin.NOkey.pem]; nested: IllegalArgumentException[File does not contain valid private key: sgadmin.NOkey.pem]; nested: InvalidKeySpecException[Neither RSA, DSA nor EC worked]; nested: InvalidKeySpecException[java.security.InvalidKeyException: IOException : algid parse error, not a sequence]; nested: InvalidKeyException[IOException : algid parse error, not a sequence];
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:292)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.<init>(DefaultSearchGuardKeyStore.java:145)
        at com.floragunn.searchguard.ssl.SearchGuardSSLPlugin.<init>(SearchGuardSSLPlugin.java:192)
        at com.floragunn.searchguard.SearchGuardPlugin.<init>(SearchGuardPlugin.java:182)
        ... 12 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: sgadmin.NOkey.pem
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:267)
        at io.netty.handler.ssl.SslContextBuilder.forServer(SslContextBuilder.java:90)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.buildSSLServerContext(DefaultSearchGuardKeyStore.java:613)
        at com.floragunn.searchguard.ssl.DefaultSearchGuardKeyStore.initSSLConfig(DefaultSearchGuardKeyStore.java:287)
        ... 15 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1024)
        at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:993)
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:265)
        ... 18 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169)
        at java.security.KeyFactory.generatePrivate(KeyFactory.java:372)
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1022)
        ... 20 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:352)
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:357)
        at sun.security.ec.ECPrivateKeyImpl.<init>(ECPrivateKeyImpl.java:73)
        at sun.security.ec.ECKeyFactory.implGeneratePrivate(ECKeyFactory.java:237)
        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:165)
        ... 22 more

[Fabio Rodrigues]

looks like it worked with a .jks of the same new cert :)

thank you for all the help guys :D/