sgadmin failure with PEM certificates

[calv...@gmail.com]

Hi there,

I'm trying to setup searchguard but sgadmin fails with the following output

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
WARNING: JAVA_HOME not set, will use /usr/bin/java
 
Search Guard Admin v5
 
Will connect to localhost:9300 ... done
 
 
### LICENSE NOTICE Search Guard ###
 
If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)
 
* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging
 
In case of any doubt mail to <sales@floragunn.com>
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-22_17-10-58.txt
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
 
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
 
 Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
 * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
 * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
 * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
 * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
 Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6OG0aBSHT6mATg4zhwvehQ}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)
 * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
 * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
 * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
 * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.
Clustername: elasticsearch
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
ERR: An unexpected ElasticsearchSecurityException occured: Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md
Trace:
ElasticsearchSecurityException[Search Guard not initialized (SG11) for indices:admin/exists. See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md]
 at com.floragunn.searchguard.filter.SearchGuardFilter.apply(SearchGuardFilter.java:128)
 at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:168)
 at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:142)
 at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:64)
 at org.elasticsearch.action.support.HandledTransportAction$TransportHandler.messageReceived(HandledTransportAction.java:54)
 at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceivedDecorate(SearchGuardSSLRequestHandler.java:178)
 at com.floragunn.searchguard.transport.SearchGuardRequestHandler.messageReceivedDecorate(SearchGuardRequestHandler.java:192)
 at com.floragunn.searchguard.ssl.transport.SearchGuardSSLRequestHandler.messageReceived(SearchGuardSSLRequestHandler.java:140)
 at com.floragunn.searchguard.SearchGuardPlugin$3$1.messageReceived(SearchGuardPlugin.java:376)
 at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69)
 at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1553)
 at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
 at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110)
 at org.elasticsearch.transport.TcpTransport.handleRequest(TcpTransport.java:1510)
 at org.elasticsearch.transport.TcpTransport.messageReceived(TcpTransport.java:1393)
 at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310)
 at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:297)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:413)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1273)
 at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1084)
 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
 at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
 at java.lang.Thread.run(Thread.java:748)

Meanwhile, in the elasticsearch logs, we have (traceback redacted for readability);

Nov 22 17:10:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
Nov 22 17:11:58 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:monitor/stats
Nov 22 17:12:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.ConfigurationLoader] Failure no such index retrieving configuration for [roles] (index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.c.IndexBaseConfigurationRepository] Unable to load configuration because of java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: java.util.concurrent.TimeoutException: Timeout after 1MINUTES while retrieving configuration for [roles](index=searchguard)
Nov 22 17:13:04 elasticsearch elasticsearch[1180]: [ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

Here we show that the admin certificate is indeed valid against the specified CA

vagrant@elasticsearch:~$ openssl x509 -noout -text -in ~/admin001.crt | grep Subject:
 Subject: CN=admin001
vagrant@elasticsearch:~$ sudo openssl verify -CAfile /etc/elasticsearch/rest.ca ~/admin001.crt
/home/vagrant/admin001.crt: OK

Here is the elasticsearch.yml configuration

vagrant@elasticsearch:~$ sudo grep -e '^$' -e '^#' --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: "elasticsearch"
node.name: "elasticsearchminion"
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:
 - '*'
searchguard.nodes_dn:
 - '*'
searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
searchguard.audit.enable_request_details: true
searchguard.audit.ignore_users:
 - kibanaserver

What am I missing here ? For what it's worth, all cryptographic material have been generated using openssl/easyrsa (I'm planning on documenting it).
Please let me know if I can provide any extra informations.

Arthur

[SG]

searchguard.authcz.admin_dn can not be a wildcard

> Am 23.11.2017 um 10:39 schrieb calv...@gmail.com:
>
> Hi there,
>
> I'm trying to setup searchguard but sgadmin fails with the following output
>
> vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl
> WARNING: JAVA_HOME not set, will use /usr/bin/java
>
> Search Guard Admin v5
>
> Will connect to localhost:9300 ... done
>
>
> ### LICENSE NOTICE Search Guard ###
>
> If you use one or more of the following features in production
> make sure you have a valid Search Guard license
> (See https://floragunn.com/searchguard-validate-license)
>
> * Kibana Multitenancy
> * LDAP authentication/authorization
> * Active Directory authentication/authorization
> * REST Management API
> * JSON Web Token (JWT) authentication/authorization
> * Kerberos authentication/authorization
> * Document- and Fieldlevel Security (DLS/FLS)
> * Auditlogging
>

> In case of any doubt mail to <sa...@floragunn.com>

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/166328f7-872a-4aec-804a-27187b477b97%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[calv...@gmail.com]

Thanks for the quick answer !

Not much success tho;

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo grep -e '^$' -e '^#' --invert-match /etc/elasticsearch/elasticsearch.yml
cluster.name: "elasticsearch"

node.name: "elasticsearchminion"
node.master: true
node.data: true
network.bind_host: 0.0.0.0
network.publish_host: 0.0.0.0
network.host: 0.0.0.0
http.port: 9201
discovery.zen.ping.unicast.hosts: [
]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: 15gb
cluster.routing.allocation.disk.watermark.high: 5gb
searchguard.ssl.transport.pemkey_filepath: transport.key
searchguard.ssl.transport.pemcert_filepath: transport.cert
searchguard.ssl.transport.pemtrustedcas_filepath: transport.ca
searchguard.ssl.transport.enabled: true
searchguard.ssl.transport.keystore_type: x509
searchguard.ssl.http.pemkey_filepath: rest.key
searchguard.ssl.http.pemcert_filepath: rest.cert
searchguard.ssl.http.pemtrustedcas_filepath: rest.ca
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.http.enabled: true
searchguard.authcz.admin_dn:

  - CN=admin001
searchguard.nodes_dn:

  - '*'
searchguard.audit.type: com.payplug.auditlog.impl.StdoutAuditLog
searchguard.audit.enable_request_details: true
searchguard.audit.ignore_users:
  -

kibanaserver
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo service elasticsearch restart
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh  -cert ~/admin001.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin001.key --disable-host-name-verification --diagnose -icl

WARNING: JAVA_HOME not set, will use /

usr/bin/java                                                                                                                                                                                                                                

Search Guard Admin v5                                                                                                                                                                                                                                                            
Will connect to localhost:9300 ... done                                                                                                                                                                                                                                          

### LICENSE NOTICE Search Guard ###

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging

In case of any doubt mail to <sales@floragunn.com>
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-23_10-52-14.txt

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{dhWeO-uaT4KaitGsjFknKA}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

The log on the Elasticsearch side are the same, final entry is still

[ERROR][c.f.s.f.SearchGuardFilter] Search Guard not initialized (SG11) for indices:admin/exists

[Abhishek Amte]

 I am also using pem certs and facing the same issue. I have put in an exact DN match. 

Should I be changing any settings in sg_config dir?

Thanks

[SG]

Can you post (or mail) the output of

openssl x509 -in ~/admin001.crt -text -noout

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1e926131-b185-495d-a894-27eeb6c66603%40googlegroups.com.

[calv...@gmail.com]

Here you are. Maybe it's missing an ExtendedKeyUsage attribute ?

vagrant@elasticsearch:…$ openssl x509 -in ~/admin001.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6 (0x6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=mycompany test ca
        Validity
            Not Before: Nov 22 15:15:31 2017 GMT
            Not After : Nov 20 15:15:31 2027 GMT
        Subject: CN=admin001
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f2:7c:8a:fd:5b:d2:1e:1e:01:52:32:9f:ae:57:
                    fd:c1:8c:94:52:dd:e7:3a:2f:8c:3f:71:44:ab:81:
                    79:37:64:08:d5:76:a8:36:be:29:60:27:13:fd:23:
                    92:db:bb:f9:de:cc:3e:88:c5:7d:69:e3:48:ca:0b:
                    3d:8e:d1:81:73:7a:14:05:95:a0:95:8b:70:ef:d5:
                    65:81:01:57:39:45:fa:c2:28:81:52:f2:4f:de:fd:
                    38:1a:f1:11:e6:9c:36:6a:51:3a:b8:5a:b1:51:c1:
                    04:3d:fe:b1:55:24:32:a6:3f:f3:83:7b:e4:77:1c:
                    45:03:49:9f:ac:e2:dc:5f:f5:8a:34:ac:3b:c2:73:
                    a3:70:5a:63:e5:32:4a:b4:99:4a:53:1c:9d:10:dd:
                    6c:ba:72:88:86:29:c7:da:7c:5a:60:ed:d8:74:cd:
                    0f:47:d8:b3:6f:be:75:25:fa:5d:23:43:fd:2c:c3:
                    b7:74:57:17:e1:04:76:6f:b9:82:08:c5:af:2b:ce:
                    f5:14:d2:4c:02:f6:47:f3:0b:2a:c9:80:4a:fd:23:
                    be:be:00:3c:4d:af:ff:b5:65:24:fb:49:d5:20:24:
                    d4:4a:26:cc:c2:71:30:94:31:68:78:7b:8b:df:d0:
                    e8:f8:eb:34:d6:ba:1c:e6:95:9a:54:f3:0c:29:2b:
                    6f:2f
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Subject Key Identifier:
                0B:8F:E0:5C:5C:02:36:C7:37:8B:17:90:0E:D8:04:D9:C8:25:29:11
            X509v3 Authority Key Identifier:
                keyid:74:32:94:50:67:DF:4C:95:03:18:D0:51:08:A6:50:14:E0:8A:42:C8
                DirName:/CN=mycompany test ca
                serial:A7:CD:62:39:B3:FF:48:76

            X509v3 Extended Key Usage:
                TLS Web Client Authentication
            X509v3 Key Usage:
                Digital Signature
    Signature Algorithm: sha256WithRSAEncryption
         a1:97:f8:e7:19:9a:18:40:af:a1:91:7d:35:14:34:2a:1a:14:
         a4:02:ed:65:27:26:00:be:02:37:dc:4e:b2:27:16:4d:06:a7:
         de:c2:5f:3f:26:36:e6:9e:19:0b:67:4f:71:3a:38:84:7a:de:
         4a:00:44:ec:02:43:9b:8c:ae:81:6b:84:34:64:1d:1b:85:ff:
         6d:ab:0e:cd:a1:43:92:15:fb:7e:6b:0e:9b:cf:aa:b1:0a:c1:
         65:14:59:29:4f:94:93:b5:91:16:f1:22:5a:12:2a:ab:a4:59:
         33:f1:47:03:3f:03:b6:3a:ad:df:2a:90:ef:71:db:ef:5f:d7:
         e2:3a:4f:6d:1c:8f:76:e1:7c:5f:a0:bb:19:b1:83:c7:1f:b3:
         f0:40:f8:c6:66:38:74:be:07:e5:5d:8d:f9:25:ca:f0:d8:cd:
         fc:ad:35:1b:67:40:1b:91:54:57:53:16:e7:a3:e0:67:9c:4c:
         7f:ad:0c:11:27:9f:c6:f3:da:88:db:38:17:04:6b:29:ff:f4:
         a4:34:ea:55:27:8e:e2:49:b4:f1:75:63:78:60:3e:1b:cc:0a:
         f7:87:d1:6f:2e:66:a4:8b:a8:87:eb:b8:16:9b:1f:75:46:d8:
         d3:fd:9c:55:30:4a:11:9c:b7:a6:f6:85:62:f4:45:0c:4e:34:
         00:38:ef:16

[SG]

seems like "Key Encipherment" and "TLS Web Server Authentication" as X509v3 Key Usage is missing

should look like

X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment

X509v3 Basic Constraints:
CA:FALSE

X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/1012f7f6-3a55-4333-aa47-7d56cfbda0f4%40googlegroups.com.

[calv...@gmail.com]

Still no luck, we can see the certificate now have the KeyEncipherment and WebServerAuth attributes. Also I updated elasticsearch.yml to match the admin002 CN;

vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ openssl x509 -in ~/admin002.crt -text -noout

Certificate:                                                                                                                                                                    
    Data:                                                                                                                                                                        
        Version: 3 (0x2)                                                                                                                                                        

        Serial Number: 7 (0x7)                                                                                                                                                  
    Signature Algorithm: sha256WithRSAEncryption                                                                                                                                
        Issuer: CN=company test ca                                                                                                                                              
        Validity                                                                                                                                                                
            Not Before: Nov 24 14:13:35 2017 GMT                                                                                                                                
            Not After : Nov 22 14:13:35 2027 GMT                                                                                                                                
        Subject: CN=admin002                                                                                                                                                    

        Subject Public Key Info:                                                                                                                                                
            Public Key Algorithm: rsaEncryption                                                                                                                                  
                Public-Key: (2048 bit)                                                                                                                                          
                Modulus:                                                                                                                                                                                                                                                          

                    00:be:95:33:6f:a8:ac:b1:4f:17:68:26:92:ec:45:                                                                                                                                                                                                                
                    74:9c:eb:17:6e:b3:eb:aa:47:51:62:be:6a:6e:cd:                                                                                                                                                                                                                
                    63:cf:6b:38:5b:56:e2:45:09:f9:77:bd:00:00:1e:                                                                                                                                                                                                                
                    10:99:8b:9e:01:89:1b:20:80:ae:b8:a3:ca:33:6c:                                                                                                                                                                                                                
                    43:97:b9:1c:39:a6:4f:fb:4e:4e:8b:91:68:4a:0e:                                                                                                                                                                                                                
                    52:42:fe:d9:1c:9a:5b:ba:6a:8f:ad:23:af:a0:f5:                                                                                                                                                                                                                
                    ed:57:e2:3e:a2:97:ec:dc:9e:91:00:ef:04:b2:bd:                                                                                                                                                                                                                
                    ec:b5:28:89:7f:3c:7f:e1:4d:5a:b3:f3:d8:ec:8a:                                                                                                                                                                                                                
                    db:54:32:67:67:b1:57:45:30:48:9a:10:96:ed:31:                                                                                                                                                                                                                
                    37:9c:73:62:d8:b2:8e:26:99:dc:d2:53:29:62:ee:
                    3f:68:e3:ff:e0:8d:e6:d1:77:d6:99:64:2e:81:9d:
                    ba:a3:c1:66:82:57:b2:75:bc:83:22:4e:94:45:2d:
                    e9:c4:c4:c6:a8:38:7f:21:28:5c:c5:a2:77:40:70:
                    2b:47:ed:1f:3b:74:60:4d:52:08:92:46:7b:c6:4d:
                    44:2d:c9:f5:ee:a0:95:c0:bb:2c:ae:41:e1:6c:3e:
                    74:bd:49:34:a9:00:9d:d0:b9:7b:d4:05:01:cc:a6:
                    9c:1d:0f:95:80:4e:87:97:f3:7d:9e:7d:4a:fc:2b:
                    cc:f7

                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Subject Key Identifier:

                F9:6F:FE:01:F4:32:14:36:9A:83:1E:47:09:72:FD:59:95:6C:AA:64

            X509v3 Authority Key Identifier:
                keyid:74:32:94:50:67:DF:4C:95:03:18:D0:51:08:A6:50:14:E0:8A:42:C8

                DirName:/CN=company test ca
                serial:A7:CD:62:39:B3:FF:48:76
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
    Signature Algorithm: sha256WithRSAEncryption
         95:d5:dc:75:57:83:a6:e0:89:93:5d:b7:e7:6a:51:75:83:5a:
         be:e5:ce:16:48:47:45:1e:6e:c5:e0:86:ce:d5:58:3e:20:f9:
         8d:b4:b2:5e:d6:8b:a0:49:94:9e:77:c1:20:64:d7:da:a9:4c:
         f8:98:2e:44:ed:1f:b9:88:56:88:a9:eb:f3:13:34:04:cf:0c:
         2e:89:c5:be:25:15:e2:dd:bd:4c:66:d5:d6:df:9d:cc:5c:73:
         f4:63:0e:2f:dd:7d:24:da:0e:2b:5d:6d:9a:30:9b:e0:11:d9:
         34:17:d1:14:44:4e:9e:b0:7f:46:87:2b:c7:25:20:a1:3e:fb:
         f2:de:38:3d:42:cc:eb:35:48:30:b3:60:6a:ff:23:fd:f0:cb:
         59:a7:e1:f8:89:fd:a5:52:44:95:d2:ab:a5:fd:75:df:9e:4c:
         a8:a8:8b:c2:0a:12:1c:17:aa:f4:84:91:54:4d:37:92:eb:4b:
         11:9c:0a:a6:e1:56:ed:03:aa:16:4e:66:1c:ae:10:4b:9a:4d:
         a0:ab:a7:21:61:5a:c8:cc:b3:a9:6b:53:35:7e:70:d9:97:a5:
         3d:ac:b9:cd:66:aa:92:2d:8c:70:73:b5:fe:9a:5b:ba:33:4c:
         65:27:3f:34:ec:2e:80:ce:f2:25:f8:e2:d7:3b:09:6f:d7:95:
         47:48:77:fe
vagrant@elasticsearch:/usr/share/elasticsearch/plugins/search-guard-5/tools$ sudo ./sgadmin.sh  -cert ~/admin002.crt -cacert /etc/elasticsearch/rest.ca -key ~/admin002.key --disable-host-name-verification --diagnose -icl

WARNING: JAVA_HOME not set, will use /

usr/bin/java

Search Guard Admin v5
Will connect to localhost:9300 ... done

### LICENSE NOTICE Search Guard ###

If you use one or more of the following features in production
make sure you have a valid Search Guard license
(See https://floragunn.com/searchguard-validate-license)

* Kibana Multitenancy
* LDAP authentication/authorization
* Active Directory authentication/authorization
* REST Management API
* JSON Web Token (JWT) authentication/authorization
* Kerberos authentication/authorization
* Document- and Fieldlevel Security (DLS/FLS)
* Auditlogging

In case of any doubt mail to <sales@floragunn.com>
###################################
Diagnostic trace written to: /usr/share/elasticsearch/plugins/search-guard-5/tools/sgadmin_diag_trace_2017-Nov-24_14-28-20.txt

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

   * Try running sgadmin.sh with -icl (but no -cl) and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * Make also sure that your keystore or cert is a client certificate (not a node certificate) and configured properly in elasticsearch.yml
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow sgadmin to operate on a red cluster.

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]. This is not an error, will keep on trying ...
  Root cause: NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{skJ0nA_FQNyFDAGkW0PS_g}{localhost}{127.0.0.1:9300}]] (org.elasticsearch.client.transport.NoNodeAvailableException/org.elasticsearch.client.transport.NoNodeAvailableException)

Any way to increase the debug output ?

[calv...@gmail.com]

Hi,

what can I do to move on this issue ?

Arthur

[Search Guard]

can you pls mail or attach the certificates?