Searchguard cannot retrieve cluster state

[Juan Martínez Palazón]

Hi,

I followed the documentation in order to set up search guard in a testing single-node ES cluster.

I initially generated both truststore.jks and keystore.jks files, using gen_root_ca.sh  and gen_client_node_cert.sh respectively, this way:

>   ./gen_root_ca.sh truestore.jks myCApass mypass
>   ./gen_client_node_cert.sh node1 myCApass mypass

Then I copied truststore.jks and node1-keystore.jks to the ES config directory and edited elasticsearch.yml as shown below:

searchguard.ssl.transport.keystore_filepath: node1-keystore.jks
searchguard.ssl.transport.keystore_password: mypass
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: mypass
searchguard.ssl.transport.enforce_hostname_verification: true
searchguard.ssl.transport.resolve_hostname: true

After this, I could restart my node, checking the cluster state is YELLOW and Searchguard is properly recognized:

[2017-02-02T12:10:22,328][INFO ][o.e.h.HttpServer         ] [dev-node1] publish_address {<IP>:49200}, bound_addresses {<IP>:49200}
[2017-02-02T12:10:22,329][INFO ][o.e.n.Node               ] [dev-node1] started
[2017-02-02T12:10:22,485][INFO ][c.f.s.a.c.TransportConfigUpdateAction] [dev-node1] searchguard index does not exist yet, so no need to load config on node startup. Use sgadmin to initialize cluster
[2017-02-02T12:10:22,485][INFO ][o.e.g.GatewayService     ] [dev-node1] recovered [1] indices into cluster_state
[2017-02-02T12:10:22,671][INFO ][o.e.c.r.a.AllocationService] [dev-node1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana][0]] ...]).


But if I try to send a request I received this message:

curl -s -XGET  http://node1:49200/
Search Guard not initialized (SG11). See https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

So I tried to initialize Search Guard with the sgadmin.sh script using this parameters:

./sgadmin.sh -ts ~/NODE/ES_config/truststore.jks -tspass mypass -ks ~/NODE/ES_config/node1-keystore.jks -kspass mypass -cd ~/NODE/src/elastic/plugins/search-guard-5/sgconfig/ -cn devcluster -h node1 -p 49300

And this is the warning displayed:

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{fn7Ips5PQ4-WBsM17H3DNg}{node1}{<IP>:49300}]. This is not an error, will keep on trying ...
   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

If I use the diagnose flag I can see this exception in the file returned, which doesn't shed light on this:

ClusterHealthRequest:
NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{6cGCk1cERqyEk9BFBJ0KJw}{node1}{<IP>:49300}]]
        at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:328)
        at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:226)
        at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59)
        at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:345)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:403)
        at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392)
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.execute(AbstractClient.java:704)
        at org.elasticsearch.client.support.AbstractClient$ClusterAdmin.health(AbstractClient.java:726)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.generateDiagnoseTrace(SearchGuardAdmin.java:676)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main0(SearchGuardAdmin.java:369)
        at com.floragunn.searchguard.tools.SearchGuardAdmin.main(SearchGuardAdmin.java:105)


Could you help me with this?

Many thanks!

Juan.

[Jochen Kressin]

Did you try to disable hostname verification with the -nhnv flag?

[Juan Martínez Palazón]

Hi Jochen,

Thanks for your quick reply.

That was the first change I did but then sgadmin.sh try to connect to localhost:

Search Guard Admin v5
Will connect to localhost:49300
ERR: Seems there is no elasticsearch running on localhost:49300 - Will exit

Regards,

[Eliran Boraks]

Was this issue resolved? 

I have the same problem. I did run sgadmin.sh with ' -icl -nhnv' but I got the same results. 

/sgadmin.sh -ts truststore.jks -tspass changeit -ks kirk-keystore.jks -kspass changeit -cd ../sgconfig -icl -nhnv -h <IP-Address>

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{10.100.0.21}{10.100.0.21:9300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

What is the right way to configure the node (referring to "Cannot retrieve cluster state due to: None of the configured nodes are available:")?

[Juan Martínez Palazón]

Hi Eliran,

No, unfortunately I'm still having this problem and I'm not able to make Searchguard work in my cluster.

Hope somebody can shed some light on this.

Regards,

[SG]

Can you both (Eliran and Juan) give us the following infos?:

- Elasticsearch and Search Guard version
- Complete elasticsearch.yml
- Complete output of elasticsearch.log from the the point in time you started elasticsearch until you get the sgadmin failures

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/e046e212-26a6-421c-92ce-da919e479c47%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Eliran Boraks]

Here is my info: 

search-guard-2-2.4.3.10.jar

ES: 2.4.3

I attached the elasticsearch.yml and elasticsarch.log to this message. 

Eliran

[Search Guard]

try (in elasticsearch.yml):

searchguard.authcz.admin_dn:

  - 'CN=kirk,OU=client,O=client,L=Test,C=DE'

To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/a3eb26ad-8559-4a10-9c28-57437ec485b6%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

<elasticsearch.log><elasticsearch.yml>

[Eliran Boraks]

This line is in the elasticsearch.yml file I attached. I double check it by copy/paste your line and I am still getting this. 

$ ./sgadmin.sh -ts truststore.jks -tspass changeit -ks kirk-keystore.jks -kspass changeit -cd ../sgconfig -icl -nhnv -h 10.100.0.21

Will connect to 10.100.0.21:9300 ... done

Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{10.100.0.21}{10.100.0.21:9300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)

   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

[Juan Martínez Palazón]

Here you have attached both .yml config and log files.

My ES version is 5.1.1 and search-guard-ssl-5.1.1-19.jar.

I also tried adding the authcz config posted in your last message but It doesn't work.

[SG]

Ok, can you both sent the elasticsearch.yml again after adding my config entry from my last post.
@Juan: The logs are too short, we need also see the log statements from the point in time when sgadmin fail

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/5ae0e931-5c34-4f4f-a28d-d32d99521e16%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

> <elasticsearch.yml><ES_SEARCHGUARD.log>

[Juan Martínez Palazón]

Hi SG,

Please find attached the elastisearch.yml with your config entry.

The log I sent you is completed, that is all the application show from starting. If I try to perform a query I can see this:

[2017-02-21T11:51:43,495][ERROR][c.f.s.a.BackendRegistry  ] Not yet initialized (you may need to run sgadmin)

And if I run sgadmin:

sgadmin.sh -ts ~/NODE/ES_config/truststore.jks -tspass <mypass> -ks ~/NODE/ES_config/datacluster1-keystore.jks -kspass <mypass> -cd ~/NODE/src/elastic/plugins/search-guard-5/sgconfig/ -cn ES-dev -h datacluster1 -p 49300
Search Guard Admin v5
Will connect to datacluster1:49300 ... done
Contacting elasticsearch cluster 'ES-dev' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{N_kzYZQTQ26KJA_7wNE-Ug}{datacluster1}{<ip>:49300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

Cannot retrieve cluster state due to: None of the configured nodes are available: [{#transport#-1}{N_kzYZQTQ26KJA_7wNE-Ug}{datacluster1}{<ip>:49300}]. This is not an error, will keep on trying ...

   * Try running sgadmin.sh with -icl and -nhnv (If thats works you need to check your clustername as well as hostnames in your SSL certificates)
   * If this is not working, try running sgadmin.sh with --diagnose and see diagnose trace log file)

And no more entries are shown in the ES logs.

[SG]

the dn configured as "searchguard.authcz.admin_dn" must match the dn in your keystore (~/NODE/ES_config/datacluster1-keystore.jks) you use for sgadmin - and it should not be node certificate but a client certificate.
I recommend that start with the search guard bundle (https://github.com/floragunncom/search-guard/wiki/Search-Guard-Bundle) to see how it's supposed to work. Please also read http://floragunncom.github.io/search-guard-ssl-docs/quickstart.html and https://github.com/floragunncom/search-guard-docs/blob/master/quickstart.md and https://github.com/floragunncom/search-guard-docs/blob/master/sgadmin.md

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/5eeeacc0-f98e-4ad0-9b42-c10e8adbdc62%40googlegroups.com.

[Juan Martínez Palazón]

Hi SG,

I have solved it. My certificates were not created using the node hostname but a similar alias with a prefix -and I suppose the original name was required. When I amend that everything works fine.

Many thanks for your help.