NoNodeAvailableException when trying to connect to ES using TransportClient

[oliver.s...@semalytix.de]

Hello!

Based on your recommendation I'm cross-posting my question on StackOverflow (see http://stackoverflow.com/questions/41878863/nonodeavailableexception-when-trying-to-connect-to-es-using-transportclient#comment70967577_41878863). In addition to the original post I provided the settings for Search Guard in elasticsearch.yml below. Please note that I haven't changed these settings since setup, so they've been active yesterday aswell.

Trying to connect to a remote ElasticSearch cluster protected by Search Guard using the `TransportClient` I keep running into a `NoNodeAvailableException`:

    Caused by: org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{e7l0vv0zRhaeESmQBBmR1w}{remote.cluster.de}{127.0.0.1:9300}]

After reading lots of posts regarding this issue I've tried:

* setting `client.transport.sniff` to `true` and `false`
* setting `cluster.name` to the remote name, an arbitrary name and not at all
* setting `http.enabled` to `true` and `false`

Unfortunately I don't know what's left to check. Could anyone help me locate the issue please?

Here's my Spring configuration class:

    @Configuration
    @PropertySource("classpath:elastic.properties")
    public class ElasticConfiguration {

        @Value("${searchguard.ssl.transport.keystore_password}")
        private String keystorePassword;

        @Value("${searchguard.ssl.transport.truststore_password}")
        private String truststorePassword;

        @Bean
        public Client elasticClient() throws UnknownHostException {
            Settings settings = Settings.builder()
                    .put("cluster.name", "the remote cluster name")
                    .put("path.home", ".")
                    .put("client.transport.sniff", false)
                    .put("http.enabled", false)
                    .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_KEYSTORE_FILEPATH, "keystore.jks")
                    .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_TRUSTSTORE_FILEPATH, "truststore.jks")
                    .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_KEYSTORE_PASSWORD, keystorePassword)
                    .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_TRUSTSTORE_PASSWORD, truststorePassword)
                    .build();

            TransportClient tc = new PreBuiltTransportClient(settings, Arrays.asList(SearchGuardSSLPlugin.class));
            tc.addTransportAddress(
                    new InetSocketTransportAddress(
                            InetAddress.getByName("remote.cluster.de"), 9300));

            return tc;
        }
    }

Here's the save method I'm calling in my Spring `CommandLineRunner`:

    @Override
    public void save(Employee employee) throws IOException {
        XContentBuilder builder = XContentFactory.jsonBuilder()
                .startObject()
                    .field("id", employee.getId())
                    .field("name", employee.getName())
                    .field("age", employee.getAge())
                .endObject();

        // this.client is above configured TransportClient auto-injected by Spring
        IndexResponse response = this.client.prepareIndex("demo-company", "employee")
                .setSource(builder)
                .get();

    }

Here's the output of `yum info elasticsearch` showing the installed version:

    Installed Packages
    Name        : elasticsearch
    Arch        : noarch
    Version     : 5.1.1
    Release     : 1
    Size        : 35 M
    Repo        : installed
    From repo   : elasticsearch-5.x
    Summary     : Elasticsearch is a distributed RESTful search engine built for the cloud. Reference documentation can be found at
            : https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html and the 'Elasticsearch: The Definitive Guide'
            : book can be found at https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html
    URL         : https://www.elastic.co/
    License     : 2009
    Description : Elasticsearch subproject :distribution:rpm

Here's an excerpt of my pom.xml showing I included the correct version of the libraries:

        <dependency>
            <groupId>org.elasticsearch</groupId>
            <artifactId>elasticsearch</artifactId>
            <version>5.1.1</version>
        </dependency>
        <dependency>
            <groupId>org.elasticsearch.client</groupId>
            <artifactId>transport</artifactId>
            <version>5.1.1</version>
        </dependency>
        <dependency>
            <groupId>com.floragunn</groupId>
            <artifactId>search-guard-ssl</artifactId>
            <version>5.1.1-19</version>
        </dependency>

In `elasticsearch.yml` the bind is set to:

    network.host: 0.0.0.0

    searchguard.ssl.transport.enforce_hostname_verification: false
    # omitting ssl.transport trust- and keystore file and pass

    searchguard.ssl.http.enabled: true
    # omitting ssl.http trust- and keystore file and pass

    # omitting searchguard.authcz.admin_dn

Here's the telnet output showing the connectivity:

    [oschlueter@B5400 ~]$ telnet 127.0.0.1 9300
    Trying ::1...
    Connected to localhost.
    Escape character is '^]'.
    ^]
    telnet>

[SG]

try adding "searchguard.ssl.transport.enforce_hostname_verification: false" also to the transport client properties

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/6bf2e0c3-6abd-4c1e-8119-72427b2069d3%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[oliver.s...@semalytix.de]

Thank you for the suggestion! I added this line to the Settings builder:

    .put(SSLConfigConstants.SEARCHGUARD_SSL_TRANSPORT_ENFORCE_HOSTNAME_VERIFICATION, false)

and executed `mvn clean package` before executing the program again. Unfortunately it yields the exact same error. 

[SG]

Ok, then maybe its an issue with the bound vs. publish address. Can you post the part of the elasticsearch logfile(s) containing these informations:

[2017-01-27T19:18:53,660][INFO ][o.e.t.TransportService ] [es1] publish_address {xxx:9300}, bound_addresses {[fe80::1]:9300}, {[::1]:9300}, {xxx:9300}
[2017-01-27T19:18:56,776][INFO ][o.e.h.HttpServer ] [es1] publish_address {xxx:9200}, bound_addresses {[fe80::1]:9200}, {[::1]:9200}, {xxx:9200}

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/5806e4bf-d3cc-41d8-b498-75f098da06b4%40googlegroups.com.

[Fabien Wernli]

As Jochen said, try adding the following in your elasticsearch.yml config:

network.host: 10.0.0.1

(replace with your IPv4)

The problem is that ES can only publish one address, so I guess it's picking the wrong one (localhost) when using the unspecified 0.0.0.0 address