SSL exception when using the remote reindex API

[Andreas Freudenreich]

* Search Guard and Elasticsearch version: searchguard 5.1.1 r11

* Installed and used enterprise modules, if any: DLSFLS, LDAP

* JVM version and operating system version: java oracle1.8.0.151, RHEL 7.4

* Search Guard configuration files: don't think they are relevant here

* Elasticsearch log messages on debug level: on request (contains basically the same info as below)

* Other installed Elasticsearch or Kibana plugins, if any: n/a

Hi,

I tried to run following remote reindex command (in the DEV tools console or using curator) - adding user/password doesn't change anything:

POST _reindex

{

  "source": {

    "remote": {

      "host": "https://kib-webtst01.ourdomain:9200"

    },

    "index": " logstash-test"

  },

  "dest": {

    "index": "logstash-radius-udet-test-2017"

  }

}

Which returns:

{

  "error": {

    "root_cause": [

      {

        "type": "s_s_l_handshake_exception",

        "reason": "General SSLEngine problem"

      }

    ],

    "type": "s_s_l_handshake_exception",

    "reason": "General SSLEngine problem",

    "caused_by": {

      "type": "s_s_l_handshake_exception",

      "reason": "General SSLEngine problem",

      "caused_by": {

        "type": "validator_exception",

        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",

        "caused_by": {

          "type": "sun_cert_path_builder_exception",

          "reason": "unable to find valid certification path to requested target"

        }

      }

    }

  },

  "status": 500

}

I have included the CA certificate (and the signed node certificate) in the truststore.jks within /etc/elasticsearch. The remote server is also whitelisted in the ES config.

In the truststore:

# keytool -list -keystore /etc/elasticsearch/truststore.jks

...

kib-webtst01.ourdomain, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): CE:34:40:87:D3:23:1B:5A:C1:DE:81:BA:02:2F:6A:22:93:38:F4:CD

root-ca-chain, Jan 17, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): A8:18:97:BA:54:94:2A:B8:75:95:E6:82:A0:27:0C:E7:2C:40:76:49

root-ca-chain-test, Nov 2, 2017, trustedCertEntry,

Certificate fingerprint (SHA1): 74:4D:F6:97:04:F6:E2:EB:BA:0C:A8:D9:7A:16:1D:89:C4:D9:20:7C

When I run SSLPoke (https://gist.github.com/4ndrej/4547029) I can successfully connect to the remote host using the same truststore.jks.

# java -Djavax.net.ssl.trustStore=/etc/elasticsearch/truststore.jks -cp /tmp SS LPoke kib-webtst01.ourdomain 9200

Successfully connected

Thanks for any hints,

Andreas

[SG]

Before digging deeper into this: Does the exception really only occure for remote reindex API?
So local reindexing or other api calls are working?

Pls. post also your elasticsearch.yml, thx

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/f4aee903-2e99-4b46-9114-f6cc33949e40%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Andreas Freudenreich]

Other API calls and local indexing on both source and target cluster work (both are using searchguard):

POST _reindex

{

  "source": {

    "index": "logstash-test-2017"

  },

  "dest": {

    "index": "logstash-test-2017_reindexed"

  }

}

Result:

{

  "took": 1239,

  "timed_out": false,

  "total": 1,

  "updated": 0,

  "created": 1,

  "deleted": 0,

  "batches": 1,

  "version_conflicts": 0,

  "noops": 0,

  "retries": {

    "bulk": 0,

    "search": 0

  },

  "throttled_millis": 0,

  "requests_per_second": -1,

  "throttled_until_millis": 0,

  "failures": []

}

[SG]

tracked here https://github.com/elastic/elasticsearch/issues/27267

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/491d58da-d490-47ad-8718-49c235463274%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

> <es_config_remote.yml><es_config_source.yml>