Enabling OpenSSL with Elasticsearch 6.2.1

[dc...@galileo.io]

Hi,

I'm trying to get Elasticsearch to start with OpenSSL, but I'm receiving an error when starting Elasticsearch.

[2018-02-20T16:29:59,742][INFO ][o.e.n.Node ] [] initializing ...
[2018-02-20T16:29:59,847][INFO ][o.e.e.NodeEnvironment ] [9KwCLS3] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [6.9gb], net total_space [7.7gb], types [ext4]
[2018-02-20T16:29:59,848][INFO ][o.e.e.NodeEnvironment ] [9KwCLS3] heap size [1.7gb], compressed ordinary object pointers [true]
[2018-02-20T16:29:59,860][INFO ][o.e.n.Node ] node name [9KwCLS3] derived from node ID [9KwCLS3GQC67P4liY3drtw]; set [node.name] to override
[2018-02-20T16:29:59,860][INFO ][o.e.n.Node ] version[6.2.1], pid[112], build[7299dc3/2018-02-07T19:34:26.990113Z], OS[Linux/4.9.76-3.78.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
[2018-02-20T16:29:59,860][INFO ][o.e.n.Node ] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.gHCP8Nfa, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Xmx1778m, -Xms1778m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
[2018-02-20T16:30:00,952][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /usr/share/elasticsearch/config
[2018-02-20T16:30:01,058][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative]
[2018-02-20T16:30:01,348][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively
[2018-02-20T16:30:01,527][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK
[2018-02-20T16:30:01,527][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK
[2018-02-20T16:30:01,527][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK
[2018-02-20T16:30:01,527][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1]
[2018-02-20T16:30:01,528][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1]

Based on http://docs.search-guard.com/latest/openssl, I've installed openssl, apr, and the appropriate jar file into plugins/search-guard-6. https://bintray.com/floragunncom/netty-tcnative/download_file?file_path=netty-tcnative-openssl-1.0.2-dynamic-2.0.5.Final-fedora-linux-x86_64.jar

Q: What should I change to get Open SSL to be made available to Elasticsearch?

More information below, for context to my ES environment.

I'm using Elasticsearch 6.2.1 and Search Guard 6.2.1-21.0. This is the open source version.

[elasticsearch@15e41b0c864c ~]$ java -version

openjdk version "1.8.0_161"

OpenJDK Runtime Environment (build 1.8.0_161-b14)

OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

[elasticsearch@15e41b0c864c ~]$ cat /etc/redhat-release

CentOS Linux release 7.4.1708 (Core)

[elasticsearch@15e41b0c864c ~]$ rpm -qa | grep openssl

openssl-libs-1.0.2k-8.el7.x86_64

openssl-1.0.2k-8.el7.x86_64

[elasticsearch@15e41b0c864c ~]$ rpm -qa | grep apr

apr-1.4.8-3.el7_4.1.x86_64

I have Elasticsearch plugins installed for discovery-ec2 and repository-s3.

My elasticsearch.yml configuration for Search Guard

searchguard.enterprise_modules_enabled: false

searchguard.ssl.transport.pemkey_filepath: node.key.pem

searchguard.ssl.transport.pemkey_password: __KEY_PASS__

searchguard.ssl.transport.pemcert_filepath: node.crt.pem

searchguard.ssl.transport.pemtrustedcas_filepath: signing-ca.pem

searchguard.ssl.transport.enforce_hostname_verification: false

searchguard.ssl.transport.enable_openssl_if_available: true

searchguard.ssl.http.enabled: true

searchguard.ssl.http.pemkey_filepath: node.key.pem

searchguard.ssl.http.pemkey_password: __KEY_PASS__

searchguard.ssl.http.pemcert_filepath: node.crt.pem

searchguard.ssl.http.pemtrustedcas_filepath: signing-ca.pem

searchguard.ssl.http.clientauth_mode: OPTIONAL

searchguard.ssl.http.enable_openssl_if_available: true

searchguard.audit.type: internal_elasticsearch

searchguard.enable_snapshot_restore_privilege: true

searchguard.check_snapshot_restore_write_privileges: true

searchguard.restapi.roles_enabled: ["sg_all_access"]

Thanks!

Dan

--

When asking questions, please provide the following information:

* Search Guard and Elasticsearch version

* Installed and used enterprise modules, if any

* JVM version and operating system version

* Search Guard configuration files

* Elasticsearch log messages on debug level

* Other installed Elasticsearch or Kibana plugins, if any

[dc...@galileo.io]

Additional information. Interesting how ssl_openssl_available is set to false, even though I have OpenSSL and apr packages installed, as far as I can tell.

{
  "principal" : null,
  "peer_certificates" : "0",
  "ssl_protocol" : "TLSv1.2",
  "ssl_cipher" : "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
  "ssl_openssl_available" : false,
  "ssl_openssl_version" : -1,
  "ssl_openssl_version_string" : null,
  "ssl_openssl_non_available_cause" : "java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative]",
  "ssl_openssl_supports_key_manager_factory" : false,
  "ssl_openssl_supports_hostname_validation" : false,
  "ssl_provider_http" : "JDK",
  "ssl_provider_transport_server" : "JDK",
  "ssl_provider_transport_client" : "JDK"
}

Noting that with the same set up, Elasticsearch 6.1.3 seems to find the OpenSSL library.

{

  "principal" : null,

  "peer_certificates" : "0",

  "ssl_protocol" : "TLSv1.2",

  "ssl_cipher" : "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",

  "ssl_openssl_available" : true,

  "ssl_openssl_version" : 268443839,

  "ssl_openssl_version_string" : "OpenSSL 1.0.2k-fips  26 Jan 2017",

  "ssl_openssl_non_available_cause" : "",

  "ssl_openssl_supports_key_manager_factory" : true,

  "ssl_openssl_supports_hostname_validation" : true,

  "ssl_provider_http" : "OPENSSL",

  "ssl_provider_transport_server" : "OPENSSL",

  "ssl_provider_transport_client" : "OPENSSL"

}

[SG]

you need the 2.0.7 version tcnative instead of 2.0.5, i updated the docs accordingly.
So pls try https://bintray.com/floragunncom/netty-tcnative/download_file?file_path=netty-tcnative-openssl-1.0.2-dynamic-2.0.7.Final-fedora-linux-x86_64.jar
See https://github.com/floragunncom/search-guard-docs/blob/6.x/_docs/tls_openssl.md for all download possibilities (dynamic and static)


> Am 20.02.2018 um 17:49 schrieb dc...@galileo.io:
>
> Additional information
>
> {

> "principal" : null,
> "peer_certificates" : "0",
> "ssl_protocol" : "TLSv1.2",
> "ssl_cipher" : "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
> "ssl_openssl_available" : false,
> "ssl_openssl_version" : -1,
> "ssl_openssl_version_string" : null,
> "ssl_openssl_non_available_cause" : "java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64, netty_tcnative_linux_x86_64_fedora, netty_tcnative]",
> "ssl_openssl_supports_key_manager_factory" : false,
> "ssl_openssl_supports_hostname_validation" : false,
> "ssl_provider_http" : "JDK",
> "ssl_provider_transport_server" : "JDK",
> "ssl_provider_transport_client" : "JDK"
> }
>
>
>

> On Tuesday, February 20, 2018 at 11:36:06 AM UTC-5, dc...@galileo.io wrote:
> Hi,
>
> I'm trying to get Elasticsearch to start with OpenSSL, but I'm receiving an error when starting Elasticsearch.
>
> [2018-02-20T16:29:59,742][INFO ][o.e.n.Node ] [] initializing ...
> [2018-02-20T16:29:59,847][INFO ][o.e.e.NodeEnvironment ] [9KwCLS3] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/xvda1)]], net usable_space [6.9gb], net total_space [7.7gb], types [ext4]
> [2018-02-20T16:29:59,848][INFO ][o.e.e.NodeEnvironment ] [9KwCLS3] heap size [1.7gb], compressed ordinary object pointers [true]
> [2018-02-20T16:29:59,860][INFO ][o.e.n.Node ] node name [9KwCLS3] derived from node ID [9KwCLS3GQC67P4liY3drtw]; set [node.name] to override
> [2018-02-20T16:29:59,860][INFO ][o.e.n.Node ] version[6.2.1], pid[112], build[7299dc3/2018-02-07T19:34:26.990113Z], OS[Linux/4.9.76-3.78.amzn1.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_161/25.161-b14]
> [2018-02-20T16:29:59,860][INFO ][o.e.n.Node ] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.gHCP8Nfa, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Xmx1778m, -Xms1778m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config]
> [2018-02-20T16:30:00,952][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /usr/share/elasticsearch/config

> [2018-02-20T16:30:01,058][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-inJDK SSL) because of java.lang.IllegalArgumentException: Failed to load any of the given libraries: [netty_tcnative_linux_x86_64,netty_tcnative_linux_x86_64_fedora, netty_tcnative]
> [2018-02-20T16:30:01,348][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /usr/share/elasticsearch/config/, from there the key- andtruststore files are resolved relatively

> --
> You received this message because you are subscribed to the Google Groups "Search Guard Community Forum" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/62e28c04-944a-4bd4-9979-55d496adc0e6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[dc...@galileo.io]

Thanks for the quick reply! Using 2.0.7 version of tcnative with Elasticsearch 6.2.1 works on my setup.

Thank you for helping me out with the problem. In addition, will http://docs.search-guard.com/latest/openssl be updated as well?

[Jochen Kressin]

Sure, there's a major update this week including also this correction.