Slow indexing with searchguard

[Jainin Shah]

Hi,

We have installed searchguard 2.3.2-beta2 in our elasticsearch cluster. Everything went well with the installation. But what we noticed is that bulk indexing (from hdfs to es particularly) is quite slow after enabling searchguard in the cluster.

For ex, we were indexing 50 million document in our cluster, which would generally take no more than few hours (usually between 2-3), but after installing searchguard the same indexing took more than 9 hours.

Is that an expected behavior while indexing with searchguard? and/or

Is that fixed (or improved) in the RC1 release?

Thank you

Jainin Shah

[SG]

This is not expected. Can you post your configuration (elasticsearch.yml and sg_config.yml) and explain a little bit more in detail how your ingestion work and what kind of authentication you use.
Forn example: Do you use HTTP or Java Client for indexing? If HTTP: Do you have SSL enabled and if so do you use native OpenSSL support?
Maybe SSL is slowing down the whole thing. Beta1 introduced caching so hitting the authentication backend should normally not cause any pain.

Please provide also informations for the folowing questions:
- How many nodes do you have in your cluster?
- JVM vendor and version
- Operation system vendor and version
- OpenSSL version

Thx

> --
> You received this message because you are subscribed to the Google Groups "Search Guard" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to search-guard...@googlegroups.com.
> To post to this group, send email to search...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/aea72d49-2f6e-40c2-8d97-464744316fd6%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Jainin Shah]

Thanks for the prompt reply.

To answer your questions:

- How many nodes do you have in your cluster? : 10

- JVM vendor and version :

java version "1.7.0_75"

Java(TM) SE Runtime Environment (build 1.7.0_75-b13)

Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)

- Operation system vendor and version 

CentOS release 6.8 (Final)

- OpenSSL version 

OpenSSL 1.0.1e-fips 11 Feb 2013

Also, attached are the elasticsearch.yml file and sg_config.yml file. sg_config.yml is exactly the same as it comes with the package.

And we do not use java client to index, it's going through http using elasticsearch-spark. And we use basic authentication (username and password) with searchguard.

[SG]

Can you try RC1 because it contains SSL related fixes which should improve speed and fix a possible memory leak with OpenSSL (Beware: config has slightly changed for RC1)

I see from you config file that you NOT using OpenSSL (enable_openssl_if_available: false). We strongly recommend to let OpenSSL handle SSL because Java built-in SSL support in slow
(especially in Java 7, therefore you should also consider upgrading to Oracle Java 8)

See here how to enable OpenSSL support for Search Guard.
https://github.com/floragunncom/search-guard-ssl/wiki/Open-SSL-setup/f618d0efa96cc9d598b28439b25464047c5cc98a

Another question: It seems do that you not encrypt your HTTP traffic. Plain HTTP together with basic authentication means that the plaintext password will be sent over the network.
Consider setting searchguard.ssl.http.enabled: true to cope with that.

> To view this discussion on the web visit https://groups.google.com/d/msgid/search-guard/c31c23da-503b-4095-9c9a-a326b1beef8c%40googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

> <sg_config.yml><elasricsearch.yml>

[Jainin Shah]

Thanks. Will give RC1 and OpenSSL a try and see if that increases that speed.