Please help configuring Elasticsearch with Search Guard

ZillaYT

unread,

Oct 6, 2016, 11:38:32 AM10/6/16

to Search Guard

This is different from Elastic search will not start after I change data path. I resolved that by updating to v2.4.x

ES v2.4.1

SG-SSL v 2.4.1.16

SG v2.4.1.6

CentOS 7.2

Im able to run ES with SSL. I generated the certs/keys via the example script from Search Guard. Here is the info on the client cert that I generated

> openssl x509 -noout -subject -in kirk-signed.pem -text
subject= /C=US/L=Raleigh/O=client/OU=client/CN=kirk
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 3 (0x3)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: DC=com, DC=example, O=Example Com Inc., OU=Example Com Inc. Signing CA, CN=Example Com Inc. Signing CA
        Validity
            Not Before: Oct  6 14:47:01 2016 GMT
            Not After : Oct  6 14:47:01 2018 GMT
        Subject: C=US, L=Raleigh, O=client, OU=client, CN=kirk
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)

If I understand correctly, I need to have the following line in my elasticsearch.yml file, correct?

# Enable SSL via Search Guard SSL plugin
# Enable HTTPS
searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node-0-keystore.jks
searchguard.ssl.http.keystore_password: pw
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password: pw

# Enable SSL between ES nodes
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.keystore_password: pw
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: pw
searchguard.ssl.transport.enforce_hostname_verification: false

# for Search Guard
searchguard.authcz.admin_dn:
  - "cn=kirk, ou=client, o=client, l=Raleigh, c=US"

searchguard.cert.oid: '1.2.3.4.5.5'

But when I run sgadmin.sh, it just times out

> /usr/share/elasticsearch/plugins/search-guard-2/tools/sgadmin.sh -cd /etc/elasticsearch/ -ks kirk-keystore.jks -ts truststore.jks -nhnv -kspass pw -tspass pw
Will connect to localhost:9300 ... done
Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.

And I see these in elasticsearch.log

[2016-10-06 15:17:41,354][DEBUG][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] Node started, try to initialize it. Wait for at least yellow cluster state....
[2016-10-06 15:17:41,523][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:41,667][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:41,698][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:41,728][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:17:42,099][DEBUG][com.floragunn.searchguard.configuration.SearchGuardIndexSearcherWrapperModule] FLS/DLS not enabled
[2016-10-06 15:18:11,746][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] index 'searchguard' not healthy yet, we try again ... (Reason: timeout)
[2016-10-06 15:18:44,747][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] index 'searchguard' not healthy yet, we try again ... (Reason: timeout)
[2016-10-06 15:19:17,749][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [Arize] index 'searchguard' not healthy yet, we try again ... (Reason: timeout)

What am I missing?

Fabien Wernli

unread,

Oct 7, 2016, 4:10:38 AM10/7/16

to Search Guard

When disabling searchguard, is the 'searchguard' index Green?

Jochen Kressin

unread,

Oct 7, 2016, 1:18:25 PM10/7/16

to Search Guard

We can't reproduce this behavior, so:

Is this a fresh install of ES 2.4.1?
How many nodes are in the cluster?
Do you have the data.dir pointing to a different location?
Can you post the complete elasticsearch.yml file
Can you post the complete logfiles, from startup and when you try to execute sgadmin?

Please set SG loglevel to debug (com.floragunn: DEBUG)

As Fabien asked: What is your cluster state when you start ES without the SG plugins?

ZillaYT

unread,

Oct 13, 2016, 2:35:33 PM10/13/16

to Search Guard

I have the same issue. With me...

ES 2-node cluster goes green, without SG-SSL or SG plugin
Stop ES
Install SG-SSL plugin
Start ES, ES goes green
Stop ES
Install SG
Start ES, ES will not even go yellow

ES v2.4.0

SG-SSl v2.4.0.16

SG v2.4.0.7

I opened a ticket here Elasticsearch cluster never goes (at least) yellow with Search Guard #218

in...@search-guard.com

unread,

Oct 15, 2016, 4:25:57 PM10/15/16

to Search Guard

try

searchguard.ssl.transport.resolve_hostname: true

searchguard.ssl.transport.enable_openssl_if_available: false

on all nodes in elasticsearch.yml

If this does not help please send the complete logfile.

ZillaYT

unread,

Oct 17, 2016, 10:00:24 AM10/17/16

to Search Guard

Here are the logs, right after I restart Elasticsearch


[2016-10-17 13:49:21,403][INFO ][node                     ] [ip-10-22-9-4] version[2.4.0], pid[22291], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-10-17 13:49:21,403][INFO ][node                     ] [ip-10-22-9-4] initializing ...
[2016-10-17 13:49:22,079][INFO ][com.floragunn.searchguard.ssl.SearchGuardSSLPlugin] Search Guard 2 plugin also available
[2016-10-17 13:49:22,086][INFO ][com.floragunn.searchguard.SearchGuardPlugin] Node [ip-10-22-9-4] is a transportClient: false/tribeNode: false/tribeNodeClient: false
[2016-10-17 13:49:22,160][INFO ][plugins                  ] [ip-10-22-9-4] modules [reindex, lang-expression, lang-groovy], plugins [head, search-guard-ssl, kopf, search-guard-2], sites [head, kopf]
[2016-10-17 13:49:22,192][INFO ][env                      ] [ip-10-22-9-4] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [26.8gb], net total_space [29.9gb], spins? [unknown], types [rootfs]
[2016-10-17 13:49:22,192][INFO ][env                      ] [ip-10-22-9-4] heap size [1007.3mb], compressed ordinary object pointers [true]
[2016-10-17 13:49:22,257][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1e-fips 11 Feb 2013 available
[2016-10-17 13:49:22,257][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Open SSL OpenSSL 1.0.1e-fips 11 Feb 2013 available
[2016-10-17 13:49:22,762][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively
[2016-10-17 13:49:22,878][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] HTTPS client auth mode OPTIONAL
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit.. That is not an issue, it just limits possible encryption strength. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files'
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportClientProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransportServerProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTPProvider:OPENSSL with ciphers [TLS_DHE_DSS_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_DHE_DSS_WITH_AES_128_CBC_SHA256, TLS_DHE_DSS_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_256_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_DHE_DSS_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_DHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslTransport protocols [TLSv1.2, TLSv1.1]
[2016-10-17 13:49:22,901][INFO ][com.floragunn.searchguard.ssl.SearchGuardKeyStore] sslHTTP protocols [TLSv1.2, TLSv1.1]
[2016-10-17 13:49:23,117][INFO ][http                     ] [ip-10-22-9-4] Using [org.elasticsearch.http.netty.NettyHttpServerTransport] as http transport, overridden by [search-guard2]
[2016-10-17 13:49:23,220][INFO ][com.floragunn.searchguard.configuration.ConfigurationModule] FLS/DLS valve not bound (noop)
[2016-10-17 13:49:23,222][INFO ][com.floragunn.searchguard.auditlog.AuditLogModule] Auditlog not available
[2016-10-17 13:49:23,308][INFO ][transport                ] [ip-10-22-9-4] Using [com.floragunn.searchguard.transport.SearchGuardTransportService] as transport service, overridden by [search-guard2]
[2016-10-17 13:49:23,308][INFO ][transport                ] [ip-10-22-9-4] Using [com.floragunn.searchguard.ssl.transport.SearchGuardSSLNettyTransport] as transport, overridden by [search-guard-ssl]
[2016-10-17 13:49:25,430][INFO ][node                     ] [ip-10-22-9-4] initialized
[2016-10-17 13:49:25,430][INFO ][node                     ] [ip-10-22-9-4] starting ...
[2016-10-17 13:49:25,512][INFO ][com.floragunn.searchguard.transport.SearchGuardTransportService] [ip-10-22-9-4] publish_address {10.22.9.4:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {10.22.9.4:9300}
[2016-10-17 13:49:25,516][INFO ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] Check if searchguard index exists ...
[2016-10-17 13:49:25,523][DEBUG][action.admin.indices.exists.indices] [ip-10-22-9-4] no known master node, scheduling a retry
[2016-10-17 13:49:25,533][INFO ][discovery                ] [ip-10-22-9-4] elk-nova-devops/HE1yxxjSTy-wct_4srljUw
[2016-10-17 13:49:55,534][WARN ][discovery                ] [ip-10-22-9-4] waited for 30s and no initial state was set by the discovery
[2016-10-17 13:49:55,549][INFO ][http                     ] [ip-10-22-9-4] publish_address {10.22.9.4:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {10.22.9.4:9200}
[2016-10-17 13:49:55,549][INFO ][node                     ] [ip-10-22-9-4] started
[2016-10-17 13:49:57,669][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-10-17 13:50:02,266][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-10-17 13:50:25,535][DEBUG][action.admin.indices.exists.indices] [ip-10-22-9-4] timed out while retrying [indices:admin/exists] after failure (timeout [1m])
[2016-10-17 13:50:25,538][ERROR][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] Failure while checking searchguard index MasterNotDiscoveredException[null]
MasterNotDiscoveredException[null]
        at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:234)
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
        at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
[2016-10-17 13:50:25,543][DEBUG][action.admin.cluster.health] [ip-10-22-9-4] no known master node, scheduling a retry
[2016-10-17 13:50:25,587][INFO ][cluster.service          ] [ip-10-22-9-4] new_master {ip-10-22-9-4}{HE1yxxjSTy-wct_4srljUw}{10.22.9.4}{10.22.9.4:9300}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-10-17 13:50:25,664][INFO ][gateway                  ] [ip-10-22-9-4] recovered [1] indices into cluster_state
[2016-10-17 13:50:34,423][ERROR][com.floragunn.searchguard.auth.BackendRegistry] Not yet initialized
[2016-10-17 13:50:55,600][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] index 'searchguard' not healthy yet, we try again ... (Reason: timeout)
[2016-10-17 13:51:28,601][WARN ][com.floragunn.searchguard.action.configupdate.TransportConfigUpdateAction] [ip-10-22-9-4] index 'searchguard' not healthy yet, we try again ... (Reason: timeout)

The last line repeats indefinitely.

ZillaYT

unread,

Oct 17, 2016, 10:53:36 AM10/17/16

to Search Guard

Here is my YML file, with the comments removed for ease in reading.

cluster.name: elk-nova-devops
node.name: ip-10-22-9-4
path.logs: /var/log/elasticsearch/
bootstrap.mlockall: true

network.host: [ _site_, _local_ ]
http.port: 9200
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [ "10.22.9.4" ]
discovery.zen.minimum_master_nodes: 1
discovery.zen.ping_timeout: 60s
action.auto_create_index: .marvel-*,.marvel_*,.security
action.auto_create_index: true

script.inline: on
script.indexed: on
index.number_of_shards: 1

index.number_of_replicas: 1

index.routing.allocation.disable_allocation: false

marvel.agent.enabled: False
marvel.agent.interval: 1m
marvel.history.duration: 1



searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_filepath: node-0-keystore.jks
searchguard.ssl.http.keystore_password: pw
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_password:

 pw
searchguard.ssl.http.enable_openssl_if_available: true



searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.keystore_password: pw
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_password: pw
searchguard.ssl.transport.enforce_hostname_verification: false


searchguard.ssl.transport.enable_openssl_if_available: true

searchguard.authcz.admin_dn:
  - cn=kirk, ou=client, o=client, l=Test, c=DE

On Saturday, October 15, 2016 at 4:25:57 PM UTC-4, in...@search-guard.com wrote:

Reply all

Reply to author

Forward