[2018-02-17T22:27:06,541][INFO ][o.e.n.Node ] [] initializing ... [2018-02-17T22:27:06,639][INFO ][o.e.e.NodeEnvironment ] [Qrogsm-] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [184gb], net total_space [274gb], types [rootfs] [2018-02-17T22:27:06,640][INFO ][o.e.e.NodeEnvironment ] [Qrogsm-] heap size [989.8mb], compressed ordinary object pointers [true] [2018-02-17T22:27:06,678][INFO ][o.e.n.Node ] node name [Qrogsm-] derived from node ID [Qrogsm-GSiOX4dG3YeCUZg]; set [node.name] to override [2018-02-17T22:27:06,679][INFO ][o.e.n.Node ] version[6.2.1], pid[32616], build[7299dc3/2018-02-07T19:34:26.990113Z], OS[Linux/3.10.0-327.28.3.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_121/25.121-b13] [2018-02-17T22:27:06,680][INFO ][o.e.n.Node ] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.OQ1hbrUi, -XX:+HeapDumpOnOutOfMemoryError, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/home_local/elasticsearch/elasticsearch-6.2.1, -Des.path.conf=/home_local/elasticsearch/elasticsearch-6.2.1/config] [2018-02-17T22:27:07,530][INFO ][c.f.s.SearchGuardPlugin ] ES Config path is /home_local/elasticsearch/elasticsearch-6.2.1/config [2018-02-17T22:27:07,586][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Open SSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2018-02-17T22:27:07,750][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /home_local/elasticsearch/elasticsearch-6.2.1/config/, from there the key- and truststore files are resolved relatively [2018-02-17T22:27:07,837][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] AES-256 not supported, max key length for AES is 128 bit. (That is not an issue, it just limits possible encryption strength. To enable AES 256 install 'Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files') [2018-02-17T22:27:07,837][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : JDK [2018-02-17T22:27:07,837][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : JDK [2018-02-17T22:27:07,837][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider : JDK [2018-02-17T22:27:07,837][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1] [2018-02-17T22:27:07,837][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer : [TLSv1.2, TLSv1.1] [2018-02-17T22:27:08,043][INFO ][c.f.s.SearchGuardPlugin ] Clustername: ELK1 [2018-02-17T22:27:08,084][WARN ][c.f.s.SearchGuardPlugin ] Directory /home_local/elasticsearch/elasticsearch-6.2.1/config has insecure file permissions (should be 0700) [2018-02-17T22:27:08,084][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/kirk.pem has insecure file permissions (should be 0600) [2018-02-17T22:27:08,084][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/esnode.pem has insecure file permissions (should be 0600) [2018-02-17T22:27:08,084][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/root-ca.pem has insecure file permissions (should be 0600) [2018-02-17T22:27:08,085][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/esnode-key.pem has insecure file permissions (should be 0600) [2018-02-17T22:27:08,085][WARN ][c.f.s.SearchGuardPlugin ] File /home_local/elasticsearch/elasticsearch-6.2.1/config/kirk-key.pem has insecure file permissions (should be 0600) [2018-02-17T22:27:08,088][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [aggs-matrix-stats] [2018-02-17T22:27:08,088][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [analysis-common] [2018-02-17T22:27:08,088][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [ingest-common] [2018-02-17T22:27:08,088][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-expression] [2018-02-17T22:27:08,088][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-mustache] [2018-02-17T22:27:08,088][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [lang-painless] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [mapper-extras] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [parent-join] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [percolator] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [rank-eval] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [reindex] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [repository-url] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [transport-netty4] [2018-02-17T22:27:08,089][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded module [tribe] [2018-02-17T22:27:08,090][INFO ][o.e.p.PluginsService ] [Qrogsm-] loaded plugin [search-guard-6] [2018-02-17T22:27:08,110][INFO ][c.f.s.SearchGuardPlugin ] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in elasticsearch.yml [2018-02-17T22:27:10,071][INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2018-02-17T22:27:10,071][INFO ][c.f.s.a.i.AuditLogImpl ] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2018-02-17T22:27:10,072][INFO ][c.f.s.a.i.AuditLogImpl ] Configured Users to ignore: [kibanaserver] [2018-02-17T22:27:10,072][INFO ][c.f.s.a.i.AuditLogImpl ] Configured Requests to ignore: [] [2018-02-17T22:27:10,073][INFO ][c.f.s.a.i.AuditLogImpl ] Audit Log class: ESAuditLog [2018-02-17T22:27:10,434][DEBUG][o.e.a.ActionModule ] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin [2018-02-17T22:27:10,537][INFO ][o.e.d.DiscoveryModule ] [Qrogsm-] using discovery type [zen] [2018-02-17T22:27:11,048][INFO ][c.f.s.SearchGuardPlugin ] 4 Search Guard modules loaded so far: [Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl]] [2018-02-17T22:27:11,049][INFO ][o.e.n.Node ] initialized [2018-02-17T22:27:11,049][INFO ][o.e.n.Node ] [Qrogsm-] starting ... [2018-02-17T22:27:11,167][INFO ][o.e.t.TransportService ] [Qrogsm-] publish_address {172.25.180.171:9300}, bound_addresses {127.0.0.1:9300}, {172.25.180.171:9300} [2018-02-17T22:27:11,182][INFO ][o.e.b.BootstrapChecks ] [Qrogsm-] bound or publishing to a non-loopback address, enforcing bootstrap checks [2018-02-17T22:27:11,193][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ... [2018-02-17T22:27:11,199][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [Qrogsm-] no known master node, scheduling a retry [2018-02-17T22:27:14,267][INFO ][o.e.c.s.MasterService ] [Qrogsm-] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{O5pzpqGdTR2ynQKGQzhkDA}{172.25.180.171}{172.25.180.171:9300} [2018-02-17T22:27:14,276][INFO ][o.e.c.s.ClusterApplierService] [Qrogsm-] new_master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{O5pzpqGdTR2ynQKGQzhkDA}{172.25.180.171}{172.25.180.171:9300}, reason: apply cluster state (from master [master {Qrogsm-}{Qrogsm-GSiOX4dG3YeCUZg}{O5pzpqGdTR2ynQKGQzhkDA}{172.25.180.171}{172.25.180.171:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) [2018-02-17T22:27:14,309][INFO ][c.f.s.h.SearchGuardHttpServerTransport] [Qrogsm-] publish_address {172.25.180.171:9200}, bound_addresses {127.0.0.1:9200}, {172.25.180.171:9200} [2018-02-17T22:27:14,309][INFO ][o.e.n.Node ] [Qrogsm-] started [2018-02-17T22:27:14,825][INFO ][o.e.g.GatewayService ] [Qrogsm-] recovered [4] indices into cluster_state [2018-02-17T22:27:16,161][INFO ][o.e.c.r.a.AllocationService] [Qrogsm-] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[searchguard][0], [sg6-auditlog-2018.02.16][2], [sg6-auditlog-2018.02.16][3], [sg6-auditlog-2018.02.16][4]] ...]). [2018-02-17T22:27:16,317][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, issueDate=2018-02-16, expiryDate=2018-04-19, issuedTo=The world, issuer=floragunn GmbH, startDate=2018-02-16, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=[], expiresInDays=60, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@1596ea50, getMsgs()=[], getExpiresInDays()=60, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.] [2018-02-17T22:27:16,319][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Search Guard License Type: TRIAL, valid [2018-02-17T22:27:16,319][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Node 'Qrogsm-' initialized [2018-02-17T22:28:11,989][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver 128.135.112.111:389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:28:11,991][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver localhost:8389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:28:11,994][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for ivy2 [2018-02-17T22:28:44,684][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver 128.135.112.111:389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:28:44,684][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver localhost:8389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:28:44,685][ERROR][c.f.s.a.BackendRegistry ] Cannot retrieve roles for User [name=admin, roles=[admin], requestedTenant=null] from ldap due to ElasticsearchSecurityException[[org.ldaptive.LdapException@569640771::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389], providerException=null]]; nested: LdapException[Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389]]; org.elasticsearch.ElasticsearchSecurityException: [org.ldaptive.LdapException@569640771::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389], providerException=null] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:547) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:558) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.ldaptive.LdapException: Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:208) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:375) ~[?:?] ... 69 more [2018-02-17T22:31:09,054][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver 128.135.112.111:389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:31:09,056][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver localhost:8389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:31:09,056][ERROR][c.f.s.a.BackendRegistry ] Cannot retrieve roles for User [name=kibanaserver, roles=[], requestedTenant=null] from ldap due to ElasticsearchSecurityException[[org.ldaptive.LdapException@1689395825::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389], providerException=null]]; nested: LdapException[Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389]]; org.elasticsearch.ElasticsearchSecurityException: [org.ldaptive.LdapException@1689395825::resultCode=null, matchedDn=null, responseControls=null, referralURLs=null, messageId=-1, message=Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389], providerException=null] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:547) ~[dlic-search-guard-enterprise-modules-6.2-30.0.jar:6.2-30.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:558) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry$6.call(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:5058) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3708) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2416) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2299) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2212) [guava-23.0.jar:?] at com.google.common.cache.LocalCache.get(LocalCache.java:4147) [guava-23.0.jar:?] at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:5053) [guava-23.0.jar:?] at com.floragunn.searchguard.auth.BackendRegistry.authcz(BackendRegistry.java:549) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.auth.BackendRegistry.authenticate(BackendRegistry.java:416) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.checkAndAuthenticateRequest(SearchGuardRestFilter.java:125) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter.access$000(SearchGuardRestFilter.java:48) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at com.floragunn.searchguard.filter.SearchGuardRestFilter$1.handleRequest(SearchGuardRestFilter.java:75) [search-guard-6-6.2.1-21.0.jar:6.2.1-21.0] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:240) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:336) [elasticsearch-6.2.1.jar:6.2.1] at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:174) [elasticsearch-6.2.1.jar:6.2.1] at com.floragunn.searchguard.ssl.http.netty.ValidatingDispatcher.dispatchRequest(ValidatingDispatcher.java:63) [search-guard-ssl-6.2.1-25.1.jar:6.2.1-25.1] at org.elasticsearch.http.netty4.Netty4HttpServerTransport.dispatchRequest(Netty4HttpServerTransport.java:500) [transport-netty4-client-6.2.1.jar:6.2.1] at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:80) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at org.elasticsearch.http.netty4.pipelining.HttpPipeliningHandler.channelRead(HttpPipeliningHandler.java:68) [transport-netty4-client-6.2.1.jar:6.2.1] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:310) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:284) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1336) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1127) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) [netty-handler-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) [netty-codec-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: org.ldaptive.LdapException: Unable to connect to any of those ldap servers [128.135.112.111:389, localhost:8389] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection0(LDAPAuthorizationBackend.java:208) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.access$000(LDAPAuthorizationBackend.java:77) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:112) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend$1.run(LDAPAuthorizationBackend.java:109) ~[?:?] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.getConnection(LDAPAuthorizationBackend.java:109) ~[?:?] at com.floragunn.dlic.auth.ldap.backend.LDAPAuthorizationBackend.fillRoles(LDAPAuthorizationBackend.java:375) ~[?:?] ... 69 more [2018-02-17T22:32:13,554][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver 128.135.112.111:389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:32:13,556][WARN ][c.f.d.a.l.b.LDAPAuthorizationBackend] Unable to connect to ldapserver localhost:8389 due to ElasticsearchException[Empty file path for searchguard.ssl.transport.truststore_filepath]. Try next. [2018-02-17T22:32:13,558][WARN ][c.f.s.a.BackendRegistry ] Authentication finally failed for ivy2