[2019-01-30T09:43:31,902][INFO ][o.e.n.Node ] [l1_pweP] stopped [2019-01-30T09:43:31,902][INFO ][o.e.n.Node ] [l1_pweP] closing ... [2019-01-30T09:43:31,945][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [l1_pweP] uncaught exception in thread [Thread-3] java.lang.NullPointerException: null at org.elasticsearch.xpack.watcher.Watcher.close(Watcher.java:702) ~[?:?] at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-6.5.1.jar:6.5.1] at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:85) ~[elasticsearch-core-6.5.1.jar:6.5.1] at org.elasticsearch.node.Node.close(Node.java:923) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:103) ~[elasticsearch-core-6.5.1.jar:6.5.1] at org.elasticsearch.core.internal.io.IOUtils.close(IOUtils.java:61) ~[elasticsearch-core-6.5.1.jar:6.5.1] at org.elasticsearch.bootstrap.Bootstrap$4.run(Bootstrap.java:184) ~[elasticsearch-6.5.1.jar:6.5.1] [2019-01-30T09:46:22,619][INFO ][o.e.e.NodeEnvironment ] [l1_pweP] using [1] data paths, mounts [[/ (/dev/vda1)]], net usable_space [38.6gb], net total_space [96.8gb], types [ext4] [2019-01-30T09:46:22,624][INFO ][o.e.e.NodeEnvironment ] [l1_pweP] heap size [990.7mb], compressed ordinary object pointers [true] [2019-01-30T09:46:39,037][INFO ][o.e.n.Node ] [l1_pweP] node name derived from node ID [l1_pwePhQB2xYnGKq64Ksg]; set [node.name] to override [2019-01-30T09:46:39,038][INFO ][o.e.n.Node ] [l1_pweP] version[6.5.1], pid[1240], build[default/deb/8c58350/2018-11-16T02:22:42.182257Z], OS[Linux/4.4.0-141-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_191/25.191-b12] [2019-01-30T09:46:39,039][INFO ][o.e.n.Node ] [l1_pweP] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch.b1kJywLV, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:/var/log/elasticsearch/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=deb] [2019-01-30T09:46:58,856][INFO ][c.f.s.SearchGuardPlugin ] [l1_pweP] ES Config path is /etc/elasticsearch [2019-01-30T09:46:59,027][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] OpenSSL not available (this is not an error, we simply fallback to built-in JDK SSL) because of java.lang.ClassNotFoundException: io.netty.internal.tcnative.SSL [2019-01-30T09:46:59,043][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] Config directory is /etc/elasticsearch/, from there the key- and truststore files are resolved relatively [2019-01-30T09:46:59,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] TLS Transport Client Provider : JDK [2019-01-30T09:46:59,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] TLS Transport Server Provider : JDK [2019-01-30T09:46:59,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] TLS HTTP Provider : null [2019-01-30T09:46:59,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] Enabled TLS protocols for transport layer : [TLSv1.1, TLSv1.2] [2019-01-30T09:46:59,517][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] [l1_pweP] Enabled TLS protocols for HTTP layer : [] [2019-01-30T09:47:01,407][INFO ][c.f.s.SearchGuardPlugin ] [l1_pweP] Clustername: elasticsearch [2019-01-30T09:47:02,650][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/preprod.hoomano.com.key has insecure file permissions (should be 0600) [2019-01-30T09:47:02,650][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/root-ca.pem has insecure file permissions (should be 0600) [2019-01-30T09:47:02,650][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/root-ca.key has insecure file permissions (should be 0600) [2019-01-30T09:47:02,650][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/admin.key has insecure file permissions (should be 0600) [2019-01-30T09:47:02,650][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/.elasticsearch.keystore.initial_md5sum has insecure file permissions (should be 0600) [2019-01-30T09:47:02,650][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/preprod.hoomano.com.pem has insecure file permissions (should be 0600) [2019-01-30T09:47:02,651][WARN ][c.f.s.SearchGuardPlugin ] [l1_pweP] File /etc/elasticsearch/admin.pem has insecure file permissions (should be 0600) [2019-01-30T09:47:02,729][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [aggs-matrix-stats] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [analysis-common] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [ingest-common] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [lang-expression] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [lang-mustache] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [lang-painless] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [mapper-extras] [2019-01-30T09:47:02,730][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [parent-join] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [percolator] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [rank-eval] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [reindex] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [repository-url] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [transport-netty4] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [tribe] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-ccr] [2019-01-30T09:47:02,731][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-core] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-deprecation] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-graph] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-logstash] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-ml] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-monitoring] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-rollup] [2019-01-30T09:47:02,732][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-security] [2019-01-30T09:47:02,733][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-sql] [2019-01-30T09:47:02,733][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-upgrade] [2019-01-30T09:47:02,733][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded module [x-pack-watcher] [2019-01-30T09:47:02,734][INFO ][o.e.p.PluginsService ] [l1_pweP] loaded plugin [search-guard-6] [2019-01-30T09:47:11,660][INFO ][c.f.s.a.i.AuditLogImpl ] [l1_pweP] Configured categories on rest layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2019-01-30T09:47:11,661][INFO ][c.f.s.a.i.AuditLogImpl ] [l1_pweP] Configured categories on transport layer to ignore: [AUTHENTICATED, GRANTED_PRIVILEGES] [2019-01-30T09:47:11,661][INFO ][c.f.s.a.i.AuditLogImpl ] [l1_pweP] Configured Users to ignore: [kibanaserver] [2019-01-30T09:47:11,661][INFO ][c.f.s.a.i.AuditLogImpl ] [l1_pweP] Configured Users to ignore for read compliance events: [kibanaserver] [2019-01-30T09:47:11,662][INFO ][c.f.s.a.i.AuditLogImpl ] [l1_pweP] Configured Users to ignore for write compliance events: [kibanaserver] [2019-01-30T09:47:11,725][INFO ][c.f.s.a.i.AuditLogImpl ] [l1_pweP] Message routing enabled: true [2019-01-30T09:47:11,735][WARN ][c.f.s.c.ComplianceConfig ] [l1_pweP] If you plan to use field masking pls configure searchguard.compliance.salt to be a random string of 16 chars length identical on all nodes [2019-01-30T09:47:11,735][INFO ][c.f.s.c.ComplianceConfig ] [l1_pweP] PII configuration [auditLogPattern=org.joda.time.format.DateTimeFormatter@5daad59a, auditLogIndex=null]: {} [2019-01-30T09:47:12,402][DEBUG][o.e.a.ActionModule ] [l1_pweP] Using REST wrapper from plugin com.floragunn.searchguard.SearchGuardPlugin [2019-01-30T09:47:21,187][INFO ][o.e.d.DiscoveryModule ] [l1_pweP] using discovery type [zen] and host providers [settings] [2019-01-30T09:47:22,526][INFO ][o.e.n.Node ] [l1_pweP] initialized [2019-01-30T09:47:22,526][INFO ][o.e.n.Node ] [l1_pweP] starting ... [2019-01-30T09:47:23,225][INFO ][o.e.t.TransportService ] [l1_pweP] publish_address {37.187.39.91:9300}, bound_addresses {[::]:9300} [2019-01-30T09:47:23,453][INFO ][o.e.b.BootstrapChecks ] [l1_pweP] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-01-30T09:47:23,472][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [l1_pweP] Check if searchguard index exists ... [2019-01-30T09:47:23,492][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [l1_pweP] no known master node, scheduling a retry [2019-01-30T09:47:26,593][INFO ][o.e.c.s.MasterService ] [l1_pweP] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {l1_pweP}{l1_pwePhQB2xYnGKq64Ksg}{6zOxpqZCSjy_rX_ReeqCMw}{37.187.39.91}{37.187.39.91:9300}{xpack.installed=true} [2019-01-30T09:47:26,600][INFO ][o.e.c.s.ClusterApplierService] [l1_pweP] new_master {l1_pweP}{l1_pwePhQB2xYnGKq64Ksg}{6zOxpqZCSjy_rX_ReeqCMw}{37.187.39.91}{37.187.39.91:9300}{xpack.installed=true}, reason: apply cluster state (from master [master {l1_pweP}{l1_pwePhQB2xYnGKq64Ksg}{6zOxpqZCSjy_rX_ReeqCMw}{37.187.39.91}{37.187.39.91:9300}{xpack.installed=true} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) [2019-01-30T09:47:26,689][INFO ][c.f.s.h.SearchGuardNonSslHttpServerTransport] [l1_pweP] publish_address {37.187.39.91:9200}, bound_addresses {[::]:9200} [2019-01-30T09:47:26,690][INFO ][o.e.n.Node ] [l1_pweP] started [2019-01-30T09:47:26,690][INFO ][c.f.s.SearchGuardPlugin ] [l1_pweP] 4 Search Guard modules loaded so far: [Module [type=DLSFLS, implementing class=com.floragunn.searchguard.configuration.SearchGuardFlsDlsIndexSearcherWrapper], Module [type=AUDITLOG, implementing class=com.floragunn.searchguard.auditlog.impl.AuditLogImpl], Module [type=MULTITENANCY, implementing class=com.floragunn.searchguard.configuration.PrivilegesInterceptorImpl], Module [type=REST_MANAGEMENT_API, implementing class=com.floragunn.searchguard.dlic.rest.api.SearchGuardRestApiActions]] [2019-01-30T09:47:29,002][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:29,760][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:34,079][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:34,801][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:39,092][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:39,812][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:44,106][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:44,824][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:49,117][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:49,836][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:54,131][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:54,849][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:56,937][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:56,981][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:57,205][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:59,144][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:47:59,872][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:01,776][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:04,157][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:04,884][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:06,498][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:09,004][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:09,171][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:09,898][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:11,686][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:14,184][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:14,367][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:14,912][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:16,875][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:18,844][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:19,195][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:19,380][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:19,925][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:21,819][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:21,820][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:21,887][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:23,504][DEBUG][o.e.a.a.i.e.i.TransportIndicesExistsAction] [l1_pweP] timed out while retrying [indices:admin/exists] after failure (timeout [1m]) org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:204) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.admin.indices.exists.indices.TransportIndicesExistsAction.checkBlock(TransportIndicesExistsAction.java:63) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.admin.indices.exists.indices.TransportIndicesExistsAction.checkBlock(TransportIndicesExistsAction.java:40) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:157) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onNewClusterState(TransportMasterNodeAction.java:234) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:303) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:191) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:492) ~[elasticsearch-6.5.1.jar:6.5.1] at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3527) ~[?:1.8.0_191] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) ~[?:1.8.0_191] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_191] at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:489) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:472) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.5.1.jar:6.5.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] [2019-01-30T09:48:23,514][ERROR][c.f.s.c.IndexBaseConfigurationRepository] [l1_pweP] Failure while checking MasterNotDiscoveredException[ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]]; nested: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]; index searchguard org.elasticsearch.discovery.MasterNotDiscoveredException: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:246) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:559) [elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) [elasticsearch-6.5.1.jar:6.5.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191] Caused by: org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized]; at org.elasticsearch.cluster.block.ClusterBlocks.indicesBlockedException(ClusterBlocks.java:204) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.admin.indices.exists.indices.TransportIndicesExistsAction.checkBlock(TransportIndicesExistsAction.java:63) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.admin.indices.exists.indices.TransportIndicesExistsAction.checkBlock(TransportIndicesExistsAction.java:40) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.doStart(TransportMasterNodeAction.java:157) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onNewClusterState(TransportMasterNodeAction.java:234) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onNewClusterState(ClusterStateObserver.java:303) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.clusterChanged(ClusterStateObserver.java:191) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateListeners$7(ClusterApplierService.java:492) ~[elasticsearch-6.5.1.jar:6.5.1] at java.util.concurrent.ConcurrentHashMap$KeySpliterator.forEachRemaining(ConcurrentHashMap.java:3527) ~[?:1.8.0_191] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:743) ~[?:1.8.0_191] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_191] at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateListeners(ClusterApplierService.java:489) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:472) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:416) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:160) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:624) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) ~[elasticsearch-6.5.1.jar:6.5.1] at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) ~[elasticsearch-6.5.1.jar:6.5.1] ... 3 more [2019-01-30T09:48:24,204][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:24,210][INFO ][o.e.l.LicenseService ] [l1_pweP] license [163e26bf-5cd0-498b-9305-66674d138a06] mode [basic] - valid [2019-01-30T09:48:24,211][INFO ][o.e.g.GatewayService ] [l1_pweP] recovered [255] indices into cluster_state [2019-01-30T09:48:24,211][WARN ][o.e.c.s.MasterService ] [l1_pweP] cluster state update task [local-gateway-elected-state] took [56s] above the warn threshold of 30s [2019-01-30T09:48:24,220][WARN ][o.e.c.s.ClusterApplierService] [l1_pweP] cluster state applier task [apply cluster state (from master [master {l1_pweP}{l1_pwePhQB2xYnGKq64Ksg}{6zOxpqZCSjy_rX_ReeqCMw}{37.187.39.91}{37.187.39.91:9300}{xpack.installed=true} committed version [2] source [local-gateway-elected-state]])] took [55.7s] above the warn threshold of 30s [2019-01-30T09:48:24,394][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:24,937][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:26,902][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:27,213][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:29,214][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:29,408][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:29,949][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:31,957][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:34,229][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:34,464][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:34,965][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:36,972][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:39,242][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:39,483][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:40,017][ERROR][c.f.s.a.BackendRegistry ] [l1_pweP] Not yet initialized (you may need to run sgadmin) [2019-01-30T09:48:40,192][INFO ][c.f.s.c.ComplianceConfig ] [l1_pweP] Compliance features are enabled [2019-01-30T09:48:40,336][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [l1_pweP] Search Guard License Info: SearchGuardLicense [uid=00000000-0000-0000-0000-000000000000, type=TRIAL, features=[COMPLIANCE], issueDate=2019-01-29, expiryDate=2019-03-30, issuedTo=The world, issuer=floragunn GmbH, startDate=2019-01-29, majorVersion=6, clusterName=*, allowedNodeCount=2147483647, msgs=[], expiresInDays=59, isExpired=false, valid=true, action=, prodUsage=Yes, one cluster with all commercial features and unlimited nodes per cluster., clusterService=org.elasticsearch.cluster.service.ClusterService@5ca6cc4e, getMsgs()=[], getExpiresInDays()=59, isExpired()=false, isValid()=true, getAction()=, getProdUsage()=Yes, one cluster with all commercial features and unlimited nodes per cluster.] [2019-01-30T09:48:40,337][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [l1_pweP] Search Guard License Type: TRIAL, valid [2019-01-30T09:48:40,337][INFO ][c.f.s.c.IndexBaseConfigurationRepository] [l1_pweP] Node 'l1_pweP' initialized [2019-01-30T09:56:57,127][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][562] overhead, spent [2.3s] collecting in the last [3.1s] [2019-01-30T09:57:04,488][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][568] overhead, spent [1.5s] collecting in the last [2.3s] [2019-01-30T09:57:08,716][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][571] overhead, spent [1.4s] collecting in the last [2.2s] [2019-01-30T09:57:13,602][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][575] overhead, spent [1.5s] collecting in the last [1.8s] [2019-01-30T09:57:24,086][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][584] overhead, spent [1.5s] collecting in the last [2.4s] [2019-01-30T09:57:28,636][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][588] overhead, spent [1.4s] collecting in the last [1.5s] [2019-01-30T09:57:30,838][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][589] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:57:32,447][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][590] overhead, spent [1.5s] collecting in the last [1.6s] [2019-01-30T09:57:34,612][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][591] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:57:36,305][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][592] overhead, spent [1.5s] collecting in the last [1.6s] [2019-01-30T09:57:38,925][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][594] overhead, spent [1.5s] collecting in the last [1.6s] [2019-01-30T09:57:40,566][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][595] overhead, spent [1.5s] collecting in the last [1.6s] [2019-01-30T09:57:42,998][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][596] overhead, spent [2.1s] collecting in the last [2.4s] [2019-01-30T09:57:44,571][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][597] overhead, spent [1.5s] collecting in the last [1.5s] [2019-01-30T09:57:46,720][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][598] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:57:48,169][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][599] overhead, spent [1.3s] collecting in the last [1.4s] [2019-01-30T09:57:50,302][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][600] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:57:51,990][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][601] overhead, spent [1.6s] collecting in the last [1.6s] [2019-01-30T09:57:54,271][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][602] overhead, spent [2.1s] collecting in the last [2.2s] [2019-01-30T09:57:55,841][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][603] overhead, spent [1.4s] collecting in the last [1.5s] [2019-01-30T09:57:58,055][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][604] overhead, spent [2s] collecting in the last [2.2s] [2019-01-30T09:57:58,709][INFO ][o.e.c.r.a.AllocationService] [l1_pweP] Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[server_monitoring_requests_2018-07][4]] ...]). [2019-01-30T09:58:00,792][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][605] overhead, spent [1.7s] collecting in the last [2.7s] [2019-01-30T09:58:02,939][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][606] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:58:04,417][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][607] overhead, spent [1.4s] collecting in the last [1.4s] [2019-01-30T09:58:06,452][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][608] overhead, spent [1.9s] collecting in the last [2s] [2019-01-30T09:58:07,985][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][609] overhead, spent [1.4s] collecting in the last [1.5s] [2019-01-30T09:58:10,270][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][610] overhead, spent [2.1s] collecting in the last [2.2s] [2019-01-30T09:58:11,701][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][611] overhead, spent [1.3s] collecting in the last [1.4s] [2019-01-30T09:58:13,855][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][612] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:58:15,159][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][613] overhead, spent [1.2s] collecting in the last [1.3s] [2019-01-30T09:58:16,689][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][614] overhead, spent [1.4s] collecting in the last [1.5s] [2019-01-30T09:58:18,864][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][615] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:58:20,260][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][616] overhead, spent [1.3s] collecting in the last [1.3s] [2019-01-30T09:58:22,441][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][617] overhead, spent [2s] collecting in the last [2.1s] [2019-01-30T09:58:24,119][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][618] overhead, spent [1.5s] collecting in the last [1.6s] [2019-01-30T09:58:26,336][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][619] overhead, spent [2s] collecting in the last [2.2s] [2019-01-30T09:58:28,027][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][620] overhead, spent [1.6s] collecting in the last [1.6s] [2019-01-30T09:58:30,336][WARN ][o.e.m.j.JvmGcMonitorService] [l1_pweP] [gc][621] overhead, spent [2.1s] collecting in the last [2.3s]