Kibana server is not ready yet

3,854 views
Skip to first unread message

Johan Vermeulen

unread,
Oct 9, 2020, 3:22:42 AM10/9/20
to Wazuh mailing list
I there,

I followed installation instructions on Centos 7 but I get error Kibana server is not ready yet when running http://192.168.1.12:5601/app/home#/

Anything I might have missed.

Regards
Johan 

Selu López

unread,
Oct 9, 2020, 3:36:16 AM10/9/20
to Wazuh mailing list
Hi Johan Vermulen!

This error could be due to different reasons:
  • Your service or Kibana configuration has some error that causes it to constantly reboot
  • Your elasticsearch service is not up or has some error.
Let's look at some logs to find out the origin of the problem:

In order to check the Kibana logs, please run these commands:

systemctl status kibana -l

and

journalctl -u kibana

And on the other hand, please check out elasticsearch:

systemctl status elasticsearch -l

and please check Elasticsearch logs just in case there is something wrong with it

cat /var/log/elasticsearch/elasticsearch.log

Please, paste here the results of the commands.

Regards,
Selu.

Johan Vermeulen

unread,
Oct 9, 2020, 3:51:27 AM10/9/20
to Wazuh mailing list
KIBANA

kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-10-09 09:16:40 SAST; 26min ago
 Main PID: 553 (node)
   CGroup: /system.slice/kibana.service
           └─553 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli

Oct 09 09:17:40 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:40Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to revive connection: http://192.168.1.12:9200/"}
Oct 09 09:17:40 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:40Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living connections"}
Oct 09 09:17:43 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:43Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to revive connection: http://192.168.1.12:9200/"}
Oct 09 09:17:43 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:43Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living connections"}
Oct 09 09:17:45 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:45Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to revive connection: http://192.168.1.12:9200/"}
Oct 09 09:17:45 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:45Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living connections"}
Oct 09 09:17:48 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:48Z","tags":["info","savedobjects-service"],"pid":553,"message":"Starting saved objects migrations"}
Oct 09 09:17:53 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:53Z","tags":["warning","plugins","monitoring","monitoring"],"pid":553,"message":"X-Pack Monitoring Cluster Alerts will not be available: X-Pack plugin is not installed on the Elasticsearch cluster."}
Oct 09 09:18:18 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:18:18Z","tags":["warning","savedobjects-service"],"pid":553,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}
Oct 09 09:22:06 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:22:06Z","tags":["warning","savedobjects-service"],"pid":553,"message":"Unable to connect to Elasticsearch. Error: [master_not_discovered_exception] null"}
[root@localhost wazuh]# 


KIBANA LOG 

Oct 09 09:17:33 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:33Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living co
Oct 09 09:17:35 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:35Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to re
Oct 09 09:17:35 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:35Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living co
Oct 09 09:17:38 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:38Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to re
Oct 09 09:17:38 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:38Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living co
Oct 09 09:17:40 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:40Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to re
Oct 09 09:17:40 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:40Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living co
Oct 09 09:17:43 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:43Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to re
Oct 09 09:17:43 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:43Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living co
Oct 09 09:17:45 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:45Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"Unable to re
Oct 09 09:17:45 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:45Z","tags":["warning","elasticsearch","data"],"pid":553,"message":"No living co
Oct 09 09:17:48 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:48Z","tags":["info","savedobjects-service"],"pid":553,"message":"Starting saved 
Oct 09 09:17:53 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:17:53Z","tags":["warning","plugins","monitoring","monitoring"],"pid":553,"message":
Oct 09 09:18:18 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:18:18Z","tags":["warning","savedobjects-service"],"pid":553,"message":"Unable to co
Oct 09 09:22:06 localhost.localdomain kibana[553]: {"type":"log","@timestamp":"2020-10-09T07:22:06Z","tags":["warning","savedobjects-service"],"pid":553,"message":"Unable to co
~

ELASTICSEARCH

elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-10-09 09:17:46 SAST; 30min ago
 Main PID: 963 (java)
   CGroup: /system.slice/elasticsearch.service
           ├─ 963 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms1g -Xmx1g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-14589139913800314529 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=536870912 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet
           └─3722 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Oct 09 09:16:47 localhost.localdomain systemd[1]: Starting Elasticsearch...
Oct 09 09:17:46 localhost.localdomain systemd[1]: Started Elasticsearch.


[2020-10-09T00:47:34,139][WARN ][r.suppressed             ] [node-1] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:47:34,141][WARN ][r.suppressed             ] [node-1] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:47:36,605][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:47:46,608][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:47:56,613][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:48:03,866][WARN ][r.suppressed             ] [node-1] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:48:06,615][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:48:06,648][WARN ][r.suppressed             ] [node-1] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:48:06,650][WARN ][r.suppressed             ] [node-1] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:48:16,616][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:48:26,621][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:48:36,623][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:48:39,152][WARN ][r.suppressed             ] [node-1] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:48:39,152][WARN ][r.suppressed             ] [node-1] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:48:46,625][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:48:56,626][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:49:06,628][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:49:11,657][WARN ][r.suppressed             ] [node-1] path: /.kibana, params: {index=.kibana}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:49:11,657][WARN ][r.suppressed             ] [node-1] path: /.kibana_task_manager, params: {index=.kibana_task_manager}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:49:16,630][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:49:19,910][WARN ][r.suppressed             ] [node-1] path: /_license, params: {human=false}
org.elasticsearch.discovery.MasterNotDiscoveredException: null
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:605) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:678) [elasticsearch-7.9.2.jar:7.9.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2020-10-09T00:49:26,632][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2020-10-09T00:49:38,498][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, [::1]:9300, [::1]:9301, [::1]:9302, [::1]:9303, [::1]:9304, [::1]:9305] from hosts providers and [{node-1}{QGhS5FK9RhmWk31HmlS56A}{D1pxOL8ZTIO1m-95uaZp9w}{192.168.1.12}{192.168.1.12:9300}{dilmrt}{ml.machine_memory=3958321152, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[root@localhost wazuh]# 

Selu López

unread,
Oct 9, 2020, 4:16:23 AM10/9/20
to Wazuh mailing list
It seems that Kibana is not able to connect to Elasticsearch.

Just to make sure, the IP that you set in the field elasticsearch.hosts of the file /etc/kibana/kibana.yml (I guess it is http://192.168.1.12:9200/) it's the same that you used in network.host inside elasticsearch.yml, right?

Let's make sure Elasticsearch is listening correctly. You can make a simple check by making the following request:
curl http://192.168.1.12:9200/

Let me know the results.

Johan Vermeulen

unread,
Oct 9, 2020, 4:28:57 AM10/9/20
to Wazuh mailing list
Hi Selo,

I think I have correct IP address yes.

Here is output of command.

[root@localhost wazuh]# curl http://192.168.1.12:9200/
{
  "name" : "node-1",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "_na_",
  "version" : {
    "number" : "7.9.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",
    "build_date" : "2020-09-23T00:45:33.626720Z",
    "build_snapshot" : false,
    "lucene_version" : "8.6.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
[root@localhost wazuh]# 


Selu López

unread,
Oct 9, 2020, 5:21:14 AM10/9/20
to Wazuh mailing list
Hi again Johan,

Elasticsearch seems to be running correctly, but there has been some kind of problem with bootstrap, as shown below:
[2020-10-09T00:49:38,498][WARN ][o.e.c.c.ClusterFormationFailureHelper][node-1] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [node-1, node-2] to bootstrap a cluster: have discovered

This can be for many reasons. For example, if the initial_master_nodes list is incorrect or if you have an Elasticsearch cluster and one of the nodes has gone offline.

However, since you say that you have followed the installation guide for Centos 7, I understand that you are not trying to deploy a cluster and that you only have one node yet. Could you paste the content of your /etc/elasticsearch/elasticsearch.yml here so I can take a look?

You can also visit this page if you are interested in learning more about Bootstraping in elasticsearch. This also explains the error you posted before

Regards.

Johan Vermeulen

unread,
Oct 13, 2020, 10:17:19 AM10/13/20
to Wazuh mailing list
Hi there,

I've attached content of file below.
Thanks.

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.1.12
#
# Set a custom port for HTTP:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true


Selu López

unread,
Oct 14, 2020, 5:02:04 AM10/14/20
to Wazuh mailing list
Hi Johan!

From what I see in your configuration, it looks like you are trying to deploy an Elasticsearch cluster. I think then the problem could be related to this line:

#discovery.seed_hosts: ["host1", "host2"]

Below you can see an example of the configuration that you should edit in each of the cluster nodes, inside /etc/elasticsearch/elasticsearch.yml.

cluster.name: elastic-cluster 

node.name: <node_name> 

network.host: <elasticsearch_ip> 

discovery.seed_hosts:
 - <elasticsearch_ip_node1>
 - <elasticsearch_ip_node2>

cluster.initial_master_nodes:
 - <master_node_1>
 - <master_node_2>

To apply changes, restart each node:
systemctl restart elasticsearch.service

Please also make sure that both nodes can see each other and are running correctly.

There are a few more steps you will need to follow to properly configure the cluster. Also, it is recommended to use at least 3 nodes to avoid future problems. I would suggest you to follow this documentation page where each step necessary to deploy a cluster is detailed:
https://documentation.wazuh.com/3.13/installation-guide/installing-elastic-stack/configure-elasticsearch-cluster.html

Best regards,
Selu.
...
Reply all
Reply to author
Forward
0 new messages