Kibana Server is not ready yet

[Ayush Agarwal]

Hello,

All of a sudden I keep getting "Kibana Server is not ready yet". I've restarted both elasticsearch and kibana service but it is stuck here from long time. 

In the elasticsearch log I see all shards failed.

[2019-10-09T07:47:32,136][WARN ][r.suppressed             ] [VltDQZW] path: /.kibana/doc/_count, params: {index=.kibana, type=doc}

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:293) ~[elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:133) ~[elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:254) ~[elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.action.search.InitialSearchPhase.onShardFailure(InitialSearchPhase.java:101) ~[elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$1(InitialSearchPhase.java:209) ~[elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.action.search.InitialSearchPhase$1.doRun(InitialSearchPhase.java:188) [elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:759) [elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.6.2.jar:6.6.2]

at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.6.2.jar:6.6.2]

at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:1.8.0_202]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:1.8.0_202]

at java.lang.Thread.run(Unknown Source) [?:1.8.0_202]

Could you please suggest how can I fix this problem?

Thanks!

Ayush Agarwal

[Pablo Rodríguez Martín]

Hi, Ayush.

All shards failing may occur when the machine runs out of resources and Elasticsearch is not able to index alerts anymore, and that will cause indices to go on the read-only mode that can be fixed with: 

curl -X PUT http://localhost:9200/_all/_settings -H 'Content-Type: application/json' -d'{ "index.blocks.read_only_allow_delete" : false } }'

Based on the logs it seems that Elasticsearch is complaining about .kibana index. You may delete that index, but make sure you have a backup of dashboards if you have any:

systemctl stop kibana

curl -XDELETE "http://localhost:9200/.kibana"

systemctl start kibana

However, Could you please provide additional information about your environment? More logs after restarting Elasticsearch node(s) would be a good start point to check what is happening. For that, you can use:

cat /var/log/elasticsearch/elasticsearch.log | grep -i -E "error|warn"

Best regards,

Pablo Rodríguez