Kibana waiting for server issue after update

[gamebeatter]

Hello we recently updated our wazuh, single node all applications on the same VM server running Ubuntu, after the upgrade of kibana we are receiving the waiting for server error, below are the logs I have already attempted to go into the kibana.yml to increase the timeout for Elasticsearch but that is looking like it is not reading it or not working, I have attached both the Elasticsearch and kibana configurations, I have Xpact security turned off because I attempted to use it to authenticate the server for security, but instead went with a NGINX proxy when I discovered that the xpack was not going to function like I thought it would, and went through the process of removing everything from xpack. I also have an increased node count to resolve an issue I had with to many shards the final step would be to make sure the merge I ran has the data in it which I would need  kibana to look at.

Kibana logs:

Aug 04 08:42:03 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:03Z","tags":["info","plugins","watcher"],"pid":115640,"message":"Your basic license does not support watcher. Please upgrade your license."}

Aug 04 08:42:03 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:03Z","tags":["info","plugins","monitoring","monitoring","kibana-monitoring"],"pid":115640,"message":"Starting monitoring stats collection"}

Aug 04 08:42:03 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:03Z","tags":["info","savedobjects-service"],"pid":115640,"message":"Starting saved objects migrations"}

Aug 04 08:42:03 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:03Z","tags":["info","savedobjects-service"],"pid":115640,"message":"Creating index .kibana_1."}

Aug 04 08:42:03 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:03Z","tags":["info","savedobjects-service"],"pid":115640,"message":"Creating index .kibana_task_manager_1."}

Aug 04 08:42:27 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:27Z","tags":["info","savedobjects-service"],"pid":115640,"message":"Pointing alias .kibana to .kibana_1."}

Aug 04 08:42:33 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:33Z","tags":["warning","savedobjects-service"],"pid":115640,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}

Aug 04 08:42:42 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:42Z","tags":["warning","savedobjects-service"],"pid":115640,"message":"Unable to connect to Elasticsearch. Error: [resource_already_exists_exception] index [.kibana_task_manager_1/ioeWtxMtS-6ZAjUA3rmz4A] already exists, wit>

Aug 04 08:42:42 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:42Z","tags":["warning","savedobjects-service"],"pid":115640,"message":"Another Kibana instance appears to be migrating the index. Waiting for that migration to complete. If no other Kibana instance is attempting migrations,>

Aug 04 08:42:43 Wazhu kibana[115640]: {"type":"log","@timestamp":"2020-08-04T13:42:43Z","tags":["info","savedobjects-service"],"pid":115640,"message":"Finished in 39635ms."}

[Yana Zaeva]

Hi,
Kibana uses two indices for its 'saved objects': .kibana_n and .kibana_task_manager_n (where n is any positive number). When multiple Kibana's are started at the same time they will all try to create this index, but some will fail because other instances have already created that index.

Apparently, your problem may be related to the mentioned before, as Kibana is trying to create an index (and might have succeeded), but because of the timeout, Kibana does not know it. Due to this, the node might have fallen back into a polling loop, waiting for another instance to complete it, but there was no other instance busy with the migration.

You should be able to solve this error following the next steps:

• Export the dashboard in the UI: management -> saved objects -> export everything. This will get searches, visualizations and dashboards.
• Delete all the Kibana indices, allowing Kibana to attempt the migration again:

curl -XDELETE http://localhost:9200/.kibana*

• Restart Kibana.
• Import all the saved object via: management -> saved objects -> import.

Hope I have been helpful, and let me know if you continue having the issue!

[gamebeatter]

same error after running the curl -XDELETE http://localhost:9200/.kibana*

[Yana Zaeva]

Hi,

First of all, sorry for the late reply. Concerning your issue, when you upgraded Kibana have you upgraded Elasticsearch as well? Your problem could be due to a version mismatch between Elasticsearch and Kibana. I will leave you a link where you can see whether they are 
compatible: https://www.elastic.co/es/support/matrix#matrix_compatibility

In case they are compatible, and no other Kibana instance is attempting migrations you can:

- First, export the dashboard in the UI: management -> saved objects -> export everything. This will get searches, visualizations and dashboards.

- Once you have everything saved, delete the .kibana_task_manager_1 and restart Kibana:

curl -XDELETE http://localhost:9200/.kibana_task_manager_1 

and then restart Kibana.

Lastly, if the above did not help, please, send me the output of the command: ps -aux | grep -i kibana in order to see how many processes we have running.  

Let me know if it does not work. Hope I have been helpful!

[gamebeatter]

that did not work either, here is the results for the requested command.

kibana    118121  0.4  4.8 1491004 292160 ?      Ssl  Aug04  43:14 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli

root      181391  0.0  0.0   9032   668 pts/0    S+   15:59   0:00 grep --color=auto -i kibana

[Yana Zaeva]

Hi, 

Checking the output you send, it seems that Kibana process has been working since August 4th. I suggest restarting it because maybe some of the changes you have performed are not being applied. If the issue remains, please let me know which versions of Elasticsearch and Kibana you are using and send me both, Elasticsearch and Kibana logs right after you delete all the indices and the task manager:

curl -XDELETE http://localhost:9200/.kibana_task_manager_1  

 curl -XDELETE http://localhost:9200/.kibana* (where n is a positive number)

You can find Elasticsearch logs at the file: /var/log/elasticsearch/elasticsearch.log while in order to get Kibana logs you have to configure it first:

- Go to /etc/kibana/kibana.yml and using the field logging.dest specify a file to store the logs. Check also the Wazuh App log file by: cat /usr/share/kibana/optimize/wazuh-logs/wazuhapp.log | grep -i -E "error|warn"

I will leave you a link where you can get more information about troubleshooting in Kibana: https://documentation.wazuh.com/3.13/user-manual/kibana-app/troubleshooting.html

Waiting for your reply.

Best regards,

Yana Zaeva.