kibana service shutdown unexpected

311 views
Skip to first unread message

Odie

unread,
Sep 10, 2018, 7:51:31 AM9/10/18
to Wazuh mailing list
Hi Everyone!

need your help how to fix kibana error " kibana service stop itself after a while" tried to restart but it stop itself

here's the logs:

Sep 10 19:37:57 mon02 kernel: Movable zone start for each node
Sep 10 19:37:57 mon02 kernel: Early memory node ranges
Sep 10 19:37:57 mon02 kernel:  node   0: [mem 0x00001000-0x0009efff]
Sep 10 19:37:57 mon02 kernel:  node   0: [mem 0x00100000-0xbfeeffff]
Sep 10 19:37:57 mon02 kernel:  node   0: [mem 0xbff00000-0xbfffffff]
Sep 10 19:37:57 mon02 kernel:  node   0: [mem 0x100000000-0x23fffffff]
Sep 10 19:37:57 mon02 kernel: Initmem setup node 0 [mem 0x00001000-0x23fffffff]
Sep 10 19:37:57 mon02 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:1 nr_cpu_ids:1 nr_node_ids:1
Sep 10 19:37:57 mon02 kernel: Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes)
Sep 10 19:37:57 mon02 kernel: node 0 initialised, 753402 pages in 21ms
Sep 10 19:37:57 mon02 systemd[1]: Starting Create list of required static device nodes for the current kernel...
Sep 10 19:37:57 mon02 systemd[1]: Started Create list of required static device nodes for the current kernel.
Sep 10 19:37:59 mon02 systemd: Stopped Create list of required static device nodes for the current kernel.
Sep 10 19:37:59 mon02 systemd: Stopping Create list of required static device nodes for the current kernel...

Thank you for your help.

jesus.g...@wazuh.com

unread,
Sep 10, 2018, 8:48:14 AM9/10/18
to Wazuh mailing list
Hi Odie,

It may be caused by OOM (out of memory), please paste the output of the next command:

cat /var/log/messages | grep node

Also try to start Kibana in foreground mode to see if there are errors being shown:

systemctl stop kibana
/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml --verbose

Once you are done, use CTRL + C to stop Kibana.

Regards,
Jesús

Odie corañes

unread,
Sep 10, 2018, 11:15:17 PM9/10/18
to jesus.g...@wazuh.com, Wazuh mailing list
Hi Jesus,

Thank you for the quick response..

I have fine tune elasticsearch and set JVM heap size to 4gb before . will try to increase my RAM from 8 GB to 16 GB


Below is the logs,  
 
It may be caused by OOM (out of memory), please paste the output of the next command:

cat /var/log/messages | grep node

cat /var/log/messages | grep node
Sep 11 10:51:56 mon02 kernel: Movable zone start for each node
Sep 11 10:51:56 mon02 kernel: Early memory node ranges
Sep 11 10:51:56 mon02 kernel:  node   0: [mem 0x00001000-0x0009efff]
Sep 11 10:51:56 mon02 kernel:  node   0: [mem 0x00100000-0xbfeeffff]
Sep 11 10:51:56 mon02 kernel:  node   0: [mem 0xbff00000-0xbfffffff]
Sep 11 10:51:56 mon02 kernel:  node   0: [mem 0x100000000-0x83fffffff]
Sep 11 10:51:56 mon02 kernel: Initmem setup node 0 [mem 0x00001000-0x83fffffff]
Sep 11 10:51:56 mon02 kernel: setup_percpu: NR_CPUS:5120 nr_cpumask_bits:2 nr_cpu_ids:2 nr_node_ids:1
Sep 11 10:51:56 mon02 kernel: Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes)
Sep 11 10:51:56 mon02 kernel: node 0 initialised, 6946496 pages in 185ms
Sep 11 10:51:56 mon02 systemd[1]: Starting Create list of required static device nodes for the current kernel...
Sep 11 10:51:56 mon02 systemd[1]: Started Create list of required static device nodes for the current kernel.
Sep 11 10:51:58 mon02 systemd: Stopped Create list of required static device nodes for the current kernel.
Sep 11 10:51:58 mon02 systemd: Stopping Create list of required static device nodes for the current kernel...



Also try to start Kibana in foreground mode to see if there are errors being shown:

systemctl stop kibana
/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml --verbose


/usr/share/kibana/bin/kibana -c /etc/kibana/kibana.yml --verbose
  log   [02:56:21.252] [debug][plugin] Found plugin at /usr/share/kibana/plugins/wazuh
  log   [02:56:21.266] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/console
  log   [02:56:21.268] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/elasticsearch
  log   [02:56:21.269] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/input_control_vis
  log   [02:56:21.270] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_doc_views
  log   [02:56:21.284] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kbn_vislib_vis_types
  log   [02:56:21.291] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/kibana
  log   [02:56:21.292] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/markdown_vis
  log   [02:56:21.292] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metric_vis
  log   [02:56:21.293] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/metrics
  log   [02:56:21.294] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/region_map
  log   [02:56:21.295] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/spy_modes
  log   [02:56:21.297] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/state_session_storage_redirect
  log   [02:56:21.298] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/status_page
  log   [02:56:21.299] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/table_vis
  log   [02:56:21.302] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tagcloud
  log   [02:56:21.303] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/tile_map
  log   [02:56:21.305] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/timelion
  log   [02:56:21.306] [debug][plugin] Found plugin at /usr/share/kibana/src/core_plugins/vega
  log   [02:56:22.360] [debug][optimize] All bundles are cached and ready to go!
  log   [02:56:22.380] [debug][plugins] Initializing plugin kibana@kibana
  log   [02:56:22.483] [info][status][plugin:kib...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:22.486] [debug][plugins] Initializing plugin elasticsearch@kibana
  log   [02:56:22.542] [info][status][plugin:elasti...@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [02:56:22.545] [debug][plugins] Initializing plugin wa...@3.2.1
  log   [02:56:22.811] [info][status][plugin:wa...@3.2.1] Status changed from uninitialized to green - Ready
  log   [02:56:22.813] [debug][plugins] Initializing plugin input_control_vis@kibana
  log   [02:56:22.821] [debug][plugins] Initializing plugin kbn_doc_views@kibana
  log   [02:56:22.824] [debug][plugins] Initializing plugin kbn_vislib_vis_types@kibana
  log   [02:56:22.826] [debug][plugins] Initializing plugin markdown_vis@kibana
  log   [02:56:22.832] [debug][plugins] Initializing plugin metric_vis@kibana
  log   [02:56:22.835] [debug][plugins] Initializing plugin region_map@kibana
  log   [02:56:22.837] [debug][plugins] Initializing plugin spy_modes@kibana
  log   [02:56:22.847] [debug][plugins] Initializing plugin state_session_storage_redirect@kibana
  log   [02:56:22.851] [debug][plugins] Initializing plugin status_page@kibana
  log   [02:56:22.853] [debug][plugins] Initializing plugin table_vis@kibana
  log   [02:56:22.863] [debug][plugins] Initializing plugin tagcloud@kibana
  log   [02:56:22.866] [debug][plugins] Initializing plugin tile_map@kibana
  log   [02:56:22.869] [debug][plugins] Initializing plugin timelion@kibana
  log   [02:56:23.148] [info][status][plugin:time...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.151] [debug][plugins] Initializing plugin console@kibana
  log   [02:56:23.160] [info][status][plugin:con...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.162] [debug][plugins] Initializing plugin metrics@kibana
  log   [02:56:23.167] [info][status][plugin:met...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.172] [debug][plugins] Initializing plugin vega@kibana
  log   [02:56:23.194] [server][uuid][uuid] Resuming persistent Kibana instance UUID: 12b8202b-a18e-4d9d-aa0e-b13ddd46949e
  log   [02:56:23.217] [fatal] Port 5601 is already in use. Another instance of Kibana may be running!




--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e9a3078d-d4a6-44b0-83e6-3846403ec028%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Odelon Corañes
09272454413
===================================================

"There are only 10 types of people in the world: those who understand binary, and those who don't."

jesus.g...@wazuh.com

unread,
Sep 11, 2018, 3:06:40 AM9/11/18
to Wazuh mailing list
Hi Odie,

Look at this log: 

log   [02:56:23.217] [fatal] Port 5601 is already in use. Another instance of Kibana may be running!

Try to check which process is using the port:

lsof -i :5601

If you haven't got lsof installed, install it as follows:

apt-get install lsof

or

yum install lsof


Regards,
Jesús
  log   [02:56:22.483] [info][status][plugin:kibana@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:22.486] [debug][plugins] Initializing plugin elasticsearch@kibana
  log   [02:56:22.542] [info][status][plugin:elasti...@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [02:56:22.545] [debug][plugins] Initializing plugin wa...@3.2.1
  log   [02:56:22.811] [info][status][plugin:wazuh@3.2.1] Status changed from uninitialized to green - Ready
  log   [02:56:22.813] [debug][plugins] Initializing plugin input_control_vis@kibana
  log   [02:56:22.821] [debug][plugins] Initializing plugin kbn_doc_views@kibana
  log   [02:56:22.824] [debug][plugins] Initializing plugin kbn_vislib_vis_types@kibana
  log   [02:56:22.826] [debug][plugins] Initializing plugin markdown_vis@kibana
  log   [02:56:22.832] [debug][plugins] Initializing plugin metric_vis@kibana
  log   [02:56:22.835] [debug][plugins] Initializing plugin region_map@kibana
  log   [02:56:22.837] [debug][plugins] Initializing plugin spy_modes@kibana
  log   [02:56:22.847] [debug][plugins] Initializing plugin state_session_storage_redirect@kibana
  log   [02:56:22.851] [debug][plugins] Initializing plugin status_page@kibana
  log   [02:56:22.853] [debug][plugins] Initializing plugin table_vis@kibana
  log   [02:56:22.863] [debug][plugins] Initializing plugin tagcloud@kibana
  log   [02:56:22.866] [debug][plugins] Initializing plugin tile_map@kibana
  log   [02:56:22.869] [debug][plugins] Initializing plugin timelion@kibana
  log   [02:56:23.148] [info][status][plugin:time...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.151] [debug][plugins] Initializing plugin console@kibana
  log   [02:56:23.160] [info][status][plugin:console@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.162] [debug][plugins] Initializing plugin metrics@kibana
  log   [02:56:23.167] [info][status][plugin:metrics@6.2.2] Status changed from uninitialized to green - Ready
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e9a3078d-d4a6-44b0-83e6-3846403ec028%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Odie corañes

unread,
Sep 11, 2018, 3:46:13 AM9/11/18
to jesus.g...@wazuh.com, Wazuh mailing list
Hi Jesus,

Here's the logs running  lsof -i :5601

[root@mon02 ~]# lsof -i :5601
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
node    14924 kibana   12u  IPv4  38904      0t0  TCP localhost:esmagent (LISTEN)


I think issue resolved by adding more RAM in my box, almost 5 hours passed kibana didn't stop.. will monitor this..

Thank you for usual support.. appreciate your effort guys..


On Tue, Sep 11, 2018 at 3:06 PM <jesus.g...@wazuh.com> wrote:
Hi Odie,

  log   [02:56:22.483] [info][status][plugin:kib...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:22.486] [debug][plugins] Initializing plugin elasticsearch@kibana
  log   [02:56:22.542] [info][status][plugin:elasti...@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [02:56:22.545] [debug][plugins] Initializing plugin wa...@3.2.1
  log   [02:56:22.811] [info][status][plugin:wa...@3.2.1] Status changed from uninitialized to green - Ready
  log   [02:56:22.813] [debug][plugins] Initializing plugin input_control_vis@kibana
  log   [02:56:22.821] [debug][plugins] Initializing plugin kbn_doc_views@kibana
  log   [02:56:22.824] [debug][plugins] Initializing plugin kbn_vislib_vis_types@kibana
  log   [02:56:22.826] [debug][plugins] Initializing plugin markdown_vis@kibana
  log   [02:56:22.832] [debug][plugins] Initializing plugin metric_vis@kibana
  log   [02:56:22.835] [debug][plugins] Initializing plugin region_map@kibana
  log   [02:56:22.837] [debug][plugins] Initializing plugin spy_modes@kibana
  log   [02:56:22.847] [debug][plugins] Initializing plugin state_session_storage_redirect@kibana
  log   [02:56:22.851] [debug][plugins] Initializing plugin status_page@kibana
  log   [02:56:22.853] [debug][plugins] Initializing plugin table_vis@kibana
  log   [02:56:22.863] [debug][plugins] Initializing plugin tagcloud@kibana
  log   [02:56:22.866] [debug][plugins] Initializing plugin tile_map@kibana
  log   [02:56:22.869] [debug][plugins] Initializing plugin timelion@kibana
  log   [02:56:23.148] [info][status][plugin:time...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.151] [debug][plugins] Initializing plugin console@kibana
  log   [02:56:23.160] [info][status][plugin:con...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.162] [debug][plugins] Initializing plugin metrics@kibana
  log   [02:56:23.167] [info][status][plugin:met...@6.2.2] Status changed from uninitialized to green - Ready
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e9a3078d-d4a6-44b0-83e6-3846403ec028%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Odelon Corañes
09272454413
===================================================

"There are only 10 types of people in the world: those who understand binary, and those who don't."

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/80962ce9-8133-4596-b026-20e084e635ac%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

jesus.g...@wazuh.com

unread,
Oct 10, 2018, 6:09:35 AM10/10/18
to Wazuh mailing list
Hi Odie,

Sorry for the late response. The command lsof is telling you that here is another Kibana process running. Have you solved this? 
Let us know if you need help.

Best regards,
Jesús
  log   [02:56:22.483] [info][status][plugin:kibana@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:22.486] [debug][plugins] Initializing plugin elasticsearch@kibana
  log   [02:56:22.542] [info][status][plugin:elasti...@6.2.2] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [02:56:22.545] [debug][plugins] Initializing plugin wa...@3.2.1
  log   [02:56:22.811] [info][status][plugin:wazuh@3.2.1] Status changed from uninitialized to green - Ready
  log   [02:56:22.813] [debug][plugins] Initializing plugin input_control_vis@kibana
  log   [02:56:22.821] [debug][plugins] Initializing plugin kbn_doc_views@kibana
  log   [02:56:22.824] [debug][plugins] Initializing plugin kbn_vislib_vis_types@kibana
  log   [02:56:22.826] [debug][plugins] Initializing plugin markdown_vis@kibana
  log   [02:56:22.832] [debug][plugins] Initializing plugin metric_vis@kibana
  log   [02:56:22.835] [debug][plugins] Initializing plugin region_map@kibana
  log   [02:56:22.837] [debug][plugins] Initializing plugin spy_modes@kibana
  log   [02:56:22.847] [debug][plugins] Initializing plugin state_session_storage_redirect@kibana
  log   [02:56:22.851] [debug][plugins] Initializing plugin status_page@kibana
  log   [02:56:22.853] [debug][plugins] Initializing plugin table_vis@kibana
  log   [02:56:22.863] [debug][plugins] Initializing plugin tagcloud@kibana
  log   [02:56:22.866] [debug][plugins] Initializing plugin tile_map@kibana
  log   [02:56:22.869] [debug][plugins] Initializing plugin timelion@kibana
  log   [02:56:23.148] [info][status][plugin:time...@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.151] [debug][plugins] Initializing plugin console@kibana
  log   [02:56:23.160] [info][status][plugin:console@6.2.2] Status changed from uninitialized to green - Ready
  log   [02:56:23.162] [debug][plugins] Initializing plugin metrics@kibana
  log   [02:56:23.167] [info][status][plugin:metrics@6.2.2] Status changed from uninitialized to green - Ready
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/e9a3078d-d4a6-44b0-83e6-3846403ec028%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


--
Odelon Corañes
09272454413
===================================================

"There are only 10 types of people in the world: those who understand binary, and those who don't."

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/80962ce9-8133-4596-b026-20e084e635ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages