Unable to start Kibana - javascript heap out of memory

[Martin Glazer]

Hi,

I had been running wazah 3.4 for a while and then decided to upgrade a few months ago. Went through the process but had some issues with elasticsearch and shards, so I gave up and left it until recently.

I upgraded to wazah 3.11, elasticsearch/logstash/filebeat/kibana 6.8.7-1 and had everything operational (after deleting and then reindexing - think that is what I did). I then decided to upgrade to elasticsearch/filebeat/kibana 7.6.1-1. All went smoothly except now when I try and start kibana, I am getting javascript heap out of memory errors and the startup of kibana fails.

This is running on a VM with 10Gb of RAM allocated and only a single instance of wazuh. OS is CentOS 7.7. Wazuh itself is running fine, just Kibana is the issue.

I have tried to install the wazuhapp kibana plugin with

sudo -u kibana NODE_OPTIONS="--max-old-space-size=4096" /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.11.4_7.6.1.zip

and also --max-old-space-size=3072 but this does not appear to make a difference.

The log files show

Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: <--- Last few GCs --->
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: [12543:0x2e340d0]   381255 ms: Mark-sweep 1330.7 (1439.8) -> 1316.8 (1440.8) MB, 1765.3 / 0.1 ms  (average mu = 0.228, current mu = 0.163)
allocation failure scavenge might not succeed
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: [12543:0x2e340d0]   383239 ms: Mark-sweep 1331.5 (1440.8) -> 1317.3 (1442.3) MB, 1642.8 / 0.1 ms  (average mu = 0.202, current mu = 0.172)
allocation failure scavenge might not succeed
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: <--- JS stacktrace --->
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: ==== JS stack trace =========================================
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 0: ExitFrame [pc: 0x3f972badbe1d]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: Security context: 0x12543d21e6e9 <JSObject>
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 1: /* anonymous */(aka /* anonymous */) [0x28650a64d661] [/usr/share/kibana/node_modules/terser/dist/bundle.min.js:~1] [pc=0x3f972c96c263](
this=0x1ef8237026f1 <undefined>,e=0x3e51856643a1 <AST_String map = 0x187c61f4abe1>,t=0x0088520823b1 <Object map = 0x7f94f57d931>)
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 2: r(aka r) [0xace824ae8c1] [/usr/share/kibana/node_modules/terser/dist/bundle.min....
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 1: 0x8fa090 node::Abort() [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 2: 0x8fa0dc  [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 3: 0xb0052e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 4: 0xb00764 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 5: 0xef4c72  [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 6: 0xef4d78 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 7: 0xf00e52 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 8: 0xf01784 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 9: 0xf043f1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 10: 0xecd874 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 11: 0x116d9fe v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/bin/../node/bin/node]
Apr 01 13:14:52 wuzah.xyz.ca kibana[12543]: 12: 0x3f972badbe1d
Apr 01 13:14:52 wuzah.xyz.ca systemd[1]: kibana.service: main process exited, code=killed, status=6/ABRT
Apr 01 13:14:52 wuzah.xyz.ca systemd[1]: Unit kibana.service entered failed state.
Apr 01 13:14:52 wuzah.xyz.ca systemd[1]: kibana.service failed.

I have also stopped logstash (as I believe it is no longer needed).

Any thoughts on what the issue could be and how to resolve?

Thanks

[Franco Giovanolli]

Hi Martin, you are right to try to solve this error increasing nodejs memory limit.

To the best of my knowledge the max-old-space-size option sets the heap limit in node.

This option can be set as an env variable in

/etc/default/kibana

NODE_OPTIONS="--max-old-space-size=4096"

After that, please restart the Kibana service:

systemctl restart kibana

Can you try this?



Regards,

Franco

[Martin Glazer]

Thanks Franco - that worked.

Is this documented anywhere?

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/58ee0635-9038-4298-86ec-1412c67d9964%40googlegroups.com.

[Franco Giovanolli]

Hi Martin, perfect!

This tip is documented as recommendation in https://github.com/wazuh/wazuh-kibana-app

Regards!

Franco

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.