Installation of Wazuh application in Kibana never finish

[Marcio Costa]

Hi !

Whenever I update ELK and it is necessary update the Wazuh application to Kibana, the installation never ends.

After abort (Control+C) the update, and restart the server everything work, but something must be wrong with this procedure.

I am using Centos 7.5 and ELK 6.

#systemctl stop kibana
#rm -rf /usr/share/kibana/optimize/bundles
#export NODE_OPTIONS="--max-old-space-size=8192"

#/usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.5.0_6.4.0.zip
Attempting to transfer from https://packages.wazuh.com/wazuhapp/wazuhapp-3.5.0_6.4.0.zip
Transferring 17568152 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

I will appreciate any help about this issue.

Thank you.

[jesus.g...@wazuh.com]

Hi Marcio Costa,

The optimizing procedure is a problem with Kibana itself and we are continuously struggling with it. From the https://discuss.elastic.co you can read

a few posts talking about it such https://discuss.elastic.co/t/kibana-plugin-stuck-at-optimizing-and-caching-browser-bundles/89819 or https://discuss.elastic.co/t/installing-x-pack-plugin-for-kibana-6-1-0-taking-20min/112210/5

Usually the main reason is the RAM usage, let's check two things about the memory:

1. Try to start a plugin installation and look at the free memory

Please use free -h or top

2. Check your total RAM, usually you need 4GB or more to work properly. If Elasticsearch is running along other components, try to stop them before trying to install the plugin.

Let us know a bit more about your machine (RAM, CPU, ..) .

Regards,

Jesús

[Marcio Costa]

Hi @Jesus!

My virtual machine have 12GB RAM and 2 vCpus.

Please check the steps bellow:

# free -m
              total        used        free      shared  buff/cache   available
Mem:          11852        6073         629           8        5149        5213
Swap:             0           0           0

# systemctl stop kibana elasticsearch logstash

# /usr/share/kibana/bin/kibana-plugin remove wazuh
Removing wazuh...
Plugin removal complete

# free -m
              total        used        free      shared  buff/cache   available
Mem:          11852         478        6650           8        4723       10951
Swap:             0           0           0

# export NODE_OPTIONS="--max-old-space-size=8192"
# rm -rf /usr/share/kibana/optimize/bundles
# /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.5.0_6.4.0.zip

Attempting to transfer from https://packages.wazuh.com/wazuhapp/wazuhapp-3.5.0_6.4.0.zip
Transferring 17568152 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

When I run the installation I see (using top) the %cpu usage increase by nodejs, but the memory usage no more than 10%.  After some minutes, nodejs 'finish the service', the load average of server down, but the process still running.

Thank you.

[jesus.g...@wazuh.com]

Ok Marcio,

Let me check by myself a bit more your environment. Also I want to research more in deep about optimizing issues because I can't reproduce it.

I'll get in touch with you.

Regards,

Jesús

[jesus.g...@wazuh.com]

Hello again Marcio,

Can you provide us a more in deep description about the Virtual Machine you are using? I meant which software (Virtualbox, VMWare...), which OS

is the host that is building the VM, also it would be nice to share screenshot or a config file from the VM.

With the above information I can try to reproduce it and research about this epic problem with Kibana, your feedback is really appreciated.

Regards,

Jesús

[C. L. Martinez]

We have detected same behavior like Marcio said (and we have tested under VMware and RHEV servers, both with 8GB RAM and 4 vCPU). Hosts are RHEL 7.X fully patched and using ELK 6.3.2

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/1aa34c23-f6e2-40a8-b480-f77ef35ad44f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcio Costa]

Hi @Jesus.

We use oVirt 4.1 for virtualization.  All 3 hosts have 128GB RAM/each and CentOS 7.4 (basic installation).  All patchs installed on hosts.

VM have 12GB RAM and 2 vCpus, Centos 7.5.  Selinux disabled.  Disk size 30GB (only 23% used).

Packages:

# rpm -qa | egrep 'elastic|kibana|logstash'
logstash-6.4.0-1.noarch
elasticsearch-6.4.0-1.noarch
kibana-6.4.0-1.x86_64

# java -version
java version "1.8.0_172"
Java(TM) SE Runtime Environment (build 1.8.0_172-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.172-b11, mixed mode)

# rpm -qa | grep node
nodesource-release-el7-1.noarch
nodejs-6.14.4-1nodesource.x86_64

# rpm -qa | grep wazuh
wazuh-manager-3.6.0-1.x86_64
wazuh-api-3.6.0-1.x86_64

Thank you again by the efforts

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.

[jesus.g...@wazuh.com]

Great inventory Marcio, 

Ok we'll copy you here with our results. Now we can try to reproduce your environment.

Regards,

Jesús 

[Marcio Costa]

Some more informations:

# systemctl stop kibana elasticsearch logstash
# /usr/share/kibana/bin/kibana-plugin remove wazuh
Removing wazuh...
Plugin removal complete

# /usr/share/kibana/bin/kibana-plugin list
/usr/share/kibana/src/cli_plugin/list/list.js:41
        throw new Error('Unable to read package.json file for plugin ' + filename);
        ^

Error: Unable to read package.json file for plugin wazuh-logs
    at forEach.filename (/usr/share/kibana/src/cli_plugin/list/list.js:41:15)
    at Array.forEach (<anonymous>)
    at list (/usr/share/kibana/src/cli_plugin/list/list.js:32:44)
    at Command.processCommand (/usr/share/kibana/src/cli_plugin/list/index.js:61:22)
    at Command.<anonymous> (/usr/share/kibana/src/cli/command.js:116:20)
    at Command.listener (/usr/share/kibana/node_modules/commander/index.js:301:8)
    at emitTwo (events.js:126:13)
    at Command.emit (events.js:214:7)
    at Command.parseArgs (/usr/share/kibana/node_modules/commander/index.js:610:12)
    at Command.parse (/usr/share/kibana/node_modules/commander/index.js:458:21)
    at Object.<anonymous> (/usr/share/kibana/src/cli_plugin/cli.js:71:9)
    at Module._compile (module.js:652:30)
    at Module._extensions..js (module.js:663:10)
    at Object.require.extensions.(anonymous function) [as .js] (/usr/share/kibana/node_modules/babel-register/lib/node.js:152:7)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)

# rm -rf /usr/share/kibana/optimize/bundles
# export NODE_OPTIONS="--max-old-space-size=8192"

# /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.6.0_6.4.0.zip
Attempting to transfer from https://packages.wazuh.com/wazuhapp/wazuhapp-3.6.0_6.4.0.zip
Transferring 17568250 bytes....................

Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...

-> I opened new terminal and check kibana.log, I see the task finished after 8 minutes, but still running on first terminal:

{"type":"log","@timestamp":"2018-09-04T13:58:35Z","tags":["warning","config","deprecation"],"pid":22259,"message":"uiSettings.enabled is deprecated and is no longer used"}
{"type":"log","@timestamp":"2018-09-04T13:59:02Z","tags":["plugin","warning"],"pid":22259,"path":"/usr/share/kibana/plugins/wazuh-logs","message":"Skipping non-plugin directory at /usr/share/kibana/plugins/wazuh-logs"}
{"type":"log","@timestamp":"2018-09-04T13:59:02Z","tags":["plugin","warning"],"pid":22259,"path":"/usr/share/kibana/src/core_plugins/spy_modes","message":"Skipping non-plugin directory at /usr/share/kibana/src/core_plugins/spy_modes"}
{"type":"log","@timestamp":"2018-09-04T13:59:04Z","tags":["info","optimize"],"pid":22259,"message":"Optimizing and caching bundles for graph, monitoring, login, logout, ml, dashboardViewer, apm, wazuh, kibana, stateSessionStorageRedirect, status_page and timelion. This may take a few minutes"}
{"type":"log","@timestamp":"2018-09-04T14:07:15Z","tags":["info","optimize"],"pid":22259,"message":"Optimization of bundles for graph, monitoring, login, logout, ml, dashboardViewer, apm, wazuh, kibana, stateSessionStorageRedirect, status_page and timelion complete in 491.43 seconds"}
{"type":"log","@timestamp":"2018-09-04T14:07:15Z","tags":["info"],"pid":22259,"message":"Plugin initialization disabled."}

Checking again for plugins installed:

# /usr/share/kibana/bin/kibana-plugin list
wa...@3.6.0
/usr/share/kibana/src/cli_plugin/list/list.js:41
        throw new Error('Unable to read package.json file for plugin ' + filename);
        ^

Error: Unable to read package.json file for plugin wazuh-logs
    at forEach.filename (/usr/share/kibana/src/cli_plugin/list/list.js:41:15)
    at Array.forEach (<anonymous>)
    at list (/usr/share/kibana/src/cli_plugin/list/list.js:32:44)
    at Command.processCommand (/usr/share/kibana/src/cli_plugin/list/index.js:61:22)
    at Command.<anonymous> (/usr/share/kibana/src/cli/command.js:116:20)
    at Command.listener (/usr/share/kibana/node_modules/commander/index.js:301:8)
    at emitTwo (events.js:126:13)
    at Command.emit (events.js:214:7)
    at Command.parseArgs (/usr/share/kibana/node_modules/commander/index.js:610:12)
    at Command.parse (/usr/share/kibana/node_modules/commander/index.js:458:21)
    at Object.<anonymous> (/usr/share/kibana/src/cli_plugin/cli.js:71:9)
    at Module._compile (module.js:652:30)
    at Module._extensions..js (module.js:663:10)
    at Object.require.extensions.(anonymous function) [as .js] (/usr/share/kibana/node_modules/babel-register/lib/node.js:152:7)
    at Module.load (module.js:565:32)
    at tryModuleLoad (module.js:505:12)

Abort the optimization on first terminal and start all services, all working, but not with the correct way :(

[jesus.g...@wazuh.com]

Hi Marcio,

Old packages stored logs under /usr/share/kibana/plugins/wazuh-logs/. While the team is trying to reproduce your situation, try to execute the next commands please:

# systemctl stop kibana
# sudo -u kibana /usr/share/kibana/bin/kibana-plugin remove wazuh
# rm -rf /usr/share/kibana/plugins/wazuh-logs
# rm -rf /usr/share/kibana/optimize/bundles
# chown -R kibana:kibana /usr/share/kibana/optimize
# chown -R kibana:kibana /usr/share/kibana/plugins
# sudo -u kibana /usr/share/kibana/bin/kibana-plugin install https://packages.wazuh.com/wazuhapp/wazuhapp-3.6.0_6.4.0.zip
# systemctl restart kibana

Now check the output using the next command:

# watch -n0 'systemctl status kibana -l'

Let us know how the above commands worked for you.

Regards,

Jesús

[Marcio Costa]

Hi Jesus.

Tested and still not working.

BR

[jesus.g...@wazuh.com]

Just a follow up into this, 

Are you still facing some kind of problem? Is your stack working as expected?

Regards,

Jesús

[Marcio Costa]

Hi Jesus!!

Yes, the 'eco system' is working.  But always when is necessary remove/install Wazul app, I wait +- 30 minutes and abort with control-C.  Reboot the server and all is working.

I know this is not the right way, but it works.

Best Regards.