clusterd on master node crashes from openvas scan (4.3.7)

[Павел Покровский]

Hi.

We're using Wazuh-4.3.7 (Revision 40320) and also OpenVAS for vulnerability scans

Once OpenVAS scan runs on wazuh master node, it causes crash of wazuh-clusterd service without any informative output except.

2023/06/16 09:18:43 INFO: [Worker] [Main] Connection from ('10.4.17.44', 38930)
/var/ossec/bin/wazuh-clusterd: line 38: 3010877 Killed ${WAZUH_PATH}/${WPYTHON_BIN} ${PYTHON_SCRIPT} "$@"

Where 10.4.17.44 is IP address of OpenVAS machine and 38930 apparently is a source port.

There're no additional log lines before crash occurs, even in debug mode (-fdd key). 

I'm not sure how we could debug this issue further, or even narrow it down to specific Python script. We'd appreciate if someone could share their experience. Thank you.

[Juan Nicolás Asselle (Nico Asselle)]

Hi Pavel,

I'm going to test this locally, but could you please share with us your OpenVAS version and configuration file?

Regards,

Nico

[Павел Покровский]

Hi Nico

Thank you for your response

We're using openvas community 22.4.0

target configuration includes only master node, port 1516

scanner: openvas default

scan config (built-in): full and fast

maximum nvt per host: 4

it crashes parent process of /var/ossec/framework/scripts/wazuh-clusterd.py script

child processes (two in our case) remain orphaned

ср, 21 июн. 2023 г. в 16:13, 'Juan Nicolás Asselle (Nico Asselle)' via Wazuh mailing list <wa...@googlegroups.com>:

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/Yk4RlCadpW8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/4d0ff9df-c401-4212-adcb-6f3092342d75n%40googlegroups.com.

[Juan Nicolás Asselle (Nico Asselle)]

Hi Pavel,

I was able to reproduce this error and I’m doing a RCA about it.

Error:

SummaryThis plugin checks if the port scanners did not kill a service. Detection ResultThis port was detected as being open by a port scanner but is now closed. This service might have been crashed by a port scanner or by a plugin Detection Method Details: Check open ports OID: 1.3.6.1.4.1.25623.1.0.10919 Version used: 2022-07-27T10:11:28Z

​

[Juan Nicolás Asselle (Nico Asselle)]

Hi Pavel,

Could you please check if your host, where Wazuh Manager and OpenVAS are being run, has an `oom-kill` log on Kernel log? next command line could be useful `cat /var/log/kern.log | grep oom-kill`

Regards,

Nico

[Павел Покровский]

Hi Nico

We have it in /var/log/messages

Jun 22 14:23:18 wazuhm01 kernel: wazuh-analysisd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jun 22 14:23:19 wazuhm01 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/wazuh-manager.service,task=python3,pid=648164,uid=989

среда, 21 июня 2023 г. в 21:46:11 UTC+3, Juan Nicolás Asselle (Nico Asselle):

[Juan Nicolás Asselle (Nico Asselle)]

Hi Pavel,

It seems that is not recommended to run OpenVAS and Wazuh Manager Stack on the same host if the Hardware requirements (https://documentation.wazuh.com/current/quickstart.html#requirements and https://greenbone.github.io/docs/latest/22.4/container/index.html#hardware-requirements) are not capable of handling both services and kernel's oom-killer is invoked.

I was able to check that this does not happen if:

- Memory is increased

- Use independent hosts

Hope this information is useful

[Павел Покровский]

Hi Nico

They are not on the same host. Loglines above are taken from wazuh master host, openvas runs on different machine

Wazuh master host only runs wazuh-manager software. Stuff like opensearch/dashboard is also runs on separate machines.

четверг, 22 июня 2023 г. в 15:09:16 UTC+3, Juan Nicolás Asselle (Nico Asselle):

[Павел Покровский]

Hi Nico

we've increased RAM for master node from 4 to 6 Gb and the issue is not reproducing anymore. Thank you very much for your support!

четверг, 22 июня 2023 г. в 15:33:18 UTC+3, Павел Покровский: