Analysysd killed by OOM Killer

[Mic G]

Dear Team,

About once every 7 days, we encounter a problem - OOM Killer terminates wazuh-analysisd, because the daemon consumes too much RAM, and this leads to the shutdown of wazuh-manager.service:

root@rumosvlzwzhap:/var/ossec/etc# grep -i "oom\|killed" /var/log/syslog /var/log/kern.log
/var/log/syslog:Sep 24 06:06:00 rumosvlzwzhap systemd[1]: system.slice: A process of this unit has been killed by the OOM killer.
/var/log/syslog:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.167766] kworker/2:0 invoked oom-killer: gfp_mask=0xdc0(GFP_KERNEL|__GFP_ZERO), order=0, oom_score_adj=0
/var/log/syslog:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.167859]  oom_kill_process.cold+0xb/0x10
/var/log/syslog:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.168308] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
/var/log/syslog:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.168446] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/wazuh-manager.service,task=wazuh-analysisd,pid=2393620,uid=115
/var/log/syslog:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.168537] Out of memory: Killed process 2393620 (wazuh-analysisd) total-vm:17287696kB, anon-rss:13092084kB, file-rss:0kB, shmem-rss:0kB, UID:115 pgtables:29812kB oom_score_adj:0
/var/log/syslog:Sep 25 05:07:33 rumosvlzwzhap systemd[1]: wazuh-manager.service: A process of this unit has been killed by the OOM killer.
/var/log/syslog:Sep 25 05:07:34 rumosvlzwzhap kernel: [9136483.200424] oom_reaper: reaped process 2393620 (wazuh-analysisd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
/var/log/syslog:Sep 25 05:07:40 rumosvlzwzhap systemd[1]: wazuh-manager.service: Failed with result 'oom-kill'.
/var/log/kern.log:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.167766] kworker/2:0 invoked oom-killer: gfp_mask=0xdc0(GFP_KERNEL|__GFP_ZERO), order=0, oom_score_adj=0
/var/log/kern.log:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.167859]  oom_kill_process.cold+0xb/0x10
/var/log/kern.log:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.168308] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
/var/log/kern.log:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.168446] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/wazuh-manager.service,task=wazuh-analysisd,pid=2393620,uid=115
/var/log/kern.log:Sep 25 05:07:32 rumosvlzwzhap kernel: [9136480.168537] Out of memory: Killed process 2393620 (wazuh-analysisd) total-vm:17287696kB, anon-rss:13092084kB, file-rss:0kB, shmem-rss:0kB, UID:115 pgtables:29812kB oom_score_adj:0
/var/log/kern.log:Sep 25 05:07:34 rumosvlzwzhap kernel: [9136483.200424] oom_reaper: reaped process 2393620 (wazuh-analysisd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

We use a distributed structure, with 16GB of RAM available on the server with the manager role. 

Manager information:
Version v4.10.1
Installation path /var/ossec
Installation type server
Agents 40

We also send logs from Kaspersky Security Center via syslog to Wazuh.

There are no dropped events in wazuh-analysisd.state, I removed all the custom rules, but RAM consumption continued to grow (I didn't wait for it to be fully filled, because this one will take 7 days). Ossec log: 2025/09/24 04:07:20 wazuh-modulesd:content-updater: WARNING: Couldn't run full content download: Error -1 from server: Timeout was reached.
2025/09/24 04:11:42 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
2025/09/24 04:16:04 wazuh-modulesd:content-updater: WARNING: Couldn't run full content download: Error -1 from server: Timeout was reached.
2025/09/24 04:20:26 wazuh-modulesd:content-updater: ERROR: Action for 'vulnerability_feed_manager' failed: Error -1 from server: Timeout was reached.
2025/09/24 04:57:23 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/09/24 04:57:30 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/09/24 05:24:46 wazuh-modulesd:content-updater: WARNING: Couldn't run full content download: Error -1 from server: Timeout was reached.
2025/09/24 05:29:09 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
2025/09/24 05:33:31 wazuh-modulesd:content-updater: WARNING: Couldn't run full content download: Error -1 from server: Timeout was reached.
2025/09/24 05:37:53 wazuh-modulesd:content-updater: ERROR: Action for 'vulnerability_feed_manager' failed: Error -1 from server: Timeout was reached.
2025/09/24 05:57:31 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2025/09/24 05:57:39 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2025/09/24 06:42:13 wazuh-modulesd:content-updater: WARNING: Couldn't run full content download: Error -1 from server: Timeout was reached.
2025/09/24 06:46:35 wazuh-modulesd:content-updater: WARNING: Offset processing failed. Triggered a snapshot download.
2025/09/24 06:50:58 wazuh-modulesd:content-updater: WARNING: Couldn't run full content download: Error -1 from server: Timeout was reached.
2025/09/24 06:55:20 wazuh-modulesd:content-updater: ERROR: Action for 'vulnerability_feed_manager' failed: Error -1 from server: Timeout was reached.

Also I attach screenshot of available memory and swap.  

I will be glad of any help.

BR,
Mikhail Glebov

[Victor Carlos Erenu]

Hello Mic

The Analysisd process usage depends on the number of alerts the process has to analyze and the resources it estimates as available, as explained in our documentation.
https://documentation.wazuh.com/current/user-manual/reference/daemons/wazuh-analysisd.html

You say you've removed the custom rules, but what causes the most usage is reviewing the logs using the decoders. The decoders indicate which alerts might be possible.

Regarding your memory usage issue, you should try reducing the number of threads working in parallel, which is explained in this section of the documentation.

[Mic G]

Vicrot, Thanks for the quick reply! 

Apparently, I don't really understand how analysisd works. Shouldn't some amount of RAM be released after the number of events decreases (for example, during night hours)?

Wouldn't an increase in RAM help in my case? It seems that if you add memory, analysisd utilizes it completely.

As for reducing number of threads working in parallel, for which event decoder do I need to reduce it (event decoder, Syscheck, Syscollector etc.)? I'm attaching a screenshot of the statistics.  

BR,

Mikhail

вторник, 30 сентября 2025 г. в 18:11:44 UTC+3, Victor Carlos Erenu:

[Victor Carlos Erenu]

Hi Mic

The usage you report, depending on the alert load Wazuh Manager may receive from the Analysisd process, may be normal usage.

You should check if the usage increases beyond what you're currently using, as this may be normal usage and depends greatly on the alert load you may be receiving from Kaspersky.