wazuh-analysisd crashing randomly after hot reloading

[Sergio]

Hi,

I’m running a 3-node cluster consisting of 1 Wazuh Manager and 2 Wazuh Workers, each with its own indexer.

All nodes are running Wazuh version 4.14, and I’ve been experiencing an intermittent issue. When applying changes to rules or decoders, the wazuh-analysisd module on one of the nodes occasionally crashes. The behavior is inconsistent: sometimes the changes are applied successfully, while other times one of the modules crashes immediately after the update.

I haven’t been able to identify a clear pattern so far. In some cases, the crash affects the wazuh-analysisd of Wazuh Manager, and in others it impacts either Worker 1 or Worker 2.

Below is the relevant information I’ve found in the ossec.log.

[Antonio David Gutiérrez]

Hi, it seems the problem could be difficult to debug if this is not reproducible in a consistent way.

I could not find recents issues like you are experiencing.

Q0. Provide the exact version of the Wazuh server.

Q1. What are the hardware specs of the Wazuh server hosts?

Q2. When applying the changes to rules or decoders, what are you doing? Only editing the files directly (or through the Wazuh dashboard) or you are executing the reload function too?

Q3. After you detect the analysisd crashed, did you need to restart the Wazuh server or this was restarted by the app after a time without user interaction?

In the provided logs, I see some of them related to duplicated rules. Consider to fix the problem with the rules, removing the duplicated rules and other problem that could appear in the logs.
Additionally, there are errors related to MITRE technique ID could not be found in the database. 

Some things to consider for troubleshooting the problem
D1. Fix the problems with the duplicated rules.
D1.1. Consider reviewing the rules definition with special attention to the custom rules: ensure there are no circular dependencies in the rules declaration => avoid the usage of if_group, syntax errors, etc...
D2. Fix the problem with the MITRE techinques can not be found. Issue with similar problem, take a look to debug: https://github.com/wazuh/wazuh/issues/25321
D3. Monitor the disk, CPU and RAM usage while replicating the error.
D4. Review the logs of the Wazuh servers. For more details, you could consider increasing the verbosity of the analysisd module (see below). 

How to increase the verbosity of analysisd module:
1. Stop the Wazuh server service
2. Add `analysisd.debug=2` to the `/var/ossec/etc/local_internal_options.conf` file
3. Restart the Wazuh server
After troubleshooting, consider revert the change.
Reference: https://documentation.wazuh.com/4.14/user-manual/reference/internal-options.html#analysisd

If you need more assistance and you go to provide logs(obfuscate sensitibve data), consider using text instead of image if possible.