wazuh indexer not starting up

[Asmit Desai]

Hi team,

My Wazuh indexer was working fine until I made some rule changes on the manager side. Since then, indexer fails to start — it consistently times out during startup rather than throwing a clear error.

Environment: Wazuh 4.14.5 (manager, indexer, dashboard all on this version), Ubuntu, systemd-managed services.

Relevant journal output across two consecutive start attempts:

Jul 23 14:24:44 systemd[1]: Starting wazuh-indexer.service... Jul 23 14:26:53 systemd-entrypoint: WARNING: System::setSecurityManager deprecated method called by OpenSearch (opensearch-2.19.5.jar) Jul 23 14:27:44 systemd[1]: wazuh-indexer.service: start operation timed out. Terminating. Jul 23 14:27:49 systemd[1]: Failed with result 'timeout'. Jul 23 14:27:49 systemd[1]: Consumed 22.493s CPU time, 16.3G memory peak, 0B swap peak. Jul 23 14:29:35 systemd[1]: Starting wazuh-indexer.service... Jul 23 14:32:05 systemd-entrypoint: WARNING: same setSecurityManager deprecation warning Jul 23 14:32:35 systemd[1]: start operation timed out. Terminating. Jul 23 14:32:37 systemd[1]: Failed with result 'timeout'. Jul 23 14:32:37 systemd[1]: Consumed 22.483s CPU time, 27.4M memory peak, 0B swap peak.

Notes:

• The setSecurityManager warning appears to be a known deprecation notice from OpenSearch 2.19.5 (Java), not an error.
• First attempt used ~16.3G memory (heap is set to -Xmx16g), suggesting it got fairly deep into JVM initialization before being killed.
• Second attempt used only 27.4M — far less — suggesting it failed much earlier in the sequence this time, despite an unchanged config in between.
• No explicit error/exception is logged — just the systemd timeout kill.

Has anyone seen indexer startup times vary this significantly between consecutive attempts on the same unchanged config? Looking for pointers on where to look next — GC logs, OpenSearch's own log file, or something specific to check given the inconsistent memory footprint between runs.

Thanks!

[Kasim Mustapha]

Hi Asmit,

Thanks for the detailed journal output — the timeout kill and the memory figures are a helpful starting point. A quick note first: the manager-side rule changes are very unlikely to be related to this. The manager ruleset and the indexer's startup path are decoupled, so I'd set that aside as a cause for now.

The most telling detail is the memory difference between your two runs: ~16.3 GB peak on the first attempt versus only 27.4 MB on the second, on the same config. That pattern (plus a heap set to -Xmx16g) points toward the indexer being resource-starved or killed before it finishes bootstrapping, with systemd's start timeout firing before OpenSearch is up — rather than a config error. To confirm that and rule out the alternatives, could you send the following?

1. Indexer's own log, not just the journal — this is where the real exception (if any) lands.
   How: cat /var/log/wazuh-indexer/wazuh-cluster.log (the file is named after your cluster; ls /var/log/wazuh-indexer/ if unsure). The last ~200 lines from a failed start attempt are what we need.

2. Total host RAM and current usage — to confirm whether a 16 GB heap fits alongside the manager and dashboard on this host.
   How: free -h and nproc

3. The JVM heap settings — to confirm what's actually configured vs. what the host can support.
   How: grep -E '^-Xm' /etc/wazuh-indexer/jvm.options (and contents of /etc/wazuh-indexer/jvm.options.d/ if anything is in there)

4. Disk usage on the data and log partitions — a full or near-full disk (past OpenSearch's flood-stage watermark) will also block startup.
   How: df -h /var/lib/wazuh-indexer /var/log/wazuh-indexer

5. Whether the OOM killer fired — this would directly confirm the memory theory.
   How: dmesg -T | grep -i -E 'oom|killed process' and journalctl -k | grep -i oom

6. Confirmation of the versions and that this is an all-in-one host — you mentioned 4.14.5 for all three components on one Ubuntu box; just confirming so we scope the repro correctly.
   How: cat /etc/os-release and /usr/share/wazuh-indexer/bin/opensearch --version

Optional but useful if the log in item 1 doesn't show a clear cause: the GC log at /var/log/wazuh-indexer/gc.log around the failure timestamp.

Once we have these — especially items 1, 2, and 5 — we should be able to confirm the root cause quickly and, if it is heap sizing, give you a corrected value to set.

Best regards,
Kasim Mustapha