Wazuh-indexer service not starting

[CJK]

Hi Team,

I have a distributed wazuh analyser setup with 3 Analyser nodes Analyser1, 2 and 3 

1 is master and while i check "GET _cat/nodeattrs?v&h=node,attr,value" i could see analyser 3 was not in my cluster i rebooted the node and still not able to turn the wazuh-indexer service up. Please find the logs below and help me on this.

Wazuh version - 4.7

/var/log/wazuh-indexer/wazuh-cluster.log

/var/log/syslog

Are attached 

Thanks in advance

[hasitha.u...@wazuh.com]

Hi CJK,

I noticed the HeapDumpOnOutOfMemoryError. To fix this, make sure your JVM heap size is large enough to handle the data. You might need to increase the heap size on your indexer nodes.

Here are some key points to keep in mind:
Use no more than 50% of your available RAM.
Don’t set the heap size over 32 GB.
Start by checking your memory with:
free -h

Then, update the heap size in the /etc/wazuh-indexer/jvm.options file. For example, if your server has 12 GB of RAM, set the heap size to 6 GB as shown below:
-Xms6g
-Xmx6g
After making these changes, restart the Wazuh indexer for them to take effect:
systemctl restart wazuh-indexer
You can refer to this link for more details:
https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking
Additionally, if the issue with JVM persists you need to increase your RAM memory of the indexer server and tune the JVM configuration accordingly.

Then try restarting the Wazuh-indexer and check if the issue persists.
systemctl restart-wazuh-indexer

Let me know the update on this.

Regards,
Hasitha Upekshitha

[CJK]

Hi  Hasitha,

Thank you so much for the quick response

My present Ram size is 32Gb and already configured 16G on jvm.options. still  systemctl restart-wazuh-indexer is failed. 

#root@anlyzr3:/home/cjk/bkp# free -h
               total        used        free      shared  buff/cache   available
Mem:            31Gi       312Mi        30Gi       2.0Mi       1.0Gi        30Gi
Swap:          8.0Gi          0B       8.0Gi

And on /etc/wazuh-indexer/jvm.options

-Xms16g
-Xmx16g

[hasitha.u...@wazuh.com]

Hi CJK,

 Dec 10 13:37:43 loganalyzer3 systemd-entrypoint[9942]: Likely root cause: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/var/lib/wazuh-indexer/nodes/0/_state/_oyv.si")))  

The index metadata or shard state file _oyv.si is corrupted. OpenSearch cannot parse it correctly, causing the node to fail during the initialization phase.

First, try to take a backup of this file path.

To do that stop the wazuh-indexer.
sudo systemctl stop wazuh-indexer

sudo cp -r /var/lib/wazuh-indexer/nodes /var/lib/wazuh-indexer/nodes_backup

Delete the corrupted _state files under /var/lib/wazuh-indexer/nodes/0/_state
sudo rm -rf /var/lib/wazuh-indexer/nodes/0/_state/_oyv.si 
If this not works try this command.
sudo rm -rf /var/lib/wazuh-indexer/nodes/0/_state/*
This removes metadata about the node's state, forcing OpenSearch to regenerate it.

Then restart the Wazuh-indexer again.
sudo systemctl start wazuh-indexer

Let me know the update on this.

Regards,
Hasitha Upekshitha

[CJK]

Hi Hasitha,

Thanks a lot!! that worked..

sudo rm -rf /var/lib/wazuh-indexer/nodes/0/_state/*

3rd Analyzer is working fine now. It is assigned as "Cold" node in cluster and some index status on it are in Red now. Is there any way to make it Green?

[hasitha.u...@wazuh.com]

Hi CJK,

I am glad that your indexer issue is resolved.
Could you please share the output of the below commands to check further?
On the Wazuh Web UI go to Index management > Dev Tools
Use this command:
GET _cluster/health
GET _cluster/allocation/explain
Regards,
Hasitha Upekshitha 

[CJK]

Hi Hasitha,

Thanks for the reply.

Unfortunately Node3 gone down again due to disk corruption. And health status is red

It will take some time to fix Node3 - mean time what can i do and how can i change health status to green and remove/repair red indexes with remaining nodes ?

Pfa for the results from Dev-tools

[hasitha.u...@wazuh.com]

Hi CJK,

I have seen that you have unassigned shards, you can remove them and check whether cluster back to green state.
curl --key /etc/wazuh-indexer/certs/admin-key.pem --cert /etc/wazuh-indexer/certs/admin.pem --insecure -XGET https://192.168.8.110:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED | awk '{print $1}' | xargs -i curl --key /etc/wazuh-indexer/certs/admin-key.pem --cert /etc/wazuh-indexer/certs/admin.pem --insecure -XDELETE "https://127.0.0.1:9200/{}"
Then check the cluster health again, if the issue persists, I suggest you to install a new OS and add new indexer node rather than using old node due to corruption frequently.
https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster.html#adding-wazuh-indexer-nodes

Let me know if this helps.

Regards,
Hasitha Upekshitha

[CJK]

Hi Hasitha,

Thanks for the reply, I can see around 20 unassigned shards some with 0 started and 1 and 2 unassigned. Is there any option to restore or repair with started one?

Or i need to delete all unassigned shards?

pfa for screenshot.

[CJK]

Hi Hasitha,

The issue is fixed I have remove all unassigned shards having 0,1,2 are unassigned and allocated "empty_primary" for the once which have at least 1 shard active. Now health status is Green.

can you help me on disabling the 3rd node or remove 3node from cluster temporarily?

Thanks

[hasitha.u...@wazuh.com]

Hi CJK,

I am glad that the cluster health back to green.

I would suggest you to please create a new post in the community channel with details about the new query. This will help us to provide more focused support on the issue that you are facing.

Regards,

Hasitha Upekshitha