wazuh-indexer issue after 4.9 upgrade

[M V]

Hello Wazuh-gurus,

Thank you for issuing a newer version of Wazuh package. I'm going from 4.8-->4.9. `wazuh-dashboard` service doesn't start due to an issue w/ the `wazuh-indexer` service. `wazuh-indexer` starts okay and the logs are fine. However, indexer as such seems to not have any errors. 

log file for dashboard is attached. I presume its a shard migration issue?

Any pointers to nudge me in the right direction would be much appreciated.

[hasitha.u...@wazuh.com]

Hello  Maulik,

From your logs I can see you are hitting a circuit breaking exception due to the data size limit being exceeded.

Sep 06 21:30:57 scanner opensearch-dashboards[940]: FATAL {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1253514848/1.1gb], which is larger than the limit of [1245184000/1.1gb]

We need to ensure the JVM heap size is adequate to handle the data. In such a case, you need to increase the JVM heap limits in your indexer nodes. Keep in mind these restrictions:

• Use no more than 50% of available RAM.
• Use no more than 32 GB.

First, let’s check the memory of your indexer nodes:
free -h

Then edit the /etc/wazuh-indexer/jvm.options file and change the JVM flags.
For example: If your server has 12GB of RAM, you can set the limits to 6GB as below:
-Xms6g
-Xmx6g

Once the heap limit is updated, you need to restart the wazuh-indexer to make this effective:
systemctl daemon-reload
systemctl restart wazuh-indexer
systemctl restart wazuh-dashboard
Reference: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-locking

If you still face the issue, check Wazuh dashboard certificates:
Run this command to list your certificates:
ls -lrt /etc/wazuh-dashboard/certs/

Make sure the path and file names match those in:
/etc/wazuh-dashboard/opensearch_dashboards.yml

After updating the certs restart the dashboard.
systemctl restart wazuh-dashboard

If the issue still persists, share the dashboard log file.
/usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log

Let me know the update for further assistance.

Regards,
Hasitha Upekshitha

[M V]

Thank you so much Hasitha for the detailed response.

With this being an upgrade, i didn't think the jvm options from previous versions needed a change. Old->new:

-Xms1250m
-Xmx1250m

-Xms2g
-Xmx2g

Re-starting (using your guidance):

systemctl daemon-reload
systemctl restart wazuh-indexer

systemctl restart wazuh-dashboard

 resulted in a different error which said, in speaking from memory, something the effect.. another process started an upgrade process but wasn't able to finish. Please check .kibana3/<really long alphanumeric word>... So, I checked the index ids, and removed all related to .kibana3 

curl -X DELETE -ku <user>:<password> https://<fqdn>:9200/.kibana_3

followed by

systemctl restart wazuh-indexer

systemctl restart wazuh-dashboard

/var/ossec/bin/wazuh-control stop

/var/ossec/bin/wazuh-control start

My wazuh world is whole again. I'll continue monitoring for any odd behavior. But, I may be out of the danger zone now.

/r

maulik

[hasitha.u...@wazuh.com]

Hello M V,

I'm glad to hear your issue is resolved. Feel free to reach out if you need any further assistance.

Regards,
Hasitha Upekshitha