Clusterstate : RED / Elasticsearch No yet initialized

[Mario Alejandro Porco]

Hi Wazuh experts!

i Have a problem with Wazuh server (all in one deployment used only for FIM )
The storage was full. Unfortunally i deleted accidentaly somes indices in /var/lib/elasticsearch/nodes/0/indices after that the  server crash i cant connect to the console.
I made some verifications   the clusterstate is red now


Open Distro Security Admin v7
Will connect to localhost:9300 ... done
Connected as CN=admin,OU=Docu,O=Wazuh,L=California,C=US
Elasticsearch Version: 7.10.2
Open Distro Security Version: 1.13.1.0
Contacting elasticsearch cluster 'elasticsearch' ...
Clustername: elasticsearch
Clusterstate: RED
Number of nodes: 1
Number of data nodes: 1

I attached somo logs: elasticsearch.log, wazuhapplog (Kibana)

How can reindex o delete indices to resolve this issue ?. ( i have a backup of the alerts)  
I just need to get server up.

Thanks in advance.

Mario

[mayte...@wazuh.com]

Hi Mario!

If you need to reinject data from the Wazuh manager, here is a good guide that shows how to achieve this: Recover-your-data-using-wazuh-alert-backups. In summary you must:

- Copy the recovery.py script from that article into your system

- Configure filebeat to read a recovery.json file

- Run the script for the time range for which you're missing data

You could delete the Elasticsearch red indices by running the following query in your Elasticsearch server for each index in red status (you may also use wildcards but be careful not to remove any healthy indices):

curl -k -u <user>:<pass> -X DELETE "https://localhost:9200/<index-name-in-red-status>"

Before deleting the indices, please make sure that your alerts backups and/or snapshots are properly performed.

It may be helpful to use the following command to list your red indices in order to avoid mistakes when deleting the red indices (It will only show red indices):

curl -k -u <user>:<pass> "https://localhost:9200/_cat/indices?s=index&health=red"

After deleting the red indices, restart the Elasticsearch and Kibana services.

If that does not solve your problem, send us the Elasticsearch logs again (all logged in since the Elasticsearch restart) to find the root cause.

Please keep us updated.

Best regards,

Mayte Ariza

[Mario Alejandro Porco]

Hi Mayte!
Thanks you so much for your assitance, deleting red indices I solved the problem, elasticsearch and the other services its running again. 

I will restore the backups of alerts and try to reinject the information following the documention as you provided.
One question more , after running health cluster command the status is red yet (atached file) I still unassigned shards
How can fix unassigned shards? or its a question of time to this cluster its OK again?

Thanks in advance

Mario

[mayte...@wazuh.com]

Hi Mario!
 
If the cluster health is in red status that means that there are still some indices in red status. It may be due to hidden indices.
 
Run the following command to show the indices in red status (it also includes the hidden ones):
curl -k -u <user>:<pass> "https://localhost:9200/_cat/indices?s=index&health=red&expand_wildcards=all”
 
After deleting the red indices, the cluster health should be fixed.

(Please be careful when removing the indices since you will lose all the data contained on those indices without the chance of recovering them if you do not have a backup)

Please keep us updated.
 
Best regards,
Mayte Ariza