Wazuh self-monitoring/alerting

[Kotory]

Hello wazuh team.
We had recently an issue with our Wazuh instance, when alerts weren't indexed due to shards being full, and it went unnoticed for quite some time. To avoid this kind of outage in the future, we are planning to set up internal monitoring through API/Scripts. My question is: what is the best practice to do that? Right now we have nothing better in mind than to check timestamp of last indexed log and if it's older than, let's say 60 min, generate alert/ send E-Mail. We would be thankful to get some advice how this can be done.

[Christopher Shaw]

I would suggest something a bit more off-the-shelf such as Uptime Kuma. I have a monitor that runs the health checks every 30 mins and monitors based of the JSON output/response. 

[Stuti Gupta]

Hi Kotory,
Hope you are doing well and thank you for using wazuh.

The shards can be full because of unassigned shards in your cluster. This can happen for a variety of reasons, including disk space issues, improper index settings, or other resource constraints. The error message like "[no_shard_available_action_exception]" suggests that dashboard is trying to access a shard that is not available. This can happen if the shard is not allocated or if there are issues with the underlying Indexer cluster.
To solve the issue you can follow the following steps:

Check all Indexer Unassigned Shards: You can check the name of the shards that are unassigned and their current state by using the command
curl -XGET -k -u admin:admin https://localhost:9200/_cat/shards?h=index,shards,state,prirep,unassigned.reason | grep UNASSIGNED

Delete or allocate Unassigned Shards: You can use the command:
curl -XGET -k -u admin:admin " https://localhost:9200/_cat/shards" | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "https://localhost:9200/{}"

Delete all the unassigned shards Or you can allocate the shards The reroute command allows for manual changes to the allocation of individual shards in the cluster. For example, a shard can be moved from one node to another explicitly, an allocation can be canceled, and an unassigned shard can be explicitly allocated to a specific node.
curl -X POST  -k -u admin:admin "https://localhost:9200/_cluster/reroute?metric=none"
reference: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-reroute.html

Delete old indicies: It is necessary to delete old indices to reduce the number of shards. It is necessary to check what the indices stored in the environment, the following API call can help:
GET _cat/indices then, it is necessary to delete indices that are not needed or older indices. Bear in mind that this cannot be retrieved unless there are backups of the data either using snapshots or Wazuh alerts backups.
The API call to delete indices is:
DELETE <index_name>

Setting a number of shards replication: We always recommend this option. In a single-node cluster, the number of replicas should be set to zero. This is accomplished by running the following command on the Wazuh indexer node or any central component allowed to authenticate using the Wazuh API:
curl -k -u "<INDEXER_USERNAME>:<INDEXER_PASSWORD>" -XPUT "https://<INDEXER_IP_ADDRESS>:9200/wazuh-alerts-" -H 'Content-Type: application/json' -d'
{
  "settings": {
    "index": {
      "number_of_replicas": 0
    }
  }
}'

Index retention: Security standards require keeping data available for audits for a minimum period of time. For data older than this retention period, you might want to delete it to save storage space. You can define specific policies to handle deletions automatically. You might also find these policies useful for index rollovers. For this, you can refer to https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.htmlFinally, restart all Wazuh components once again. Hope this information helps you. Please feel free to reach out to us for any information/issues.

To run the health check you can go to wazuh-dashboard > Settings> Miscellaneous>> Cick on Run

 

Hope this will help.
Regards,