Alerts Missing from Wazuh Index

[Miran Ul Haq]

Hi Team,

I am currently facing a problem. I was receiving logs/alerts on wazuh server but not on dashboard. Upon checking filebeat logs, it was identified that 1000/1000 shards were utilized.

I deleted the old shards which made the dashboard working again (receiving alerts). However, the alerts for the past 12 hours (during which the shards were full) are still missing.
Could you please guide me how to restore the alerts which were missing in those hours?

Thanks.

[Francisco Tuduri]

Hello Miran,

It is possible to recover the missing events using the alerts.json files, you should use the procedure outlined in this Wazuh blog post: https://wazuh.com/blog/recover-your-data-using-wazuh-alert-backups/
The recovery process involves the following steps:
 - Execute python script that processes the alerts files generated by the Wazuh server within a specified timeframe.
   - The script generates a new file with all the alerts that need to be indexed.
 - Configure filebeat to read the newly generated file.
 - Restart filebeat to apply the changes.

Please, refer to the blog post linked above for full details. If you need further assistance, feel free to reach out!

Regards!

[Miran Ul Haq]

Hi  Francisco,

Much appreciated for assistance. I have few other questons as well related to indexing if you could answer them. 

1) how to check how many shards are open/left

2) I found a guide to limit 1 shard to per index but it wasn't working. Could you please assist here as well?

TIA.

Regards,
Miran

[Francisco Tuduri]

Hi Miran,

You can navigate to Index Management > Dev Tools. In the Dev Tools console, you can execute API calls directly to manage and retrieve information about your OpenSearch cluster.

To check the status of shards in your cluster, use the following API call in the Dev Tools console:

GET _cat/shards?v  

This command provides a detailed overview of all the shards in your cluster, including their states (e.g., STARTED, UNASSIGNED). You can learn more about this API here: https://opensearch.org/docs/latest/api-reference/cat/cat-shards/

Another useful command is:

GET _cluster/health/  

This gives an overall view of your cluster's health, including details about active and unassigned shards. For additional details, refer to: https://opensearch.org/docs/latest/api-reference/cluster-api/cluster-health/


Regarding your question about setting the number of shards to one, have you reviewed this guide? https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#setting-the-number-of-shards

Let me know if you have any problem following that.

Regards!

[Miran Ul Haq]

Hi Francisco, 

I really appreciate your help.

After Calling the health API, I got the following result:
{
  "cluster_name": "wazuh-cluster",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 692,
  "active_shards": 692,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 30,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 95.84487534626038
}

It shows the health is Yellow. Is this good enough, or I need to fix something?
If so, could you suggest what would that be?

Other than that, I also noticed unassigned shards, could you shed some light on this as well.

As far as the number of shards guide is concerned, I did not follow that rather some other guide. Let me go through it and let you know.

Thanks.

Miran

[Francisco Tuduri]

Hi Miran,

Based on the OpenSearch documentation: "A green status means all primary shards and their replicas are allocated to nodes. A yellow status means all primary shards are allocated to nodes, but some replicas aren’t. A red status means at least one primary shard is not allocated to any node."

So, yellow is not critical, but it means your cluster does not currently have fault tolerance.
You should aim at having it in green.

To obtain more information you can use this API request:

GET _cluster/allocation/explain

This will provide an explanation for why each shard is unassigned.

[Miran Ul Haq]

Hi Francisco,

Much appreciate it.

I got the overall idea now on how to re-index (with new template) and manage cluster health.

Last thing I am having trouble with is, though we were able to re-index old indices to 1 shard only. However, the newer ones are still being created on the old template.
I want my new indices to have 1 shard as well and not 3. 

Am I missing/lacking something?

Best Regards,
Miran

[Francisco Tuduri]

Hi Miran,

It sounds like the issue is that the index template isn't being applied to your new indices.

Did you follow this guide? https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#setting-the-number-of-shards

Did you encounter any problem with it?

[Miran Ul Haq]

Hi Francisco,

Yes, it worked. 

Thanks for all your help.

Best Regards,
Miran

[Francisco Tuduri]

Hi Miran!

Great! Thanks for letting us know.

Have nice day!