Wazuh Shards limitation

[SP_11]

Hi Team,

Wazuh has utilized all the 1000/1000 shards and data is not getting populated to dashboards due to this. (once I delete some shards,new data can be seen)

I would like to know, how can I mitigate this? 

What are the consequences of deleting all the shards and free up 1000 shards? Will it delete any data I currently have on wazuh?

Thank you

[Antonio David Gutiérrez]

Hi,

Thank you for using Wazuh!

This means you reached the shards limit count (1000 by default in the node). To fix this issue, there are multiple options:

- Delete indices. This frees shards. You could do it with old indices you don't want/need. Or even, you could automate it with ILM/ISM policies to delete old indices after a period of time as explained in this post: https://wazuh.com/blog/wazuh-index-management.
 Note:
  ILM: Index Lifecycle Management (used by X-Pack).
  ISM: Index State Management (used by Open Distro for Elasticsearch and OpenSearch)

- Add more nodes to your Elasticsearch/Wazuh indexer cluster.

- Increment the max shards per node (not recommended). But if you do this option, make sure you do not increase it too much, as it could provoke inoperability and performance issues in your Elasticsearch/Wazuh indexer cluster. To do this:
  curl -k -u USERNAME:PASSWORD -XPUT ELASTICSEARCH_HOST_ADDRESS/_cluster/settings -H "Content-Type: application/json" \
  -d '{ "persistent": { "cluster.max_shards_per_node": "MAX_SHARDS_PER_NODE" } }'
  
  replace the placeholders, where:
- USERNAME : username to do the request
- PASSWORD : password for the user
- ELASTICSEARCH_HOST_ADDRESS: Elasticsearch/Wazuh indexer host address. Include the protocol https if needed.
- MAX_SHARDS_PER_NODE: Maximum shards by node. Maybe you could try with 1200 o something like that, depending of your case.

More info: https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster

You are right, delete shards by deleting indices, remove data so you should be careful with the indices that you delete.

A good approach is to automate the indices deletion through ILM/ISM policies, so you never have to worry about it again.

For another hand, you could review the indices you have and check how many shards they use. For example, the indices called `wazuh-alerts-4.x-YYYY-MM-dd` could be using 3 shards by index, in the case you have less than 3 Elasticsearch/Wazuh indexer nodes, you could modify it to use 1, so you could reduce in 2 shards, the total shards occupied. Same for others indices related to Wazuh, as `wazuh-monitoring-` or `wazuh-statistics-`.

The number of shards of `wazuh-alerts-4.x-` indices is defined in the template indexed by Filebeat. You could change and index the template, so the future indices will be created with a different number of shards.
The `wazuh-monitoring-` and `wazuh-statistics` shards count can be modified through the Wazuh plugin for Kibana or Wazuh dashboard.

[SP_11]

Hi David,

Thanks for the detailed information.

When I execute "GET _cat/shards" it shows both "wazuh-alerts..." and "security-auditlog.."

May I know the impact of removing "security-auditlog.."

Thank you

[Antonio David Gutiérrez]

Hi, the security-auditlog-* indices store information related to accessing to your Wazuh indexer/Elasticsearch (if using OpenDistro for Elasticsearch) cluster and are useful for compliance purposes or in the aftermath of a security breach. So, it depends on your use case, you could remove them if you don't need them, only store the required ones for some compliance. If you want to automatize the deletion of these indices after some condition you could use the ILM/ISM policies.

References:
- Using Wazuh indexer (based on OpenSearch): https://opensearch.org/docs/1.2/security-plugin/audit-logs/index/
- Using Elasticsearch with Open Distro for Elasticsearch: https://opendistro.github.io/for-elasticsearch-docs/docs/security/audit-logs/