Cluster Max Shards per Node Modification

[Viren Kumar]

Hi, I recently noticed that logs in my dashboard failed to populate as the max number of shards (1000) was reached. We have an organizational requirement to store logs for at least 2 years. I increased the max number of shards per node using the dev tool as follows:

PUT _cluster/settings
{
  "persistent": {
    "cluster.max_shards_per_node": 1200
  }
}

This should give me an additional 1-2 months in my current environment. I now plan to add two additional Wazuh-indexer nodes to my all-in-one deployment to ensure I have enough shards for the next two years. My question is, after I add the two additional indexers, can I drop the max shards per node to 1000 again, or would that cause issues? I also noticed that the default number of shards per index is 3 in an all-in-one deployment. Would changing that to 1 after setting up the two new indexers be better or cause any issues?

[Md. Nazmur Sakib]

Hi Viren Kumar,

If you are planning to add two indexers. Make sure they are added to the cluster and the shards are balanced to other nodes before chaining the max_shards_per_node.


You can run this command to check if the nodes are connected in the cluster.
curl -XGET -k -u admin:<password> "https://<Indexer_IP>:9200/_cluster/health?pretty"

Run this command to check the node’s status in the cluster.
curl -XGET -k -u admin:<password> "https://<Indexer_IP>:9200/_cat/nodes?pretty"

You can run this command to get the result of which shard is located in which node.
curl -XGET -k -u admin:<password> "https://<Indexer_IP>:9200/_cat/shards?pretty"

Once the existing node moves some data to the new nodes, you can change the number cluster.max_shards_per_node to 1000 again.


This document will be useful for adding a new indexer node to the existing cluster.
https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/add-wazuh-indexer-nodes.html

If you have 3 indexer nodes, the ideal configuration will be 3 primary shards and 1 replica shard.

The number of shards needed for optimal performance depends on the number of nodes in the Wazuh indexer cluster. As a general rule, the number of shards must be the same as the number of nodes. For example, a cluster with three nodes should have three shards, while a cluster with only one node would only need one.

Each shard is a fully functional and independent "index" that can be hosted on any node in the Wazuh indexer cluster. The splitting is important for two main reasons:

Horizontal scaling.

Distribution and parallelization operations across shards, increasing performance and throughput.

If you change the number of primary shards to apply this to already created indices, you need to reindex those indices.

Check this document to learn more:
https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#shards-and-replicas

https://documentation.wazuh.com/current/user-manual/wazuh-indexer/re-indexing.html

Let me know if you need any further assistance.