ERROR Could not create wazuh-monitoring-2022.09.14 index on elasticsearch due to validation_exception

[James Rosser]

I believe this error is related to 1000/1000 shards being used, but I do not know how to fix this.. Can anyone help?

[Nicolas Agustin Guevara Pihen]

Hi, thank you for using Wazuh!
Elasticsearch has a limit of 1000 shards per node to avoid performance issues in the environment. You can either reduce the number of shards or increase the limit to solve the situation right now. I'd recommend the first approach, as the second one will need more resources and can lead to performance issues.

To reduce the number of shares you need to delete old indices, you can do it from your web interface opening the side bar, going to dev tools, and writing this query. For example, this will delete all alerts indices from january: 

DELETE wazuh-alerts-4.x-2022.01*

You can run GET _cat/indices to list all your indices to better analyze which of them do you want to delete. There are also wazuh monitoring and wazuh statistics indices. Those indices contains internal information from Wazuh (and not alerts from the agents or manager), and are safe to delete unless you need them. 


If you want to go with the second option and increase the limit of shards (not recommended), you can do it by running this command on the Dev Tools: 

PUT /_cluster/settings
{ "persistent" : {
     "cluster.max_shards_per_node": "2000"
  }
}

In that example, you will set the maximum limit to 2000. 


I will also recommend to implement a lifecycle policy to automatically delete the indices after a specified time, and thus avoiding this issue from happening again. You can read more information about that on this link. 

I hope you find this information helpful. Let me know if you have any questions or doubts! 

Regards,

[James Rosser]

Thank you very much for the assistance.

I have deleted some indices and nothing much has happened.

The errors I am seeing are the following:

Sep 15, 2022 @ 11:30:00 ERROR Could not create wazuh-monitoring-2022.09.15 index on elasticsearch due to validation_exception Sep 15, 2022 @ 11:30:00 ERROR index_not_found_exception Sep 15, 2022 @ 11:35:00 ERROR validation_exception Sep 15, 2022 @ 11:35:00 INFO [object Object] Sep 15, 2022 @ 11:35:00 ERROR validation_exception Sep 15, 2022 @ 11:35:00 INFO [object Object]

These are continuous.

Do you have any idea what might lead to this?

[James Rosser]

Correction. This is now fixed... I deleted the different types of indices from 2021 that you suggested above and also increased shards count to 3000 and it is working perfectly again.

Many many thanks

[Nicolas Agustin Guevara Pihen]

Hello, 
I'm glad to know that the problem is resolved. Don't hesitate to contact us again if you need more assistance! 
Regards,