wazuh-indexer and shard limit saturation, i need documentation.

[valombre.d Delanhuyi]

hi,
I noticed today that i had no more indexes/shard created .. (no more updates on last 7 days datas)
looking logs and services => /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log

Jun 9, 2022 @ 15:00:02  ERROR  Could not create wazuh-monitoring-2022.23w index on elasticsearch due to validation_exception
Jun 9, 2022 @ 15:00:02  ERROR  index_not_found_exception

checking log history =>

{"date":"2022-05-31T09:36:04.256Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status code 500"}
{"date":"2022-06-02T22:00:18.247Z","level":"error","location":"monitoring:fetchAllAgentsFromApiHost","message":"ApiID: default, Error request with offset/limit 0/500: Request failed with status code 500"}
{"date":"2022-06-03T17:45:15.022Z","level":"error","location":"monitoring:getApiInfo","message":"Request failed with status code 500"}
{"date":"2022-06-05T22:00:11.606Z","level":"error","location":"cron-scheduler|SaveDocument","message":"resource_already_exists_exception"}
{"date":"2022-06-05T22:00:11.616Z","level":"error","location":"monitoring:createIndex","message":"Could not create wazuh-monitoring-2022.23w index on elasticsearch due to validation_exception"}
{"date":"2022-06-05T22:00:11.625Z","level":"error","location":"monitoring:insertMonitoringDataElasticsearch","message":"index_not_found_exception"}

After trying to understand where is the problem i saw in the web interface an error on policie index with shard saturation ..
i had a similar saturation previously on the 2.3.x and used the solution as discribe here https://groups.google.com/g/wazuh/c/YoByxPoKMjY and with the blog post https://wazuh.com/blog/wazuh-index-management/

But after migration, wazuh-indexer included, the index policy in place didnt work anymore, i had to clean the policy and remove from "index management/managed indices" all entries, they were stuck for each index.

After that i remove manually from dev tool some wazuh-alerts,wazuh-monitoring and wazuh-statistics.

Index now wazuh-monitoring-2022.23w created (without restart or so) and i have no more alerts in the log, data and graphs populating again.

Now i would like to create a new policy (or several) and would like a clean updated example (or documentation) for wazuh-indexer (the old one seams deprecated), could you help me with this ?

I think this part is really complex and should be automatically configured depending of available resources (ram/cpu) or at least some policies documented for general cases, for mine it's 8GB ram 8 thread/cpu(2.13GHz) all in one debian11 VM for 50/60 agents.

Bonus : if you have a web (or simple) integrated/method to be alert by mail when shard saturation occurred it would be nice ;)

Thanks for the help.

Regards

[José Fernández]

Hello Valombre.d,

Yes, we have to improve our index management blog post to include some of the new changes introduced but basically, it works nearly the same as OpenDistro ISM described section. I will attach some screenshots of how to replicate the Elastic configuration (I made a testing policy). Maybe you will need to edit or tune it to match your requirements.

You could use the error notification section inside the policy for the notifications part. Including a custom or slack webhook will produce the desired event by slack/mail. Currently, we haven't published any method.

Please open us an issue, asking for an automated policy generation. We will include in the roadmap as soon as possible. https://github.com/wazuh/wazuh-kibana-app/issues/new/choose

I hope it helps you, don't hesitate to ask us if you have any doubts.

José.

[valombre.d Delanhuyi]

Hi José,

thanks a lot for all the screenshots and explanations, i'll going to test it asap, and i will ask for an automated policy generation as you suggest ;)

Regards

[valombre.d Delanhuyi]

Hi,

i created my policy minimising the replica to 0 and adding wazuh-alerts-*,wazuh-monitoring-* and wazuh-statistics-*, it seams to work well and hards count are around 500

# curl -X GET "https://localhost:9200/_cluster/health?pretty" -u xxxx:xxxx -k           

{
 "cluster_name" : "wazuh-cluster",
 "status" : "yellow",
 "timed_out" : false,
 "number_of_nodes" : 1,
 "number_of_data_nodes" : 1,
 "discovered_master" : true,
 "active_primary_shards" : 513,
 "active_shards" : 513,
 "relocating_shards" : 0,
 "initializing_shards" : 0,
 "unassigned_shards" : 237,
 "delayed_unassigned_shards" : 0,
 "number_of_pending_tasks" : 0,
 "number_of_in_flight_fetch" : 0,
 "task_max_waiting_in_queue_millis" : 0,
 "active_shards_percent_as_number" : 68.4
}

But no way to create an Error notification with mail using my wazuh_supervison  from Alerting/Destinations/, i check the syntax from the help link => https://opensearch.org/docs/latest/im-plugin/ism/policies/#error-notifications, channel syntax is not usable in policy

I don't want to use destination options like "Slack, Amazon Chime, or webhook URL" as i don't have those services available.

If you have another method to alert me on shard saturation (or index problem) using classic mail ( i already have alerts from ossec.conf rules etc ..) i take it ;)

Thanks.

PS: i created the issue on github => https://github.com/wazuh/wazuh-kibana-app/issues/4265

[José Fernández]

Hello Valombre.d,

I have done some tests over the mail notification. You could include a custom webhook to manage the notifications may be a simple python API that forwards the webhook request to a file and monitor such file with Wazuh then, you will need to configure a custom rule to trigger the e-mail from Wazuh postfix configuration.

We are working on a notifications plugin that will include a better way to perform that matter but it is intended for 4.4.0.

Thanks for your feedback and issue. I hope it helps you, don't hesitate to ask us if you have any doubts.