Assistance Required: Wazuh Indices Retention Policy Issue

[xeption]

Hello Team,

I hope you are doing well.

I am currently facing an issue related to Wazuh indices retention.

Earlier, I had configured the indices retention policy for 90 days. Later, I changed the retention period to 30 days. However, after updating the policy, the indices older than 30 days are not being deleted automatically.

Additionally, I noticed an error related to index management, which I have pasted below. The error indicates a high disk usage issue. However, the Wazuh server is currently utilizing only about 66% of the total disk capacity.

[2026-03-29T03:44:59,323][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [tVGHjt6WSU6bqtSncOrOLg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

Could you please help me understand why the indices are not being deleted according to the updated retention policy and advise on how to resolve this issue?

Thank you for your assistance. I look forward to your guidance.

Regards,

Chandra

[Himanshu Sharma]

Hi Team,
Thank you for the details. Based on the error provided, the issue appears to be related to the Index State Management (ISM) system being unable to update or manage indices because the cluster previously entered a flood-stage watermark state.

The key error is:

ClusterBlockException[index [.opendistro-ism-config] blocked by:
[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark,
index has read-only-allow-delete block]

Even though the server is currently showing approximately 66% disk utilization, it is possible that at some point the disk usage exceeded the configured flood-stage watermark. When this occurs, OpenSearch automatically sets affected indices to read_only_allow_delete, and these blocks may remain in place even after disk usage returns to normal.

Could you please share the output of the following commands?

1. Cluster health

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/health?pretty

2. Watermark settings

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/settings?include_defaults=true&pretty

3. Verify whether indices are still blocked

curl -k -u <user>:<password>  "https://<indexer>:9200/_all/_settings/index.blocks.read_only_allow_delete?pretty"

Your 67% current usage is real and fine, but it's the current state, not the state when the watermark tripped. The block is a leftover, not a live reflection of disk usage right now.

The fix 1. Remove the read-only block manually (this is the actual unblock step)

curl -k -u <user>:<pass> -X PUT "https://<indexer-host>:9200/_all/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index.blocks.read_only_allow_delete": null
  }'

This clears the block cluster-wide. Specifically target .opendistro-ism-config too if you want to be surgical first:

curl -k -u <user>:<pass> -X PUT "https://<indexer-host>:9200/.opendistro-ism-config/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index.blocks.read_only_allow_delete": null
  }'

2. Verify the block is actually cleared

curl -k -u <user>:<pass> "https://<indexer-host>:9200/.opendistro-ism-config/_settings?pretty"

Confirm read_only_allow_delete no longer appears (or shows false/absent).

Please share the requested outputs, and we will help you investigate further.  

[xeption]

Hello Himanshu,

Thank you for the response,

I have perform operations which you actually want please find the below:

1. Cluster health

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/health?pretty

2. Watermark settings

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/settings?include_defaults=true&pretty

for this i have attached file.

3. Verify whether indices are still blocked

curl -k -u <user>:<password>  "https://<indexer>:9200/_all/_settings/index.blocks.read_only_allow_delete?pretty"

for this i find found nothing.

I followed steps which you given and i found some error and these operations i performed using admin user.
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}

Regards,

Chandra

[Himanshu Sharma]

Hello Chandra,

Thank you for sharing the outputs.

From the cluster health output, I can see that the cluster is currently in a yellow state:

"status": "yellow",
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"unassigned_shards": 30

Since this is a single-node cluster, the yellow status is expected if replica shards cannot be allocated.

The 403 error — this is the important one for your original ISM problem

"reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"

This is not a normal RBAC permission gap — .opendistro-ism-config is a protected system index in the OpenSearch Security plugin. Even a user with the admin backend role is blocked from directly modifying system indices via the standard REST API + basic auth, by design — this is a hardening feature so that a compromised or misconfigured admin account can't tamper with security-critical internal indices.

To modify a protected system index, you need to use the actual TLS admin certificate:

System index protection requires authenticating with the admin client certificate configured in opensearch.yml under plugins.security.authcz.admin_dn, not just basic auth with an admin role. Find your admin cert/key (usually named something like admin.pem / admin-key.pem in /etc/wazuh-indexer/certs/ or similar):

curl --cacert /etc/wazuh-indexer/certs/root-ca.pem \
     --cert /etc/wazuh-indexer/certs/admin.pem \
     --key /etc/wazuh-indexer/certs/admin-key.pem \

     -X PUT "https://<indexer-host>:9200/.opendistro-ism-config/_settings" \
     -H "Content-Type: application/json" \
     -d '{ "index.blocks.read_only_allow_delete": null }'

This bypasses the basic-auth RBAC layer entirely and authenticates as the actual cluster admin identity trusted at the transport/TLS level — this is the account that's allowed to touch protected system indices.

Regards,