Assistance Required: Wazuh Indices Retention Policy Issue

[xeption]

Hello Team,

I hope you are doing well.

I am currently facing an issue related to Wazuh indices retention.

Earlier, I had configured the indices retention policy for 90 days. Later, I changed the retention period to 30 days. However, after updating the policy, the indices older than 30 days are not being deleted automatically.

Additionally, I noticed an error related to index management, which I have pasted below. The error indicates a high disk usage issue. However, the Wazuh server is currently utilizing only about 66% of the total disk capacity.

[2026-03-29T03:44:59,323][ERROR][o.o.i.i.ManagedIndexCoordinator] [node-1] Failed to clear ManagedIndexMetadata for index uuid: [tVGHjt6WSU6bqtSncOrOLg#metadata], failureMessage: ClusterBlockException[index [.opendistro-ism-config] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];]

Could you please help me understand why the indices are not being deleted according to the updated retention policy and advise on how to resolve this issue?

Thank you for your assistance. I look forward to your guidance.

Regards,

Chandra

[Himanshu Sharma]

Hi Team,
Thank you for the details. Based on the error provided, the issue appears to be related to the Index State Management (ISM) system being unable to update or manage indices because the cluster previously entered a flood-stage watermark state.

The key error is:

ClusterBlockException[index [.opendistro-ism-config] blocked by:
[TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark,
index has read-only-allow-delete block]

Even though the server is currently showing approximately 66% disk utilization, it is possible that at some point the disk usage exceeded the configured flood-stage watermark. When this occurs, OpenSearch automatically sets affected indices to read_only_allow_delete, and these blocks may remain in place even after disk usage returns to normal.

Could you please share the output of the following commands?

1. Cluster health

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/health?pretty

2. Watermark settings

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/settings?include_defaults=true&pretty

3. Verify whether indices are still blocked

curl -k -u <user>:<password>  "https://<indexer>:9200/_all/_settings/index.blocks.read_only_allow_delete?pretty"

Your 67% current usage is real and fine, but it's the current state, not the state when the watermark tripped. The block is a leftover, not a live reflection of disk usage right now.

The fix 1. Remove the read-only block manually (this is the actual unblock step)

curl -k -u <user>:<pass> -X PUT "https://<indexer-host>:9200/_all/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index.blocks.read_only_allow_delete": null
  }'

This clears the block cluster-wide. Specifically target .opendistro-ism-config too if you want to be surgical first:

curl -k -u <user>:<pass> -X PUT "https://<indexer-host>:9200/.opendistro-ism-config/_settings" \
  -H "Content-Type: application/json" \
  -d '{
    "index.blocks.read_only_allow_delete": null
  }'

2. Verify the block is actually cleared

curl -k -u <user>:<pass> "https://<indexer-host>:9200/.opendistro-ism-config/_settings?pretty"

Confirm read_only_allow_delete no longer appears (or shows false/absent).

Please share the requested outputs, and we will help you investigate further.  

[xeption]

Hello Himanshu,

Thank you for the response,

I have perform operations which you actually want please find the below:

1. Cluster health

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/health?pretty

2. Watermark settings

curl -k -u <user>:<password> https://<indexer>:9200/_cluster/settings?include_defaults=true&pretty

for this i have attached file.

3. Verify whether indices are still blocked

curl -k -u <user>:<password>  "https://<indexer>:9200/_all/_settings/index.blocks.read_only_allow_delete?pretty"

for this i find found nothing.

I followed steps which you given and i found some error and these operations i performed using admin user.
{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"},"status":403}

Regards,

Chandra

[Himanshu Sharma]

Hello Chandra,

Thank you for sharing the outputs.

From the cluster health output, I can see that the cluster is currently in a yellow state:

"status": "yellow",
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"unassigned_shards": 30

Since this is a single-node cluster, the yellow status is expected if replica shards cannot be allocated.

The 403 error — this is the important one for your original ISM problem

"reason":"no permissions for [] and User [name=admin, backend_roles=[admin], requestedTenant=null]"

This is not a normal RBAC permission gap — .opendistro-ism-config is a protected system index in the OpenSearch Security plugin. Even a user with the admin backend role is blocked from directly modifying system indices via the standard REST API + basic auth, by design — this is a hardening feature so that a compromised or misconfigured admin account can't tamper with security-critical internal indices.

To modify a protected system index, you need to use the actual TLS admin certificate:

System index protection requires authenticating with the admin client certificate configured in opensearch.yml under plugins.security.authcz.admin_dn, not just basic auth with an admin role. Find your admin cert/key (usually named something like admin.pem / admin-key.pem in /etc/wazuh-indexer/certs/ or similar):

curl --cacert /etc/wazuh-indexer/certs/root-ca.pem \
     --cert /etc/wazuh-indexer/certs/admin.pem \
     --key /etc/wazuh-indexer/certs/admin-key.pem \

     -X PUT "https://<indexer-host>:9200/.opendistro-ism-config/_settings" \
     -H "Content-Type: application/json" \
     -d '{ "index.blocks.read_only_allow_delete": null }'

This bypasses the basic-auth RBAC layer entirely and authenticates as the actual cluster admin identity trusted at the transport/TLS level — this is the account that's allowed to touch protected system indices.

Regards,

[xeption]

Hello Himanshu,

i will try this configuration and let you know its working or not.

Regards

[xeption]

Hello Himanshu,

I have tried which command you mentioned in previous mail but got some error.

1. Tried with server Private ip:

2.Tried with localhost:

Kindly review and suggest me how i can fix this.

Regards,

Chandra

[Himanshu Sharma]

Hi Team,

Thank you for sharing the screenshots.

Based on the errors shown, there are actually two different issues:

1. Using the private IP address

The error: curl: (7) Failed to connect to <IP> port 9200: Couldn't connect to server

indicates that nothing is listening on port 9200 for that IP address, or the connection is being blocked by a firewall/security group.

Please first verify that the Indexer service is running and listening on port 9200:

ss -tulpn | grep 9200

2. Using localhost

The error: curl: (60) SSL: no alternative certificate subject name matches target host name 'localhost'

indicates that the certificate was not issued for the hostname localhost. To bypass hostname verification for testing purposes, please use the -k option:

curl -k \

--cacert /etc/wazuh-indexer/certs/root-ca.pem \
--cert /etc/wazuh-indexer/certs/admin.pem \
--key /etc/wazuh-indexer/certs/admin-key.pem \

-X PUT "https://127.0.0.1:9200/.opendistro-ism-config/_settings" \

-H "Content-Type: application/json" \
-d '{
"index.blocks.read_only_allow_delete": null
}'

Alternatively, use the node hostname that matches the certificate Subject Alternative Name (SAN) instead of localhost.

Additional Validation

Please share the output of the following commands:

• systemctl status wazuh-indexer
• ss -tulpn | grep 9200
• curl -k \

• 
  --cacert /etc/wazuh-indexer/certs/root-ca.pem \
  --cert /etc/wazuh-indexer/certs/admin.pem \
  --key /etc/wazuh-indexer/certs/admin-key.pem \
  

• https://127.0.0.1:9200

This will help us confirm:

• Whether the Indexer service is running correctly.
• Which interface and hostname the service is listening on.
• Whether certificate authentication is functioning properly.

Once you share the outputs, we can provide the exact command required for your environment.

[xeption]

Hello Himanshu,

I have performed given steps as you suggest and attached screenshot. Kindly review.

1. ss -tulpn | grep 9200

2. Using localhost

3. Indexer status;

Please let me know if any information need.

Regards,

Chandra

[Himanshu Sharma]

Hello Chandra,

Thank you. This confirms the fix was applied successfully. The {"acknowledged":true} response means the read-only block has been cleared cluster-wide, and the indexer service itself is healthy and stable.

Next step — please verify ISM is now actively processing the retention policy:

curl -k --cacert /etc/wazuh-indexer/certs/root-ca.pem \
     --cert /etc/wazuh-indexer/certs/admin.pem \
     --key /etc/wazuh-indexer/certs/admin-key.pem \

     "https://127.0.0.1:9200/_plugins/_ism/explain/wazuh-alerts-*?pretty"

Please share the output specifically; look for each index's policy_id (should show alert-retention-policy) and whether it's actively stepping through states rather than stuck. This will confirm that indices older than 30 days are now being picked up for deletion on the next ISM check cycle.

If any older indices are still present after the next cycle completes, we can manually force a retry with:

curl -k --cacert /etc/wazuh-indexer/certs/root-ca.pem \
     --cert /etc/wazuh-indexer/certs/admin.pem \
     --key /etc/wazuh-indexer/certs/admin-key.pem \

     -X POST "https://127.0.0.1:9200/_plugins/_ism/retry/wazuh-alerts-*"

Regards,

[xeption]

Hello  Himanshu,

I have attached output of given command and old indices forcefully deleted from UI side.

Regards,

Chandra

[Himanshu Sharma]

Hello Chandra,

Thank you. This output confirms everything is working correctly now.

All 30 currently managed indices show step_status: "condition_not_met", which simply means ISM is actively evaluating each index against the 30-day retention rule, and none of the remaining indices have crossed that threshold yet — the oldest one (wazuh-alerts-4.x-2026.06.10) is exactly 30 days old as of today, so it's right at the edge and should transition to deletion on the next check cycle.

This is the expected, healthy state — not an error. The manual deletion of the older indices via the UI was a helpful one-time cleanup of the backlog that had built up while ISM was blocked; going forward, you shouldn't need to do that again, since ISM is now correctly tracking and will automatically delete each index once it passes 30 days old.

Please re-check the explain output tomorrow — you should see wazuh-alerts-4.x-2026.06.10 disappear from the list on its own (transitioning to the delete_alerts state and being removed), without any manual action needed. If that happens as expected, this issue is fully resolved.

Regards,

[xeption]

Hello Himanshu,

I can see that the indices in the Wazuh Indices tab are not being deleted automatically as expected, even though everything appears to be configured correctly in the screenshot I shared. Is there anything else I should check?

Regards,

[Himanshu Sharma]

Hello,

Thanks for the details "configured correctly" and "not deleting automatically" is a very common combination, and it's usually one of a handful of underlying causes. 

1. Confirm the index has actually crossed 30 days — ISM only evaluates on its scheduled cycle, not instantly at the threshold.
2. Check if the policy is actually attached to the index (not just defined):

   curl -k -u <user>:<pass> "https://<indexer-host>:9200/_plugins/_ism/explain/wazuh-alerts-*?pretty"

3. Check indexer resource usage (CPU/RAM/heap) — ISM runs as a background job and can be delayed or silently skipped if the indexer is under sustained CPU or memory pressure:

   top
   free -h
   curl -k -u <user>:<pass> "https://<indexer-host>:9200/_nodes/stats/os,jvm,process?pretty"

Could you share the explain output along with current CPU/RAM/heap usage? That should tell us whether this is a policy-attachment gap or resource starvation on the indexer.

Regards,

[xeption]

Hi Himanshu,

I have pasted the output of the given commands:

1. Confirm the index has actually crossed 30 days : yes its 34

2. Check if the policy is actually attached to the index:

3. Check indexer resource usage (CPU/RAM/heap):

top:

Free -h

 curl -k -u <user>:<pass> "https://<indexer-host>:9200/_nodes/stats/os,jvm,process?pretty" (JVM heap current memory 4GB)

    

if you need any information please feel free to ask.

Regards,

[Himanshu Sharma]

Hi Team,

Thanks for sending these over. Looking at the top, free -h, and _nodes/stats output:

• CPU is mostly idle (78.7% idle), load average is low
• JVM heap is committed at 4GB max, currently at 73% used
• OS-level memory shows 98% used; for that you can clear (buff/cache absorbs free RAM)

So resource exhaustion is not why the index isn't being deleted.

From the  ISM, explain the output (Image 1), it shows "step_status" : "condition_not_met" for the index 2026.06.20 so can you share the output for the 30-day-old index that should be deleted by policy?

curl -k -u <user>:<pass> "https://<indexer-host>:9200/_plugins/_ism/explain/wazuh-alerts-4.x-2026.06.10?pretty"

Can you share the actual policy definition? Run:

curl -k -u <user>:<pass> "https://<indexer-host>:9200/_plugins/_ism/policies/alert-retention-policy?pretty"

I want to see the exact min_index_age value (and units) configured on the transition condition under the initial state

  Once I see the policy JSON, I can pinpoint exactly which condition is blocking the transition.  

[xeption]

Hello Himanshu,

I have attached evidences.

1.  curl -k -u <user>:<pass> "https://<indexer-host>:9200/_plugins/_ism/explain/wazuh-alerts-4.x-2026.06.10?pretty"

2. curl -k -u <user>:<pass> "https://<indexer-host>:9200/_plugins/_ism/policies/alert-retention-policy?pretty"

Also attached json file.

Regards,

Chandra

[Himanshu Sharma]

Hi Team,

Thanks for the details. We found the root cause of the issue. Look at the version numbers:

Policy the index is actually running (from _ism/explain)
seq_no: 0 
primary_term: 1

Current live policy (from _ism/policies)

seq_no:  20399194
primary_term: 111

In OpenSearch ISM, every policy edit increments its seq_no/primary_term. But once an index starts being managed under a policy, ISM caches that specific version on the index — it does not automatically re-sync to newer versions of the policy just because you edited the policy document. The huge gap here (primary_term 1 → 111) tells us this policy has been edited roughly 110 times since this index was first attached, and this index has been silently running on the very first version ever created the entire time. 

This also explains the age math: Your index was created June 10, 2026 (index_creation_date: 1781049602336). Today is July 16, 2026 — that's 36 days old, well past the min_index_age: 30d condition in the current policy. Since the index isn't actually evaluating against the current policy version, whatever condition was baked into that ancient primary_term: 1 snapshot is what's silently controlling behavior — which is why condition_not_met keeps firing even though the current policy's 30-day threshold is clearly satisfied.  

The Fix — Force the Index to Re-sync to the Current Policy

Run the ISM Update Managed Index API against the affected indexes:

curl -k -u <user>:<pass> -X POST \

"https://<indexer-host>:9200/_plugins/_ism/update/wazuh-alerts-*?pretty" \ -H 'Content-Type: application/json' \ -d '{"policy_id": "alert-retention-policy"}'

This tells ISM: "drop your cached version, re-fetch the current policy document, and re-evaluate against it." After running this, re-check the explain endpoint — it should now show policy_seq_no: 20399194 and policy_primary_term: 111, and since the index is already 36 days old, it should transition to delete_alerts on its next ISM cycle (ISM runs on a scheduled interval, typically every few minutes by default).

In the future, whenever this policy is edited, the update call above should be run against wazuh-alerts-* to keep all managed indices in sync — otherwise this same drift will just recur silently.  

  Let me know once you've run this and I'll help interpret the next explain output if anything still looks off.  

[xeption]

Hello HImanshu,

I got an error.

[Himanshu Sharma]

Hi Team,

Following up on the retention policy issue — please try the approach below to force the index to re-sync with the current policy version.

Remove the policy, then re-apply it fresh:

# Step 1: Detach the index from ISM management curl -k -u <user>:<pass> -X POST \
  "https://<indexer-host>:9200/_plugins/_ism/remove/wazuh-alerts-4.x-2026.06.10?pretty" # Step 2: Re-attach it to the policy curl -k -u <user>:<pass> -X POST \
  "https://<indexer-host>:9200/_plugins/_ism/add/wazuh-alerts-4.x-2026.06.10?pretty" \

  -H 'Content-Type: application/json' \
  -d '{"policy_id": "alert-retention-policy"}'

This is a harder reset — it fully detaches and reattaches the index to the policy, so it picks up the current version with no old reference left behind.

Verify the fix (run after either approach)

curl -k -u <user>:<pass> \
  "https://<indexer-host>:9200/_plugins/_ism/explain/wazuh-alerts-4.x-2026.06.10?pretty"

Please confirm the output shows:

• policy_seq_no: 20399194
• policy_primary_term: 111
• state.name: "initial" with a fresh start_time

Since this index is already past the 30-day threshold, it should transition to delete_alerts on ISM's next scheduled evaluation cycle once it's synced correctly.

Best regards,

[xeption]

Hello Himanshu,

Issue is still  Persistence  , can we connect if you available?

Regards,

Chandra

[Himanshu Sharma]

Hi Team,

Could you please share the latest output of the explain command so I can see exactly where it's stuck now?

curl -k -u <user>:<pass> \
  "https://<indexer-host>:9200/_plugins/_ism/explain/wazuh-alerts-4.x-2026.06.10?pretty"

Specifically, I need to check:

1. policy_seq_no and policy_primary_term — to confirm whether the remove/re-apply steps actually synced the index to the current policy version (20399194 / 111), or if it's still showing old/stale values.
2. step.step_status — to see if it's still condition_not_met, or if it's changed to something else (e.g., a new error, or condition_met but stuck at the action stage).
3. state.start_time — to confirm whether this reset to a fresh timestamp after the remove/add, or if it's unchanged (which would suggest the remove/add didn't actually take effect).

Also, please confirm:

• Did both the remove and add curl commands return a success response (e.g., {"_index":...,"_id":...,"result":"updated"} or similar), or did either one return an error?
• Are the old indices still present and not deleted?

Once I see the fresh explain output, I can pinpoint whether this is still a sync issue, a different condition blocking the transition, or something at the ISM job scheduler level (e.g., the job interval not having run yet, or being disabled).

Best regards,

[xeption]

Hello Himanshu,

The issue is resolved now. Sorry, the mistake was on my side. I had applied the policy only to the latest alert indices, so it wasn't working as expected. The policy would have started working only after 30 days from the date it was applied to those latest indices, which is why the transition state was not working as we expected.

Later, I detached the policy and applied it to all the previous indices as well, and now it is working fine as expected.

Thanks for your support.

Regards,

Chandra