exceeded flood-stage watermark

[Manuel Christen]

hey folks,

Im a happy administrator with a wazuh 4.10.1 installation. The indexer and the Manager are two different systems.

Via the Dashboard i try to create a new internal User and i have the Errormessage "

Update error

{"status":"INTERNAL_SERVER_ERROR","message":"Error index [.opendistro_security] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];"}"

OK my disk has a usage of 90%, so I reconfigured the limits. 

Funfact: 10% are 376GB free diskspace of the Partition.

curl -k -u admin:password -XPUT "https://wazuh-indexer:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.watermark.low": "200gb", "cluster.routing.allocation.disk.watermark.high": "50gb", "cluster.routing.allocation.disk.watermark.flood_stage": "10gb", "cluster.info.update.interval": "1m" } } '
{
  "acknowledged" : true,
  "persistent" : { },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "disk" : {
            "watermark" : {
              "low" : "200gb",
              "flood_stage" : "10gb",
              "high" : "50gb"
            }
          }
        }
      },
      "info" : {
        "update" : {
          "interval" : "1m"
        }
      }
    }
  }
}

and the wazuh-cluster.log is happy:

[2025-11-12T14:09:54,636][INFO ][o.o.c.s.ClusterSettings  ] [wazuh-indexer] updating [cluster.info.update.interval] from [30s] to [1m]
[2025-11-12T14:09:54,636][INFO ][o.o.c.s.ClusterSettings  ] [ wazuh-indexer ] updating [cluster.routing.allocation.disk.watermark.low] from [85%] to [200gb]
[2025-11-12T14:09:54,636][INFO ][o.o.c.s.ClusterSettings  ] [ wazuh-indexer ] updating [cluster.routing.allocation.disk.watermark.high] from [90%] to [50gb]
[2025-11-12T14:09:54,636][INFO ][o.o.c.s.ClusterSettings  ] [ wazuh-indexer ] updating [cluster.routing.allocation.disk.watermark.flood_stage] from [95%] to [10gb]

and 2nd, after my tests are failed, i added additionally

curl -k -u admin:password -XPUT "https://wazuh-indexer:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "transient": { "cluster.routing.allocation.disk.threshold_enabled": false } } '

but I found no log output. Then i restarted the indexer and im still waiting. 

What im missing? What im doing wrong? Or i have a missunderstading of the errormessage?

Regards,

Manuel

[J. Rome]

Hi Manuel,

After changing this settings you still get an error when trying to create a new internal user, or you don't even get that error message?

[J. Rome]

An option is to free up disk space by deleting old indices, you can do this from the wazuh dashboard, in the Index management menu, inside Indixes, you can look for old (big) wazuh-alerts indices to remove.

Just to be on the safe side, can you grep your wazuh (server, indexer, dashboard) logs for error messages?