Is there a way to prevent the flood-stage disk

452 views
Skip to first unread message

Maria Juárez

unread,
Jan 27, 2023, 11:24:01 AM1/27/23
to Wazuh mailing list
Last week an incident occurred to me, apparently, the disk space was filled due to the number of generated indexes that were increasing daily, reaching an index blockage. I solved it by manually deleting old indexes but my question is, is there any method I can implement to my Wazuh server to prevent this from happening again? Maybe some configuration that automatically deletes or (preferably) closes old indexes from time to time.
Thank youu!.

The error I had last week was:
FATAL Error: Unable to complete saved object migrations for the [.kibana_task_manager] index. Please check the health of your Elasticsearch cluster and try again. Unexpected Elasticsearch ResponseError: statusCode: 429, method: PUT, url: /.kibana_task_manager_7.17.6_001/_mapping?timeout=60s error: [cluster_block_exception]: index blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];,

Eduardo Leon Aldazoro

unread,
Jan 27, 2023, 1:00:41 PM1/27/23
to Wazuh mailing list
Hi, Thanks for using Wazuh!

You can create an index state policy to manage ElasticSearch Indexes

To create an index state policy follow these steps:

- Go to Index Management.
- Create policy
- Provide a policy ID name.
- Create your Policy Json. Example below.
- Click Create

Policy Example:
{
  "policy": {
    "description": "Wazuh index state management for OpenDistro to move indices into a cold state after 30 days and delete them after 90 days.",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [
          {
            "replica_count": {
              "number_of_replicas": 1
            }
          }
        ],
        "transitions": [
          {
            "state_name": "cold",
            "conditions": {
              "min_index_age": "30d"
            }
          }
        ]
      },
      {
        "name": "cold",
        "actions": [
          {
            "read_only": {}
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "90d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": [
        "wazuh-alerts*"
      ],
      "priority": 100
    }
  }
}

The previous policy defines the following states:
  • Hot state. It sets 1 replica for the indices and a transition to the cold state when indices are older than 30 days.
  • Cold state. It sets indices into read-only mode and a transition to the delete state when indices are older than 90 days.
  • Delete state. Indices in this state are deleted.
For already existing indices you can use an API call or the Index Management running the following command.

POST _opendistro/_ism/add/wazuh-alerts-* { "policy_id": "NAME_OF_YOUR_POLICY" }

Here you can check a Wazuh blog about wazuh index management: https://wazuh.com/blog/wazuh-index-management/

Hope this answers your question.

Feel free to ask if you need more help.

Best Regards.
Reply all
Reply to author
Forward
0 new messages