Wazuh ILM with rollover failing

[Mark Thomassen]

Hi there,

I am configuring a Wazuh distributed solution, and I am getting stuck with successfully assigning an ILM to the wazuh-alerts and wazuh-archives indicies.

Below is the policy. When the primary shard size hits 30gb rollover using alias 'wazuh-alias".

When this policy gets applied to new or existing indicies I receive this error.

{ "message": "Missing rollover_alias index setting [index=wazuh-archives-4.x-2024.04.22]" }

Under "Aliases" i did create "wazuh-alias" assigning it to the existing indicies prior to the policy applying although the same error shows.

Any ideas.

"{

"id": "hot_cold_delete_workflow", "seqNo": 68273, "primaryTerm": 4, "policy": { "policy_id": "hot_cold_delete_workflow", "description": "Hot data will be searchable for 90 days\nCold data will be set to read only for the remainder until 365 days after which it will be deleted.\nPrimary shards will rollover once 30gb is reached.\nIndex alias added for rolling over of shards.", "last_updated_time": 1713740714236, "schema_version": 18, "error_notification": null, "default_state": "hot", "states": [ { "name": "hot", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "rollover": { "min_primary_shard_size": "30gb" } }, { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "alias": { "actions": [ { "add": { "aliases": [ "wazuh-alias" ] } } ] } } ], "transitions": [ { "state_name": "cold", "conditions": { "min_rollover_age": "90d" } } ] }, { "name": "cold", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "read_only": {} } ], "transitions": [ { "state_name": "delete", "conditions": { "min_rollover_age": "365d" } } ] }, { "name": "delete", "actions": [ { "retry": { "count": 3, "backoff": "exponential", "delay": "1m" }, "delete": {} } ], "transitions": [] } ], "ism_template": [ { "index_patterns": [ "wazuh-alerts*", "wazuh-archives*" ], "priority": 100, "last_updated_time": 1713740386329 } ] } }"

[Franco Giovanolli]

Hi Mark!

I think that issue was that the indices needed to be created after the ism policy and template.

In your case it appears that the policy was not correctly applying to your indices which is likely a result of you creating your indices before the policy or template were created.

To add the policy to your indexes you can run something like:

POST _plugins/_ism/add/my-index
{
  "policy_id": "my-policy-id"
}

Please, let me know if this helps.

Regards,
Franco.

[Mark Thomassen]

Hi Franco,

Indicie creation, correct me if I am wrong is preconfigured. Assigning the shard and replica values are preset which makes this more difficult since you should be allocating the alias when the index is created.

I executed via dev tools

POST _plugins/_ism/add/wazuh-alerts*
{
  "policy_id": "hot_cold_delete_workflow"
}

Currently in a running state although this is just assigning the policy again to the same existing indicies.

How can I modify the already created wazuh-alerts index to include the alias?

[Mark Thomassen]

Okay, I have not fixed this although this is what I have come across so far.

 

The index that is created (alerts, archive etc) needs to have an alias bootstrapped with write index enabled. Then you create your ism policy for those indicies to include the alias

Someone at Wazuh throught it was a good idea to have this configured out of the box due to complaints from other users about their inidices not being managed correctly.

Currently on pause.

Index State Management default policy · Issue #18999 · wazuh/wazuh · GitHub

I have tried creating a new indicie bootstrapping it with the alias which seemed to work although it fails when transitioning the rollover to an incremented new indicie since the indicie i created is missing.

index.plugins.index_state_management.rollover_alias

[Franco Giovanolli]

Hi Mark,

Yes, the approach you mention of incorporating the ILM natively via filebeat is an option that will be reviewed for Wazuh 5 onwards.

I apologize because it is not clear to me what the origin of the problem you are having is.

If you manually create and apply the ISM policy as indicated here https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html, do you have the same problem?

Regards,
Franco.

[Mark Thomassen]

Hi Franco,

Your documentation does not illustrate rolling over indices.

What I am getting stuck on is supporting rolling over of the alerts and archive indicies.

I have created two additional entries in the existing wazuh-templates.json file

{
  "order": 2,
  "index_patterns": [
    "wazuh-alerts-*"
  ],
  "settings": {
    "index.plugins.index_state_management.rollover_alias": "wazuh-alerts"
  },
   "order": 1,
  "index_patterns": [
    "wazuh-archives-*"
  ],
  "settings": {
    "index.plugins.index_state_management.rollover_alias": "wazuh-archives"
  },
  "order": 0,
  "index_patterns": [
    "wazuh-alerts-4.x-*",
    "wazuh-archives-4.x-*"
  ],

 Although I can tell that the management.rollover.alias is not being applied.

GET wazuh-archives*/_settings

{
  "wazuh-archives-4.x-2024.04.26-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "10000"
          }
        },
        "refresh_interval": "5s",
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1",
        "provided_name": "<wazuh-archives-4.x-{now/d}-000001>",
        "query": {

I have to manually run this although it only applies to the current indicies, not to any newly created indicies which should happen in the template file.

PUT /wazuh-archives*/_settings
{
     "index.plugins.index_state_management.rollover_alias" : "wazuh-archives"
}

Index State Management default policy · Issue #18999 · wazuh/wazuh · GitHub

"Setting rollover alias to the indices

The wazuh-template.json applies to the wazuh-alerts and wazuh-archives indices, so the only way to apply an individual alias for each one is to create 2 new index templates only to add the alias, acting on these indices separately and with a higher order than the base wazuh-template.

{ "order": 3, "index_patterns": ["wazuh-alerts-*"], "settings": { "index.plugins.index_state_management.rollover_alias": "wazuh-alerts" } }
{ "order": 3, "index_patterns": ["wazuh-archives-*"], "settings": { "index.plugins.index_state_management.rollover_alias": "wazuh-archives" } }"

[Franco Giovanolli]

Hi Mark,
I understand. With this change have you been able to solve your problem?