Set up hot-warm architecture

[Thaynara Soares]

I would like help implementing the hot-warm architecture in my infrastructure

My Environment

-Two Indexers
-Wazuh 4.9.2

-I checked the documentation https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html, and I wanted help with the node configuration part where I would put node.attr.temp: hot in the file /etc/wazuh-indexer/opensearch.yml

[Diego Andrés Cappri]

Hi Thaynara,

apologies for the delay. You can place this line at the bottom of the file if you want, but to keep the structure organized it's recommended to add it to the "node configuration section" where node.roles is placed.

Hope this help.

[Thaynara Soares]

Another question, do I need to inform which index will be hot and which will be warm? Or do I configure both? I have two indexes in my environment.

[Thaynara Soares]

I wanted to test it first in my homologation environment, but it is only one machine for the index, master and work. It is not in a cluster, as I would test the hot-warm architecture, or is it only done in a cluster?

[Diego Andrés Cappri]

Hi,

you can add these lines in your .yaml:

node.name: "wazuh-indexer-homologation"
node.roles: [data, ingest, master]
node.attr.temp: hot_warm
cluster.routing.allocation.awareness.attributes: temp

Then restart the wazuh-indexer and check for the attribute is applied -> curl -X GET "http://localhost:9200/_nodes?pretty" | grep -A5 "node.attr.temp"

Hope this help

[Thaynara Soares]

After adding it, do I put the policy in and apply it to the indexes?

- Policy

PUT _plugins/_ism/policies/hot_warm
{
    "policy": {
        "description": "Send shards from hot to warm nodes",
        "schema_version": 17,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "replica_count": {
                            "number_of_replicas": 0
                        }
                    },
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "allocation": {
                            "require": {
                                "temp": "warm"
                            },
                            "include": {},
                            "exclude": {},
                            "wait_for": false
                        }
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "wazuh-alerts-*"
                ],
                "priority": 1
            }
        ]
    }
}

[Diego Andrés Cappri]

Hi, you will need to create Index Lifecycle Management:

PUT _ilm/policy/wazuh-hot-warm-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "10gb"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "require": {
              "temp": "hot_warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      }
    }
  }
}

Then apply it:
PUT _index_template/wazuh-hot-warm-template
{
  "index_patterns": ["wazuh-alerts-*"],
  "template": {
    "settings": {
      "index.lifecycle.name": "wazuh-hot-warm-policy"
    }
  }
}

[Thaynara Soares]

Not sure, is that how you add it?

[Diego Andrés Cappri]

Run the following command on your Wazuh Indexer server:

curl -X PUT "http://localhost:9200/_ilm/policy/wazuh-hot-warm-policy" -H "Content-Type: application/json" -d '

{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "7d",
            "max_size": "10gb"
          }
        }
      },
      "warm": {
        "actions": {
          "allocate": {
            "require": {
              "temp": "hot_warm"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          }
        }
      }
    }
  }

}'

[Thaynara Soares]

[Diego Andrés Cappri]

Hi, please restart wazuh-indexer service, the try this command to check -> curl -X GET "http://localhost:9200/_ilm/policy/wazuh-hot-warm-policy?pretty"

[Thaynara Soares]

[Diego Andrés Cappri]

Is there any error log you can share? Is the wazuh-indexer starting ok? Please check if it's listening : curl -X GET "http://localhost:9200/_ilm/policy?pretty" -u <user>:<passwd>

[Thaynara Soares]

It gives the same error (curl: (52) Empty reply from server), I think my homologation environment needs resources, I would have to test it in the production environment.

-My production environment is in a cluster, could you help me? I have 2 Indexes

[Diego Andrés Cappri]

Hi, here's the documentation for this: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/index-life-management.html. As mentioned before, you can place this line at the bottom of the file if you want, but to keep the structure organized it's recommended to add it to the "node configuration section" where node.roles is placed.

[Thaynara Soares]

Add to:

-Index01

node.attr.temp: hot

-Index02

node.attr.temp: warm 

That way ?

[Thaynara Soares]

My setup:

configuration  nano  /etc/wazuh-indexer/opensearch.yml

-I added the Test Policy for 1 day

{
    "policy": {
        "description": "Send shards from hot to warm nodes",
        "schema_version": 17,
        "error_notification": null,
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "warm",
                        "conditions": {

                            "min_index_age": "1d"

-I applied the Policy to the test indexers

-wazuh-alerts-4.x-2024.10.03

-wazuh-alerts-4.x-2024.10.02

-Is it correct?

[Thaynara Soares]

-It ended up like this, is that correct? If I want to access this data on the panel, can I? Or do I need to remove it from warm?

[Thaynara Soares]

Does this architecture help to save disk space?

[Diego Andrés Cappri]

Hi, Hot-Warm architecture doesn’t directly reduce disk space but enhances storage efficiency by leveraging forcemerge, automated deletions, and tiered storage. You will need to take a look to ILM policies to manage disk usage. 

[Thaynara Soares]

Is there any way to compress old data to free up more space without having to delete it?

[Thaynara Soares]

How do I create a policy for this, with Index01 being the hot node and Index02 the cold node, and so that the data on Index02 is compact and I can then decompress it and it is not deleted?

[Diego Andrés Cappri]

Please try creating the policy

PUT _ilm/policy/wazuh-hot-cold-policy
{
  "policy": {
    "phases": {
      "hot": { "actions": { "rollover": { "max_age": "30d", "max_size": "50gb" } } },
      "cold": {
        "actions": {
          "allocate": { "require": { "temp": "cold" } },
          "forcemerge": { "max_num_segments": 1 },
          "set_priority": { "priority": 0 }
        }
      }
    }
  }
}


then assign the polixy to indeces

PUT _index_template/wazuh-hot-cold-template
{
  "index_patterns": ["wazuh-alerts-*"],
  "template": { "settings": { "index.lifecycle.name": "wazuh-hot-cold-policy" } }
}

check the status

GET _ilm/policy/wazuh-hot-cold-policy?pretty
GET _ilm/explain/wazuh-alerts-2024.01.01
GET _cat/indices?v

decompress data when required

POST wazuh-alerts-2024.01.01/_refresh
POST wazuh-alerts-2024.01.01/_forcemerge?max_num_segments=5


it is possible that you need to adapt/modify/troubleshoot in your environment to reach the final result, please let me know

[Thaynara Soares]

You need to make some changes to the nano /etc/wazuh-indexer/opensearch.yml file

[Thaynara Soares]

Because I want Index01 as a hot node and Index02 as a cold node, Adding just this policy will free up space in Index 02. As cold and I want it to free up space in Index01 because those in Index02 I don't want it to stay in Index01

[Diego Andrés Cappri]

Hi, please check this:

PUT _ilm/policy/wazuh-hot-cold-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age": "30d",
            "max_size": "50gb"
          },

          "shrink": {
            "number_of_shards": 1

          }
        }
      },
      "cold": {
        "actions": {
          "allocate": {
            "require": {
              "temp": "cold"
            }
          },
          "forcemerge": {
            "max_num_segments": 1
          },
          "set_priority": {
            "priority": 0
          }
        }
      }
    }
  }
}

And apply the policy to indices:

[Thaynara Soares]

It didn't work

[Diego Andrés Cappri]

Hi. Ensure the field names (policy, phases, actions, etc.) are all lowercase and properly nested, make sure the "phases" key is inside the "policy" object and each phase (hot, cold) must have an actions object.

[Thaynara Soares]

It's not working, could you show me the correct way?

[Diego Andrés Cappri]

Hi, sorry for the delayed response. Could you please for OpenSearch version doing: curl -X GET "http://localhost:9200/" and try the following command to simulate the policy and identify the issue:

PUT _ilm/policy/wazuh-hot-cold-policy?error_trace=true

[Thaynara Soares]

These commands requested by you do not work on my server.

[Diego Andrés Cappri]

Is there an error you can share?

[Thaynara Soares]

curl: (52) Empty reply from server

[Nicolas Zapata]

Hi Thaynara!

The error curl: (52) Empty reply from server typically indicates that the server closed the connection without sending a response.
Can you please check the service status using the below command:
systemctl status wazuh-indexer