Open Distro - Index backup management

[Eric]

Hi guys, 

We currently have 1TB of data on Elastic Stack with one year of use, the data increase quickly; therefore, I'm looking for a solution for index backup on the OpenDistro. My goal is, after 30 days, to move the index to a "recent" state that performs a snapshot on the index and sets it as read-only. After 90 days, it's moved to S3 Storage. 

Has anyone here has experience with this case? Can you please share with me an idea to do?

[elw...@wazuh.com]

Hello Vuvanthancnc,

Assuming that you have two repositories ( SnapshotRepo1 & SnapshotAWSrepo https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-register-repository.html#snapshots-register-repository), the ISM policy would be (adding a phase to delete indices after 120 days) :

{
    "policy": {
        "policy_id": "Vuvanthancnc ISM policy ",
        "description": "my policy",
        "schema_version": 1,
        "default_state": "hot",
        "states": [{
                "name": "hot",
                "actions": [{
                    "replica_count": {
                        "number_of_replicas": 0
                    }
                }],
                "transitions": [{
                    "state_name": "snapshots30days",
                    "conditions": {
                        "min_index_age": "30d"
                    }
                }]
            },
            {
                "name": "snapshots30days",
                "actions": [{
                        "snapshot": {
                            "repository": "SnapshotRepo1",
                            "snapshot": "wazuh-alerts-snapshot"
                        }
                    },
                    {
                        "read_only": {}
                    }
                ],
                "transitions": [{
                    "state_name": "S3Storage",
                    "conditions": {
                        "min_index_age": "90d"
                    }
                }]
            },
            {
                "name": "S3Storage",
                "actions": [{
                    "snapshot": {
                        "repository": "SnapshotAWSrepo",
                        "snapshot": "wazuh-alerts-AWS-snapshot"
                    }
                }],
                "transitions": [{
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "120d"
                    }
                }]
            },
            {
                "name": "delete",
                "actions": [{
                    "delete": {}
                }],
                "transitions": []
            }
        ]
    }
}


Make sure to specify the policy in the Wazuh template and load it into Elasticsearch:

sed -i 's/  "settings": {/  "settings": {\n    "opendistro.index_state_management.policy_id": "Vuvanthancnc ISM policy",/g' /etc/filebeat/wazuh-template.json
filebeat setup --index-management

Please note that ISM in Opendistro will not take care of deleting the snapshots and it should be done manually or using a cronjob script.

Hope it helps.

Regards,
Wali