filebeat with ILM

[Andrew Wason]

Is it possible to use ILM on the wazuh elastic indexes created via filebeat?

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-index-lifecycle-management.html

The wazuh filebeat config sets setup.ilm.enabled: false

https://github.com/wazuh/wazuh/blob/master/extensions/filebeat/7.x/filebeat.yml#L13

We have been using curator to delete old indices, but ILM seems simpler.

https://github.com/elastic/curator

The wazuh modules ingest pipelines use the date_index_name processor so the index name format isn't compatible with ILM.

https://github.com/wazuh/wazuh/blob/master/extensions/filebeat/7.x/wazuh-module/alerts/ingest/pipeline.json#L41

Any plans to support ILM, or how do people typically manage elastic indices, just continue using curator?

[Javier Escobar]

Hi Andrew,

We can help you out here. Which version of Elastic Stack and Wazuh are you using?

Regards, 

Javier

[Andrew Wason]

On Tuesday, August 27, 2019 at 10:52:34 AM UTC-4, Javier Escobar wrote:

Hi Andrew,

We can help you out here. Which version of Elastic Stack and Wazuh are you using?

 

elasticsearch 7.3.0 and wazuh 3.9.5

[Javier Escobar]

Hi Andrew, sorry for the late response.

After looking in your case and testing it I came up with this solution that may help you. 

You are right about the ingest pipeline not being compatible with ILM but only has conflicts with the alias and rollover feature for Hot indices. But it is possible to configure ILM for Warm, Cold and Deletion phases using ILM.

These are the steps to configure a policy to delete your indices after 180 days:

    -Go to Kibana->Management->Index Lifecycle Policies.

    -Create a new ILM policy as it follows:

As you can see by disabling rollover, you can configure the rest of ILM phases but in this case, it's going to use the "index creation date" as a reference instead of "days from rollover".

You can click on 'Show JSON' to obtain the policy in JSON format:

I also recommend using Filebeat to upload and manage your policy.

Follow these instructions to configure Filebeat to upload your ILM policy and Wazuh template:

-Edit the template /etc/filebeat/wazuh-template.json settings section. Add the next code:

   

{
 "order": 0,

  "index_patterns": [

    "wazuh-alerts-3.x-*",

    "wazuh-archives-3.x-*"

  ],

  "settings": {

    "index.refresh_interval": "5s",

    "index.number_of_shards": "3",

    "index.number_of_replicas": "0",

    "index.lifecycle.name": "retention_policy",

    "index.auto_expand_replicas": "0-1",

    "index.mapping.total_fields.limit": 10000,

    "index.query.default_field": [

    ...
    ...
}

-Edit /etc/filebeat/filebeat.yml with the code:

# Wazuh - Filebeat configuration file

filebeat.modules:

  - module: wazuh

    alerts:

      enabled: true

    archives:

      enabled: false

setup.template.json.enabled: true

setup.template.json.path: '/etc/filebeat/wazuh-template.json'

setup.template.json.name: 'wazuh'

setup.template.overwrite: true

setup.ilm.enabled: true

setup.ilm.policy_name: 'retention_policy'

setup.ilm.policy_file: '/etc/filebeat/policy.json'

# Send events directly to Elasticsearch

output.elasticsearch:

  hosts: ['http://ELASTICSEARCH_IP:9200']

-Restart Filebeat to apply the changes and force it to refresh your template and policy:

filebeat setup --index-management -E setup.template.json.enabled=false
systemctl restart filebeat

You can keep track of your indices ILM at Kibana->Management->Index Management.

Be aware that the ILM policy will apply only to newly created indices.

To apply your ILM to already existing indices use:

curl -X PUT "localhost:9200/wazuh-alerts-3.x-2019.08.20/_settings?pretty" -H 'Content-Type: application/json' -d'

{

    "index" : {

        "lifecycle.name" : "retention_policy"

    }

}

'

I hope it helps.

Don't hesitate to reach us if you have any further questions.

Regards,

Javier

[Andrew Wason]

On Thursday, August 29, 2019 at 11:28:56 AM UTC-4, Javier Escobar wrote:

Hi Andrew, sorry for the late response.

After looking in your case and testing it I came up with this solution that may help you. 

Thanks. I tried this and see this policy being applied to new wazuh-alerts indexes, but it also seems to be getting applied to new filebeat indices (e.g. filebeat-7.3.0-2019.09.05-000004)

We are using filebeat with ILM to send logs to the same ES cluster that wazuh is using. So it looks like the wazuh policy is overriding our existing ILM policy that filebeat is using.

[Javier Escobar]

Hi again Andrew, sorry for the late response.

Filebeat shouldn't behave like that. Could you please share with us your Filebeat configuration at /etc/filebeat/filebeat.yml?

Also, go to Kibana->Dev tools and copy the output of the template of Filebeat:

GET _template/filebeat-7.3.0

Replace "filebeat-7.3.0" with your current Filebeat version.

Regards,

Javier