ILM for Wazuh indexes

[jonatha...@a2secure.com]

Hi guys!

I have some doubt, Is it possible configure wazuh indexes with ILM?

Or is this feature is on roadmap?

Kind regards.

Jonathan A.

[Mayte Ariza]

Hi Jonathan,

 

Yes, you can set ILM for your Wazuh indices in Elasticsearch. The process will vary depending on whether you use standard Elastic or Open Distro.

 

You can follow this blog post to configure it and manage your Wazuh indices over time: https://wazuh.com/blog/wazuh-index-management/

 

Let me know if you have any questions.

 

Best regards,

Mayte Ariza

[jonatha...@a2secure.com]

Hi Mayte,

Thank so much for your comment.

Is it possible configure the ILM with the rollover? I can configure rollover using max size or age?

I have alot of small index less of 50M and I have problems of performance.

I would to rollover por example the index wazuh-monitoring with 30 days or 5GB.

Is it possible? Many thanks. 

[jonatha...@a2secure.com]

Hi,

I'm trying to use a ALIAS for the rollover index,

but is not working well.

Can you help me? Or Wazuh is not prepared for this actions?

Kind regards.

[Mayte Ariza]

Hi Jonathan,

 

If you are interested in rolling the wazuh-monitoring indices every 30 days, I think the easier solution is to configure the wazuh.monitoring.creation monthly instead of using ILM. You can set it up on Kibana: Wazuh > App settings > configuration  and then edit the value for wazuh.monitoring.creation to monthly instead of daily:

 

I would recommend setting ILM policies for wazuh-alerts indices though. Since wazuh-alerts indices are populated when sending alerts to Elasticsearch using Filebeat or Logstash, you can set ILM policies for wazuh-alerts indices as for any other handled by elastic. The following link may come in handy: https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html

 

I hope it helps.

 

Best regards,

Mayte Ariza

[jonatha...@a2secure.com]

Hi Mayte,

Thanks so much!! Solved the problem with index (wazuh-monitoring.)

Now, Can I do the same with wazuh-alerts? It's possible configure the ILM with the rollover?
Could I configure rollover using max size or age?

Kind regards.

[Mayte Ariza]

Hi  Jonathan,

 

Yes, you can automate rollover with ILM for wazuh-alerts indices The rollover may be performed taking into account the maximum age of the index, number of documents and/or index size. 

You can find more information about it here: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-rollover-index.html

 

However, ILM is an Elasticsearch X-Pack feature. X-Pack is installed by default when using Elasticsearch default distribution, but OSS distribution is not compatible with X-Pack. Which Elasticsearch distribution are you using?

 

You can check your Elasticsearch distribution running the following command on your Elasticsearch server:

curl -XGET http://localhost:9200

(you may need to change http to https or add your Elasticsearch credentials)

Please, keep us updated.

 

Best regards,

Mayte Ariza

[jonatha...@a2secure.com]

Hi Mayte,

Yes I know the ILM solution and I have other custom index with ILM,

but in this case, I don't know Where I need config the ALIAS.

I think I need configure on filebeat before to ingest on Elastic:

(filebeat.yml)

....
output.elasticsearch:

....

setup.ilm.enabled: true

setup.template.name:  "wazuh-alerts-ilm"

setup.template.pattern: "wazuh-alerts-*"

....

Do I need config this setup.ilm? 

Elastic Server:

{

  "name" : "node-1",

  "cluster_name" : "wazuh-cluster01",

  "cluster_uuid" : "6KHleQIBRtWRkY6iaqxhcA",

  "version" : {

    "number" : "7.7.1",

    "build_flavor" : "default",

    "build_type" : "rpm",

    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",

    "build_date" : "2020-05-28T16:30:01.040088Z",

    "build_snapshot" : false,

    "lucene_version" : "8.5.1",

    "minimum_wire_compatibility_version" : "6.8.0",

    "minimum_index_compatibility_version" : "6.0.0-beta1"

  },

  "tagline" : "You Know, for Search"

}

Kind regards.

[Mayte Ariza]

Hi Jonathan,

 

Sorry for the late response.

 

I have tested the ILM-rollover configuration using the Wazuh 3.13.1 version. Use wazuh-alerts-4.x instead of wazuh-alerts-3.x if you are using Wazuh 4.0 version.

 

1.  Create the ILM policy with rollover enabled

 

I created the ILM policy on Kibana following the Create a policy step explained here: https://wazuh.com/blog/wazuh-index-management/. However, in this case, we also enabled the rollover option:

 

 

Change the values to suit your needs

 

After creating the ILM policy, do not add the policy to an index template using Kibana and neither use the Index Management tool to add it to your indices. 

 

Instead, we are going to modify the Filebeat configuration files so we do not lose the setup when restarting Filebeat.

 

 2. Update the wazuh template

 

We need to add the following settings in the /etc/filebeat/wazuh-template.json file:

 

"index.lifecycle.name": "<policy_name>"
"index.lifecycle.rollover_alias": "<rollover_alias_name>"

In my case it looks like follows:

 

head -n 10 /etc/filebeat/wazuh-template.json

{
  "order": 0,
  "index_patterns": [
    "wazuh-alerts-3.x-*",
    "wazuh-archives-3.x-*"
  ],
  "settings": {
    "index.lifecycle.name": "wazuh-alert-retention-policy",
    "index.lifecycle.rollover_alias": "wazuh-alerts-3.x",

 

3. Modify the /etc/filebeat/filebeat.yml file, adding or updating the following lines:

setup.ilm.enabled: true
setup.ilm.policy_name: "<policy_name>"
setup.ilm.rollover_alias: "<rollover_alias_name>"

 

In my case the /etc/filebeat/filebeat.yml file looks like this:

# Wazuh - Filebeat configuration file
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false
 
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.template.overwrite: true
setup.ilm.enabled: true
setup.ilm.policy_name: "wazuh-alert-retention-policy"
setup.ilm.rollover_alias: "wazuh-alerts-3.x"
output.elasticsearch.hosts: ['http://localhost:9200']

 

4. Delete the following block from the /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json file:

 

    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },

 

5. Use the following command to reload the ingest pipeline

filebeat setup --pipelines --modules wazuh

 

6. Restart the Filebeat service

 

The ILM with rollover enabled is already configured. When new alerts are received, a new index with the following format <alias_name>-<yyyy.MM.dd>-000001 will be created.

For instance:

curl http://localhost:9200/_cat/indices/wazuh-alerts-3.x*?s=index

green open wazuh-alerts-3.x-2020.11.04        n1dAHY9-QTuLGsfX6HRHwA 3 0 9 0 88.2kb 88.2kb
green open wazuh-alerts-3.x-2020.11.04-000001 6kFCDr9sQa-luq3yrKYxwA 3 0 5 0 36.5kb 36.5kb

curl 'http://localhost:9200/_cat/aliases/wazuh-alerts*?v&pretty'

alias            index                              filter routing.index routing.search is_write_index
wazuh-alerts-3.x wazuh-alerts-3.x-2020.11.04-000001 -      -             -              true

This index will roll over to a new index based on the rollover conditions.

 

I hope it helps. Let me know if you have any questions.

 

Best regards,

Mayte Ariza

[jonatha...@a2secure.com]

Hi Mayte,

Thanks so much! Is working well on my Dev Cluster.

I will prepare to do changes on production :)

Kind regards.

[Mayte Ariza]

Hi Jonathan,

 

I'm glad it worked out! 

Do not hesitate to contact us again if you have any questions.

 

Best regards,

Mayte Ariza