Re: Monthly Index Creation

[Sandra Ocando]

Hello Syed Ammar,

To create monthly indices you may modify /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json the following way:

      "date_index_name": {

        "field": "timestamp",

        "date_rounding": "m",

        "index_name_prefix": "{{fields.index_prefix}}",

        "index_name_format": "yyyy.MM",

        "ignore_failure": false

      }

Once modified, reload the pipeline to apply the changes: filebeat setup --pipelines

It's recommended to aim for shards sizes between 10 GB and 50 GB, so consider this when creating monthly/weekly indices: https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#shard-size-recommendation

Best regards,

Sandra.

On Friday, September 17, 2021 at 9:12:21 AM UTC+2 syedam...@gmail.com wrote:

Hi Team,
I want to create monthly indices(wazuh-alerts-4.x-2021-09) instead of daily (wazuh-alerts-4.x-2021-09) using filebeat. I am using wazuh and elastic clusters and filebeat is sending data to elastic search. Please also tell me, will decreasing the indices will increase the performance of elastic search, and what other techniques should I use to increase the performance of elastic, wazuh, etc.

Specs:

10 elastic nodes - 128GB ram
5 wazuh manager 
400 - 500 agents

OS: Ubuntu 

Opendistro Elastic

Kind regards.

Syed Ammar

[syed ammar]

Thank you for your response,

1) For weekly indices, what will I have to write in this field "index_name_format": "???". 

2) Is it possible to send data to specific indices through filebeat. For e.g send the application to app_indices and firewall data to firewall_indices etc??

Best regards,

Syed Ammar

[Sandra Ocando]

Hello Syed Ammar,

To create weekly indices you may use the following configuration:
    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "w",
        "index_name_prefix": "{{fields.index_prefix}}",
        "index_name_format": "YYYY.ww",
        "ignore_failure": false
      }
    },

Regarding your second question, yes, it's possible to create specific indices for different types of alerts, for example, syslog group. To do so, modify /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json and add a new index wazuh-alerts-4.x-syslog-* for alerts that contains syslog in rule.groups:

    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "w",
        "index_name_prefix": "{{fields.index_prefix}}syslog-",
        "index_name_format": "YYYY.ww",
        "ignore_failure": false,
        "if" : "if (ctx.rule.groups.contains('syslog')){return true;}"
      }
    },

After modifying the pipeline don't forget to reload it to apply the changes: filebeat setup --pipelines

Best regards,
Sandra.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/705f592b-0200-4399-a031-21c3fa70912bn%40googlegroups.com.

[syed ammar]

Thank You! It worked