Re: Monthly Index Creation

55 views
Skip to first unread message
Message has been deleted

Sandra Ocando

unread,
Sep 17, 2021, 5:37:11 AMSep 17
to Wazuh mailing list
Hello Syed Ammar,


To create monthly indices you may modify /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json the following way:

      "date_index_name": {

        "field": "timestamp",

        "date_rounding": "m",

        "index_name_prefix": "{{fields.index_prefix}}",

        "index_name_format": "yyyy.MM",

        "ignore_failure": false

      }

Once modified, reload the pipeline to apply the changes: filebeat setup --pipelines

It's recommended to aim for shards sizes between 10 GB and 50 GB, so consider this when creating monthly/weekly indices: 
https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#shard-size-recommendation

Best regards,

Sandra.


On Friday, September 17, 2021 at 9:12:21 AM UTC+2 syedam...@gmail.com wrote:
Hi Team,
I want to create monthly indices(wazuh-alerts-4.x-2021-09) instead of daily (wazuh-alerts-4.x-2021-09) using filebeat. I am using wazuh and elastic clusters and filebeat is sending data to elastic search. Please also tell me, will decreasing the indices will increase the performance of elastic search, and what other techniques should I use to increase the performance of elastic, wazuh, etc.
Specs:
10 elastic nodes - 128GB ram
5 wazuh manager 
400 - 500 agents
OS: Ubuntu 
Opendistro Elastic

Kind regards.
Syed Ammar

syed ammar

unread,
Sep 17, 2021, 6:32:42 AMSep 17
to Wazuh mailing list
Thank you for your response,

1) For weekly indices, what will I have to write in this field "index_name_format": "???". 
2) Is it possible to send data to specific indices through filebeat. For e.g send the application to app_indices and firewall data to firewall_indices etc??

Best regards,
Syed Ammar


Sandra Ocando

unread,
Sep 20, 2021, 5:48:57 AMSep 20
to syed ammar, Wazuh mailing list
Hello Syed Ammar,

To create weekly indices you may use the following configuration:
    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "w",
        "index_name_prefix": "{{fields.index_prefix}}",
        "index_name_format": "YYYY.ww",
        "ignore_failure": false
      }
    },

Regarding your second question, yes, it's possible to create specific indices for different types of alerts, for example, syslog group. To do so, modify /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json and add a new index wazuh-alerts-4.x-syslog-* for alerts that contains syslog in rule.groups:

    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "w",
        "index_name_prefix": "{{fields.index_prefix}}syslog-",
        "index_name_format": "YYYY.ww",
        "ignore_failure": false,
        "if" : "if (ctx.rule.groups.contains('syslog')){return true;}"
      }
    },

After modifying the pipeline don't forget to reload it to apply the changes: filebeat setup --pipelines

Best regards,
Sandra.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/705f592b-0200-4399-a031-21c3fa70912bn%40googlegroups.com.

syed ammar

unread,
Sep 21, 2021, 1:05:04 AMSep 21
to Wazuh mailing list
Thank You! It worked
Reply all
Reply to author
Forward
0 new messages