what is filebeat index file

[Emil David]

Hi Team,

What is the purpose of the filebeat index file?

Generating everyday an index with name filebeat-7.17.6-2023.x.x (please find the attached screenshot also)

Eg:

filebeat-7.17.6-2023.10.31

This index is taking huge size, so please let me know how to stop/block this index creation.

Please find the attached screenshot.

[Ariel Ojeda]

Hi Emil,

I hope you are well today!

Regarding your question, this index shouldn't be generated by default according to the documentation. Could you please share the contents of the file  /etc/filebeat/filebeat.yml from every Wazuh Manager?

Please remember to obscure any sensitive information.

Ariel.

[Emil David]

Hi,

It is all in one deployment.

Ref link: https://documentation.wazuh.com/current/deployment-options/elastic-stack/all-in-one-deployment/index.html

root@xxxx-xxxx-xxxx:~# cat /etc/filebeat/filebeat.yml
# Wazuh - Filebeat configuration file
output.elasticsearch.hosts: ["127.0.0.1:9200"]
output.elasticsearch.password: ydsdsffrvvaawewefeNkC

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

setup.template.json.enabled: true
setup.template.json.path: /etc/filebeat/wazuh-template.json
setup.template.json.name: wazuh
setup.template.overwrite: true
setup.ilm.enabled: false

output.elasticsearch.protocol: https
output.elasticsearch.ssl.certificate: /etc/elasticsearch/certs/elasticsearch.crt
output.elasticsearch.ssl.key: /etc/elasticsearch/certs/elasticsearch.key
output.elasticsearch.ssl.certificate_authorities: /etc/elasticsearch/certs/ca/ca.crt
output.elasticsearch.ssl.verification_mode: strict
output.elasticsearch.username: elastic


logging.metrics.enabled: false

seccomp:
  default_action: allow
  syscalls:
  - action: allow
    names:
    - rseq

Note, that we need to get only selected group logs in index files, the remaining all logs need to be negated from filebeat ingest.

We made the below changes also in the configuration file.

In this configuration, have added group1,group2, and group3 configurations with if.

Output: The selected group logs are coming to the wazuh-alerts-4.x-group-2023.09.x, But the remaining logs are stored to the  filebeat-7.17.6-2023.x.x index.

Requirement: We need to get the selected group logs only in the index, remains can be skipped, no need to push to elastic, please help.

/usr/share/filebeat/module/wazuh/alerts/ingest# cat pipeline.json

{
  "description": "Wazuh alerts pipeline",
  "processors": [
    { "json" : { "field" : "message", "add_to_root": true } },
    {
      "geoip": {
        "field": "data.srcip",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.win.eventdata.ipAddress",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.aws.sourceIPAddress",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.gcp.jsonPayload.sourceIP",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.office365.ClientIP",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "date": {
        "field": "timestamp",
        "target_field": "@timestamp",
        "formats": ["ISO8601"],
        "ignore_failure": false
      }
    },
    {
      "date_index_name": {
        "if" : "if (ctx.rule.groups.contains('group1')){return true;}",
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}group-",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },
    {
      "date_index_name": {
        "if" : "if (ctx.rule.groups.contains(' group2')){return true;}",
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}group-",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },
    {
      "date_index_name": {
        "if" : "if (ctx.rule.groups.contains(' group3')){return true;}",
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}group-",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },

    { "remove": { "field": "message", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "ecs", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "beat", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "input_type", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "tags", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "count", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "@version", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "log", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "offset", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "type", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "host", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "fields", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "event", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "fileset", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "service", "ignore_missing": true, "ignore_failure": true } }
  ],
  "on_failure" : [{
    "drop" : { }
  }]
}

[Ariel Ojeda]

Hi,

I am looking at this, but in the meantime, if you don't need these indices you mentioned, you can create a policy to have them deleted automatically a day after they were created.

https://wazuh.com/blog/wazuh-index-management/

Could you please share the configuration steps you went through in this installation to achieve your goal?

Ariel.

[Ariel Ojeda]

Also, as mentioned before, please remember to obscure any sensitive information such as passwords as this is a public group and anyone can see that information.