Pipeline help

[Serano Tomas]

Hello,

I have a delicate problem, and I would greatly appreciate your help
I have to ingest fulllog only for certain agents,

I made the below changes , it creates indexes correctly " wazuh-archives-4.x-grupA-2023.08.25" and "wazuh-archives-4.x-grupB-2023.08.25"
But additionally it creates an index: filebeat-7.10.2-2023.08.25
which contains all received logs

My question is, where am I going wrong, how can I stop ingesting these logs in the filebeat-7.10.2* index?

It takes up a lot of space with some logs that we don't need

---------------------------------------------
step 1 -
---------------------------------------------
in /var/ossec/etc/ossec.conf are enabled: <logall_json>yes</logall_json>
---------------------------------------------
step 2 - I created 2 groups where I added the agents for which ingestion is desired and I created labels for each grup
---------------------------------------------
<agent_config>
<labels>
<label key="grup">grupA</label>
</labels>
</agent_config>

and

<agent_config>
<labels>
<label key="grup">grupB</label>
</labels>
</agent_config>

---------------------------------------------
step 3 - in /usr/share/filebeat/module/wazuh/archives/manifest.yml
---------------------------------------------
module_version: 0.1

var:
  - name: paths
    default:
      - /var/ossec/logs/archives/archives.json
  - name: index_prefix
    default: wazuh-archives-4.x-

  - name: paths
    default:
      - /var/ossec/logs/archives/archives.json
  - name: grupA-
    default: wazuh-archives-4.x-grupA-

  - name: paths
    default:
      - /var/ossec/logs/archives/archives.json
  - name: grupB-
    default: wazuh-archives-4.x-grupB-


input: config/archives.yml

ingest_pipeline: ingest/pipeline.json

---------------------------------------------
step 4
---------------------------------------------
{
  "description": "Wazuh events pipeline",
  "processors": [
    { "json" : { "field" : "message", "add_to_root": true } },
    {
      "geoip": {
        "field": "data.srcip",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.win.eventdata.ipAddress",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.aws.sourceIPAddress",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.gcp.jsonPayload.sourceIP",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "geoip": {
        "field": "data.office365.ClientIP",
        "target_field": "GeoLocation",
        "properties": ["city_name", "country_name", "region_name", "location"],
        "ignore_missing": true,
        "ignore_failure": true
      }
    },
    {
      "date": {
        "field": "timestamp",
        "target_field": "@timestamp",
        "formats": ["ISO8601"],
        "ignore_failure": false
      }
    },

    {  
      "date_index_name": {
        "if": "ctx?.agent?.labels?.grup == 'grupA'",
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}grupA-",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },
    {
      "date_index_name": {
        "if": "ctx?.agent?.labels?.grup == 'grupB'",
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}grupB-",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },
    { "remove": { "field": "message", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "ecs", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "beat", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "input_type", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "tags", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "count", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "@version", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "log", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "offset", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "type", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "host", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "fields", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "event", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "fileset", "ignore_missing": true, "ignore_failure": true } },
    { "remove": { "field": "service", "ignore_missing": true, "ignore_failure": true } }
  ],
  "on_failure" : [{
    "drop" : { }
  }]
}

---------------------------------------------
Additionally, how can I change the logs for each group to be saved in a different file than /var/ossec/logs/archives/archives.json

for example:/var/ossec/logs/archives/archives-groupA.json and /var/ossec/logs/archives/archives-groupB.json and to ingest from these 2 new files? it is posible ?

[Julio Cesar Biset]

Hi Tomás,

I'm going to try to reproduce what you did in order to provide a more accurate response. In the meantime, could you please share the sources you used to carry out this configuration? Any other information you believe might be helpful for me to recreate this scenario will be greatly appreciated.

Thank you in advance.

[Serano Tomas]

Hello,

Thanks for the help.

the information is collected and put together from here on this group.

Thank you and I am waiting for a reply

[Julio Cesar Biset]

Hi,
I have applied the configurations that you have indicated, but I cannot reproduce the problem, nor can I get the indexes corresponding to each group to be generated. I am consulting with the team so I can get more information.
For now, as far as I know, it is not possible to change the logs of each group to save them in a different file than /var/ossec/logs/archives/archives.json.
Is there any other configuration that you have done that is not in the previous steps?
Could you also give me the complete configuration of /var/ossec/etc/ossec.conf?

Kind regards

[Serano Tomas]

    Hello,

Julio, in the meantime I solved it.
I used drop for anything that does not contain that label and that solved my problem.

Thank you very much for your time and sorry for the inconvenience

however, if it is possible to save the logs separately it would be perfect, do you think you can help me here with an answer ?

[Julio Cesar Biset]

Hi.

Great that you were able to fix the problem. I checked with the team about the archives.json file. Currently, I'm afraid that it is not configurable. Archives are only stored in /var/ossec/logs/archives/archives.log and/or /var/ossec/logs/archives/archives.json.

Kind regards.