Custom index

[Matthias A]

Hi,

I have been trying for days now, but I can't get a new custom index to work

So I follow these steps? 

1. Make custom index in  /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json  

2. Create custom index pattern in wazuh-template.json

3. Insert template in wazuh indexer

this is what I put into  /usr/share/filebeat/module/wazuh/alerts/ingest/pipeline.json  

    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "wazuh-vpn-*",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false,
        "if" : "ctx.rule?.id == '201010'"
      }
    },
    {
      "date_index_name": {
        "field": "timestamp",
        "date_rounding": "d",
        "index_name_prefix": "{{fields.index_prefix}}",
        "index_name_format": "yyyy.MM.dd",
        "ignore_failure": false
      }
    },

this is what I put in my template.json file:

  "index_patterns": [
    "wazuh-alerts-4.x-*",
    "wazuh-archives-4.x-*",
    "wazuh-vpn-*"
  ],

I then try this command, but it's not working at all

I also tried with this command:

filebeat setup --pipelines --modules wazuh

Can someone help me step by step to create an index for VPN with this event ID

"if" : "ctx.rule?.id == '201010'"

Thanks in advance

Matthias

[Juan Cabrera]

Hi Matthias,

It seems like you're encountering issues with Filebeat regarding custom indexing for specific events with the rule_id '201010', is that correct? Understanding the reasons behind this customization could assist us in finding a more suitable solution for your scenario.

Upon reviewing the screenshot, it appears that the curl command is incomplete as it lacks the indexer port, although it still seems to be receiving a response. However, it's crucial to ensure the command is complete and accurate. Could you confirm if the file is indeed named "template.json" as mentioned in the curl command?

Best regards!

[Matthias A]

Okay, I added port 9200 to the command, but now I get a connection refused.

root@WAZUH-VM:~# curl -XPUT -k -u admin:XXX 'https://192.168.154.179:9200/_template/wazuh' -H 'Content-Type: application/json' -d @template.json
curl: (7) Failed to connect to 192.168.154.179 port 9200 after 0 ms: Connection refused

Op dinsdag 14 mei 2024 om 12:13:34 UTC+2 schreef Juan Cabrera:

[Matthias A]

i also added it to my firewall btw

Op dinsdag 14 mei 2024 om 13:03:28 UTC+2 schreef Matthias A:

[Juan Cabrera]

Hi,

To assist you better, could you please provide detailed information about your specific objective? Understanding your exact requirements will enable me to offer tailored assistance and provide you with an appropriate solution for your request.   

To verify if the pipeline is configured correctly, you can refer to the documentation available at https://www.elastic.co/guide/en/elasticsearch/reference/current/date-index-name-processor.html

To update the pipelines and the template (since you've modified it), you should execute the following commands:

filebeat setup --pipelines filebeat setup --index-management

The pipeline looks correct, except for the index-prefix. I think it should be wazuh-vpn- instead of wazuh-vpn-*.

Regards !

[Matthias A]

Hi Juan,

I updated it from wazuh-vpn-* to wazuh-vpn- like you said, and then updated the pipelines with that command, is there anything else I need to do now?

My objective is: I want logs with rule.id = 201010 to be in a seperate index.

To give more context on my objective:

I have made a decoder for my firewall

My setup logs alll data to archives and alerts also to alerts

My firewall has a few types of traffic, some logs are for allow or deny actions, some are for VPN acction, some are from VPN logins

I want to seperate the log in from the others, so I can change the retention period for those logs.

I made a decoder for the vpn login logs, and and a rule with id = 201010

Kind regards,

Matthias

Op dinsdag 14 mei 2024 om 14:17:11 UTC+2 schreef Juan Cabrera:

[Matthias A]

I made a typo:

I have made a decoder for my firewall

My setup logs alll data to archives and alerts also to alerts

My firewall has a few types of traffic, some logs are for allow or deny actions, some are for VPN acction, some are from VPN logins

I want to seperate the vpn log in logs from the others, so I can change the retention period for those logs.

I made a decoder for the vpn login logs, and and a rule with id = 201010

Op dinsdag 14 mei 2024 om 14:23:44 UTC+2 schreef Matthias A:

[Juan Cabrera]

Hello Matthias,

I noticed that in your configuration, the order of processors defining the index pattern’s name could lead to unintended results. Specifically, the processor setting the index name to the default one comes after the conditional processor. Reversing this order is advisable. Otherwise, all documents passing through the pipeline will be stored in the default index instead of the intended ‘wazuh-vpn-*’ index. Here’s the relevant part of your configuration:

{ "date_index_name": { "field": "timestamp", "date_rounding": "d", "index_name_prefix": "wazuh-vpn-*", "index_name_format": "yyyy.MM.dd", "ignore_failure": false, "if": "ctx.rule?.id == '201010'" } }, { "date_index_name": { "field": "timestamp", "date_rounding": "d", "index_name_prefix": "{{fields.index_prefix}}", "index_name_format": "yyyy.MM.dd", "ignore_failure": false } }

Additionally, the conditional processor’s prefix to set the index name to ‘wazuh-vpn-‘ includes an asterisk. It’s recommended to remove it to prevent index names like ‘wazuh-vpn-2024.05.15’. Instead, use the ‘wazuh-vpn-‘ prefix to ensure index names like ‘wazuh-vpn-2024.05.15’ (without the asterisk).

To update the template file that Filebeat uses in its configuration, you can execute the following command:

filebeat setup --index-management

Regards!

​