Elasticsearch pipeline with ID 'filebeat-7.10.2-wazuh-alerts-pipeline' loaded

[Carlos Lopez]

Hi all,

Why this index is automatically created?

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .security-7 KSBGCxuFSeGwKJsKpq5MkQ 1 0 7 0 25.1kb 25.1kb
yellow open filebeat-7.10.2-2021.03.01-000001 fHrJPbiJTA6cAeIvMysAPg 1 1 0 0 208b 208b
green open wazuh-alerts-4.x-2021.03.01 e_OAJDgmSpWO73pFKeUmbg 3 0 156 0 295.2kb 295.2kb

This only happens when I use ILM configured in the filebeat side. For example:

setup.ilm.enabled: true
setup.ilm.policy_name: 'wazuh-retention-policy'
setup.ilm.policy_file: '/etc/filebeat/wazuh-retention-policy.json'

Is it possible to avoid the creation of this index?

Regards.

[Carlos Lopez]

Ok, it seems the problem is with ILM ...

Actually my current elastic node shows the following indices:

green open wazuh-alerts-4.x-2021.03.01-000001 Nl8d8_NBRWimqcNX71Atlg 3 0 686 0 1.1mb 1.1mb
green open wazuh-monitoring-2021.03.01 fajZJLknSpqhGnw5kbDZDQ 2 0 99 0 173.2kb 173.2kb
green open wazuh-monitoring-2021.03.02 YGcULKr1Tn2ni_bMBR0Ayw 2 0 99 0 361.3kb 361.3kb
green open wazuh-statistics-2021.10w 6ft15YToRiS82OgH40l-_w 2 0 770 0 369.1kb 369.1kb

As you can see, there is not wazuh-alerts-4.x-2021.03.01 index ...

I have modified filebeat.yml, wazuh-template.json and create my retention policy ... As I have done in the past with Wazuh 3.X, dns works without problems ...

Do I need to consider something special for Wazuh 4.X?

________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of Carlos Lopez <clo...@outlook.com>
Sent: 01 March 2021 12:26
To: wa...@googlegroups.com
Subject: Elasticsearch pipeline with ID 'filebeat-7.10.2-wazuh-alerts-pipeline' loaded

Hi all,

Regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/PR3PR07MB66173FB8DCA5168CE1267C8BDB9A9%40PR3PR07MB6617.eurprd07.prod.outlook.com.

[mayte...@wazuh.com]

Hi Carlos Lopez,

Your ILM configuration seems to be working properly since your alerts are being indexed in the wazuh-alerts-4.x-2021.03.01-000001 index. 

In the previous output we can see it has been populated with 686 documents. Could you explain your issue with more details?

Best regards,

Mayte Ariza

[Carlos Lopez]

Hi Mayte,

Thanks for your help ... What is not clear to me is why it does not generate an index per day + the -0000X file ... Actually all data goes to wazuh-alerts-4.x-2021.03.01-000001 index .... For example it doesn't exist an index for today ...

________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of mayte...@wazuh.com <mayte...@wazuh.com>
Sent: 02 March 2021 11:19
To: Wazuh mailing list
Subject: Re: Elasticsearch pipeline with ID 'filebeat-7.10.2-wazuh-alerts-pipeline' loaded

Hi Carlos Lopez,

Best regards,
Mayte Ariza

Hi all,

Regards.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com<mailto:wazuh+un...@googlegroups.com>.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/ef433760-4dcc-40ce-8a3d-982d37578d76n%40googlegroups.com<https://groups.google.com/d/msgid/wazuh/ef433760-4dcc-40ce-8a3d-982d37578d76n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Carlos Lopez]

More info. I have followed all these steps: https://groups.google.com/g/wazuh/c/tajQa0VIymA/m/jI4SbwY1EwAJ

And my filebeat log:

2021-03-02T10:38:05.688Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2021-03-02T10:38:05.744Z INFO [license] licenser/es_callback.go:51 Elasticsearch license: Basic
2021-03-02T10:38:05.747Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2021-03-02T10:38:05.793Z INFO [index-management] idxmgmt/std.go:261 Auto ILM enable success.
2021-03-02T10:38:05.798Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2021-03-02T10:38:05.798Z INFO [index-management] idxmgmt/std.go:274 ILM policy successfully loaded.
2021-03-02T10:38:05.798Z INFO [index-management] idxmgmt/std.go:407 Set setup.template.name to '{wazuh-alerts-4.x {now/d}-000001}' as ILM is enabled.
2021-03-02T10:38:05.798Z INFO [index-management] idxmgmt/std.go:412 Set setup.template.pattern to 'wazuh-alerts-4.x-*' as ILM is enabled.
2021-03-02T10:38:05.798Z INFO [index-management] idxmgmt/std.go:446 Set settings.index.lifecycle.rollover_alias in template to {wazuh-alerts-4.x {now/d}-000001} as ILM is enabled.
2021-03-02T10:38:05.798Z INFO [index-management] idxmgmt/std.go:450 Set settings.index.lifecycle.name in template to {wazuh-retention-policy {"policy":{"phases":{"hot":{"actions":{"rol
lover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-03-02T10:38:05.802Z INFO template/load.go:183 Existing template will be overwritten, as overwrite is enabled.
2021-03-02T10:38:05.802Z INFO template/load.go:117 Try loading template wazuh to Elasticsearch
2021-03-02T10:38:05.913Z INFO template/load.go:109 template with name 'wazuh' loaded.
2021-03-02T10:38:05.913Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2021-03-02T10:38:06.134Z INFO [index-management] idxmgmt/std.go:309 Write alias successfully generated.

If this is ok, when we hace an index for today?

________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of Carlos Lopez <clo...@outlook.com>

Sent: 02 March 2021 11:41
To: mayte...@wazuh.com; Wazuh mailing list

Hi Mayte,

Hi Carlos Lopez,

Best regards,
Mayte Ariza

Hi all,

Regards.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/PR3PR07MB6617F02DB59425E534563EBFDB999%40PR3PR07MB6617.eurprd07.prod.outlook.com.

[mayte...@wazuh.com]

Hi Carlos Lopez,

It depends on how the ILM policy rollover is configured. Your indices may rotate taking into account different settings:

• Maximum size
• Maximum documents
• Maximum age

You can find more info about the ILM rollover options here: ILM rollover options

Best regards,

Mayte Ariza

[mayte...@wazuh.com]

Hi Carlos Lopez,

If you followed this tutorial, the roll over will be triggered when your index reaches 5gb or after 30 days from index creation.

Best regards,

Mayte Ariza

[Carlos Lopez]

Thanks Mayte ... You are right ... It was my mistake ... I have misinterpreted how indexes are managed with ILM policies ....

Sorry for the noise ...

________________________________________
From: wa...@googlegroups.com <wa...@googlegroups.com> on behalf of mayte...@wazuh.com <mayte...@wazuh.com>

Sent: 02 March 2021 12:33

To: Wazuh mailing list
Subject: Re: Elasticsearch pipeline with ID 'filebeat-7.10.2-wazuh-alerts-pipeline' loaded

Hi Carlos Lopez,

If you followed this tutorial<https://groups.google.com/g/wazuh/c/tajQa0VIymA/m/jI4SbwY1EwAJ>, the roll over will be triggered when your index reaches 5gb or after 30 days from index creation.

Best regards,
Mayte Ariza

On Tuesday, March 2, 2021 at 12:26:04 PM UTC+1 mayte...@wazuh.com wrote:
Hi Carlos Lopez,

It depends on how the ILM policy rollover is configured. Your indices may rotate taking into account different settings:

* Maximum size
* Maximum documents
* Maximum age

You can find more info about the ILM rollover options here: ILM rollover options<https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html#ilm-rollover-options>

Best regards,
Mayte Ariza

Hi Carlos Lopez,

Best regards,
Mayte Ariza

Hi all,

Regards.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c336b9c1-cd5f-42ef-9cea-1d2f6924f8d8n%40googlegroups.com<https://groups.google.com/d/msgid/wazuh/c336b9c1-cd5f-42ef-9cea-1d2f6924f8d8n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[mayte...@wazuh.com]

Hi Carlos Lopez,

 

Pleased to help!

Do not hesitate to contact us again if you have any questions.

 

Best regards,

Mayte Ariza