Problems with alert index patterns in CCS Setup

252 views

Skip to first unread message

Sebastian

unread,

Jan 10, 2025, 4:19:27 AM1/10/25

to Wazuh | Mailing List

Hi,

I recently installed a CCS Wazuh Setup in order to manage alerts from multiple different Wazuh clusters in a single Dashboard. I followed this blog post: https://wazuh.com/blog/managing-multiple-wazuh-clusters-with-cross-cluster-search/

So far i have 3 wazuh instances connected, each containing one Wazuh-Server and one Wazuh-Indexer. On top of that i have a single Dashboard aswell as the CCS-Indexer.

Now my problem: Each time i log in, this error appears in the health check

The error log contains this text:

INFO: Index pattern id in cookie: yes [*:wazuh-alerts-*] INFO: Getting list of valid index patterns... INFO: Valid index patterns found: 1 INFO: Found default index pattern with title [*:wazuh-alerts-*]: yes INFO: Checking the app default pattern exists: id [*:wazuh-alerts-*]... INFO: Default pattern with id [*:wazuh-alerts-*] exists: yes ACTION: Default pattern id [*:wazuh-alerts-*] set as default index pattern INFO: Checking the index pattern id [*:wazuh-alerts-*] exists... INFO: Index pattern id exists [*:wazuh-alerts-*]: yes INFO: Index pattern id in cookie: yes [*:wazuh-alerts-*] INFO: Checking if the index pattern id [*:wazuh-alerts-*] exists... INFO: Index pattern id [*:wazuh-alerts-*] found: yes title [*:wazuh-alerts-*] INFO: Checking if exists a template compatible with the index pattern title [*:wazuh-alerts-*] INFO: Template found for the selected index-pattern title [*:wazuh-alerts-*]: no ERROR: No template found for the selected index-pattern title [*:wazuh-alerts-*] INFO: Index pattern id in cookie: [*:wazuh-alerts-*] INFO: Getting index pattern data [*:wazuh-alerts-*]... INFO: Index pattern data found: [yes] INFO: Refreshing index pattern fields: title [*:wazuh-alerts-*], id [*:wazuh-alerts-*]... ACTION: Refreshed index pattern fields: title [*:wazuh-alerts-*], id [*:wazuh-alerts-*] INFO: Getting settings... INFO: Check dashboard setting [timeline:max_buckets]: 200000 INFO: App setting [timeline:max_buckets]: 200000 INFO: Settings mismatch [timeline:max_buckets]: no INFO: Getting settings... INFO: Check dashboard setting [metaFields]: ["_source","_index"] INFO: App setting [metaFields]: ["_source","_index"] INFO: Settings mismatch [metaFields]: no INFO: Getting settings... INFO: Check dashboard setting [timepicker:timeDefaults]: {"from":"now-24h","to":"now"} INFO: App setting [timepicker:timeDefaults]: "{\"from\":\"now-24h\",\"to\":\"now\"}" INFO: Settings mismatch [timepicker:timeDefaults]: no

INFO: Index pattern id in cookie: yes [*:wazuh-alerts-*]
INFO: Getting list of valid index patterns...
INFO: Valid index patterns found: 1
INFO: Found default index pattern with title [*:wazuh-alerts-*]: yes
INFO: Checking the app default pattern exists: id [*:wazuh-alerts-*]...
INFO: Default pattern with id [*:wazuh-alerts-*] exists: yes
ACTION: Default pattern id [*:wazuh-alerts-*] set as default index pattern
INFO: Checking the index pattern id [*:wazuh-alerts-*] exists...
INFO: Index pattern id exists [*:wazuh-alerts-*]: yes
INFO: Index pattern id in cookie: yes [*:wazuh-alerts-*]
INFO: Checking if the index pattern id [*:wazuh-alerts-*] exists...
INFO: Index pattern id [*:wazuh-alerts-*] found: yes title [*:wazuh-alerts-*]
INFO: Checking if exists a template compatible with the index pattern title [*:wazuh-alerts-*]
INFO: Template found for the selected index-pattern title [*:wazuh-alerts-*]: no
ERROR: No template found for the selected index-pattern title [*:wazuh-alerts-*]
INFO: Index pattern id in cookie: [*:wazuh-alerts-*]
INFO: Getting index pattern data [*:wazuh-alerts-*]...
INFO: Index pattern data found: [yes]
INFO: Refreshing index pattern fields: title [*:wazuh-alerts-*], id [*:wazuh-alerts-*]...
ACTION: Refreshed index pattern fields: title [*:wazuh-alerts-*], id [*:wazuh-alerts-*]
INFO: Getting settings...
INFO: Check dashboard setting [timeline:max_buckets]: 200000
INFO: App setting [timeline:max_buckets]: 200000
INFO: Settings mismatch [timeline:max_buckets]: no
INFO: Getting settings...
INFO: Check dashboard setting [metaFields]: ["_source","_index"]
INFO: App setting [metaFields]: ["_source","_index"]
INFO: Settings mismatch [metaFields]: no
INFO: Getting settings...
INFO: Check dashboard setting [timepicker:timeDefaults]: {"from":"now-24h","to":"now"}
INFO: App setting [timepicker:timeDefaults]: "{\"from\":\"now-24h\",\"to\":\"now\"}"
INFO: Settings mismatch [timepicker:timeDefaults]: no

Other than the failing health check, the setup works just fine with each environment containing own alerts and they all also look fine. Is this a problem with the health check or am i missing something that is not working correctly?

I think the problem stems from changing the index pattern in the "App Settings" of the Dashboard from wazuh-alerts-* to *:wazuh-alerts-* in order to handle index patterns of multiple environments. I havent been able to fix this on my own, trying multiple things including installing the alert template again on each Server. Also looking at this troubleshooting guide i wasnt able to find the root problem: https://documentation.wazuh.com/4.4/user-manual/elasticsearch/troubleshooting.html

If this is a problem with the health check i can just deactivate this check, however im not sure this is the best way to go about this as i am not certain there really is no problem.

Thanks for the assistance!
Sebastian

Stuti Gupta

unread,

Jan 10, 2025, 4:44:12 AM1/10/25

to Wazuh | Mailing List

Hi

It is because the alerts template was not installed correctly.
/etc/filebeat/wazuh-template.json

You can also manually add the index by running the following command:
curl https://raw.githubusercontent.com/wazuh/wazuh/v4.9.2/extensions/elasticsearch/7.x/wazuh-template.json | curl -X PUT "https://localhost:9200/_template/wazuh" -H 'Content-Type: application/json' -d @- -u <user>:<password> -k

If that still causes an error then Please check the Filebeat and wazuh indexer service status using the command: systemctl status wazuh-indexer. Please share the cat /var/log/wazuh-indexer/wazuh-cluster.log and cat /var/log/filebeat/filebeat output

Please take a look at this link: https://documentation.wazuh.com/current/user-manual/elasticsearch/troubleshooting.html#no-template-found-for-the-selected-index-pattern

Make sure you have followed the steps carefully to set up the cross cluster search chttps://wazuh.com/blog/managing-multiple-wazuh-clusters-with-cross-cluster-search/#:~:text=or%20B%20environments.-,Set%20up%20Cross%2DCluster%20Search%C2%A0,-Perform%20the%20following

Configured the index pattern as explained here https://wazuh.com/blog/managing-multiple-wazuh-clusters-with-cross-cluster-search/#:~:text=Configure%20the%20wazuh%2Dalerts%2D*%20index%20pattern

Hope to hear from you soon

Sebastian

unread,

Jan 10, 2025, 5:39:13 AM1/10/25

to Wazuh | Mailing List

Hi,

i installed the template on each individual indexer aswell as the ccs one, then restarted each one, the error still appears. Is it a problem that the default template has these index patterns:

index_patterns": [ "wazuh-alerts-4.x-*", "wazuh-archives-4.x-*" ],

when what actually i used is something like *:wazuh-alerts-* ?

Because when installing the template according to the blog post the template is not changed in any way.

Anyways here is the output from the ccs-indexer:
● wazuh-indexer.service - wazuh-indexer
Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2025-01-10 10:21:08 UTC; 6min ago
Docs: https://documentation.wazuh.com
Main PID: 720814 (java)
Tasks: 151 (limit: 38377)
Memory: 1.5G
CPU: 56.302s
CGroup: /system.slice/wazuh-indexer.service
└─720814 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headle>

Jan 10 10:20:59 wa-dashboard systemd-entrypoint[720814]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Jan 10 10:20:59 wa-dashboard systemd-entrypoint[720814]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Jan 10 10:20:59 wa-dashboard systemd-entrypoint[720814]: WARNING: System::setSecurityManager will be removed in a future release
Jan 10 10:20:59 wa-dashboard systemd-entrypoint[720814]: Jan 10, 2025 10:20:59 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Jan 10 10:20:59 wa-dashboard systemd-entrypoint[720814]: WARNING: COMPAT locale provider will be removed in a future release
Jan 10 10:21:00 wa-dashboard systemd-entrypoint[720814]: WARNING: A terminally deprecated method in java.lang.System has been called
Jan 10 10:21:00 wa-dashboard systemd-entrypoint[720814]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)
Jan 10 10:21:00 wa-dashboard systemd-entrypoint[720814]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
Jan 10 10:21:00 wa-dashboard systemd-entrypoint[720814]: WARNING: System::setSecurityManager will be removed in a future release
Jan 10 10:21:08 wa-dashboard systemd[1]: Started wazuh-indexer.

The command " cat /var/log/wazuh-indexer/wazuh-cluster.log" doesnt work on any indexer as i dont have any indexer clusters installed, a single indexer is used in each environment

Filebeat output:

Server 1:

2024-12-11T06:44:28.176Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2024-12-11T06:44:28.176Z INFO instance/beat.go:653 Beat ID: 5eac6650-6782-4607-a676-3d696f600934
2024-12-11T06:44:28.176Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.2' as ILM is enabled.
2024-12-11T06:44:28.177Z INFO eslegclient/connection.go:99 elasticsearch url: https://<indexer>:9200
2024-12-11T06:44:28.190Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2

Server 2:

2025-01-08T07:22:51.365Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2025-01-08T07:22:51.366Z INFO instance/beat.go:653 Beat ID: 39af3f11-45d8-4495-b64c-8abadff93530
2025-01-08T07:22:51.366Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2025-01-08T07:22:51.366Z INFO [beat] instance/beat.go:981 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "39af3f11-45d8-4495-b64c-8abadff93530"}}}
2025-01-08T07:22:51.366Z INFO [beat] instance/beat.go:990 Build info {"system_info": {"build": {"commit": "aacf9ecd9c494aa0908f61fbca82c906b16562a8", "libbeat": "7.10.2", "time": "2021-01-12T22:10:33.000Z", "version": "7.10.2"}}}
2025-01-08T07:22:51.366Z INFO [beat] instance/beat.go:993 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.14.12"}}}
2025-01-08T07:22:51.367Z INFO [beat] instance/beat.go:997 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2024-11-29T10:21:43Z","containerized":false,"name":"hswazuhserver","ip":["127.0.0.1/8","::1/128","<server>/24","fe80::20c:29ff:fe68:6dd2/64"],"kernel_version":"5.15.0-94-generic","mac":["00:0c:29:68:6d:d2"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.4 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":4,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0,"id":"092b27f0cecc495ca2f6a10158618b3f"}}}
2025-01-08T07:22:51.367Z INFO [beat] instance/beat.go:1026 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 544001, "ppid": 1, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2025-01-08T07:22:50.930Z"}}}
2025-01-08T07:22:51.367Z INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.10.2
2025-01-08T07:22:51.368Z INFO eslegclient/connection.go:99 elasticsearch url: https://<indexer>:9200
2025-01-08T07:22:51.368Z INFO [publisher] pipeline/module.go:113 Beat name: hswazuhserver
2025-01-08T07:22:51.369Z INFO beater/filebeat.go:117 Enabled modules/filesets: wazuh (alerts), ()
2025-01-08T07:22:51.369Z INFO instance/beat.go:455 filebeat start running.
2025-01-08T07:22:51.369Z INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2025-01-08T07:22:51.396Z INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=2067
2025-01-08T07:22:51.396Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 1
2025-01-08T07:22:51.396Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2025-01-08T07:22:51.396Z INFO log/input.go:157 Configured paths: [/var/ossec/logs/alerts/alerts.json]
2025-01-08T07:22:51.396Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 9132358592892857476)
2025-01-08T07:22:51.396Z INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2025-01-08T07:22:51.396Z INFO log/harvester.go:302 Harvester started for file: /var/ossec/logs/alerts/alerts.json
2025-01-08T07:22:52.397Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://<indexer>:9200))
2025-01-08T07:22:52.397Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2025-01-08T07:22:52.397Z INFO [publisher] pipeline/retry.go:223 done
2025-01-08T07:22:52.408Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2025-01-08T07:22:52.408Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2025-01-08T07:22:52.409Z INFO template/load.go:97 Template wazuh already exists and will not be overwritten.
2025-01-08T07:22:52.409Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2025-01-08T07:22:52.411Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://<indexer>:9200)) established
2025-01-09T00:00:02.544Z INFO log/harvester.go:302 Harvester started for file: /var/ossec/logs/alerts/alerts.json
2025-01-09T00:05:07.757Z INFO log/harvester.go:333 File is inactive: /var/ossec/logs/alerts/alerts.json. Closing because close_inactive of 5m0s reached.
2025-01-10T00:00:04.162Z INFO log/harvester.go:302 Harvester started for file: /var/ossec/logs/alerts/alerts.json
2025-01-10T00:05:08.108Z INFO log/harvester.go:333 File is inactive: /var/ossec/logs/alerts/alerts.json. Closing because close_inactive of 5m0s reached.
2025-01-10T10:05:40.809Z ERROR [elasticsearch] elasticsearch/client.go:224 failed to perform any bulk index operations: Post "https://<indexer>:9200/_bulk": dial tcp <indexer>:9200: connect: connection refused
2025-01-10T10:05:40.809Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2025-01-10T10:05:40.809Z INFO [publisher] pipeline/retry.go:223 done
2025-01-10T10:05:42.392Z ERROR [publisher_pipeline_output] pipeline/output.go:180 failed to publish events: Post "https://<indexer>:9200/_bulk": dial tcp <indexer>:9200: connect: connection refused
2025-01-10T10:05:42.392Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(elasticsearch(https://<indexer>:9200))
2025-01-10T10:05:42.392Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2025-01-10T10:05:42.393Z INFO [publisher] pipeline/retry.go:223 done
2025-01-10T10:05:44.969Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://<indexer>:9200)): Get "https://<indexer>:9200": dial tcp <indexer>:9200: connect: connection refused
2025-01-10T10:05:44.969Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://<indexer>:9200)) with 1 reconnect attempt(s)
2025-01-10T10:05:44.969Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2025-01-10T10:05:44.969Z INFO [publisher] pipeline/retry.go:223 done
2025-01-10T10:05:51.959Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://<indexer>:9200)): Get "https://<indexer>:9200": dial tcp <indexer>:9200: connect: connection refused
2025-01-10T10:05:51.959Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://<indexer>:9200)) with 2 reconnect attempt(s)
2025-01-10T10:05:51.959Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2025-01-10T10:05:51.960Z INFO [publisher] pipeline/retry.go:223 done
2025-01-10T10:06:05.243Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://<indexer>:9200)): 503 Service Unavailable: OpenSearch Security not initialized.
2025-01-10T10:06:05.244Z INFO [publisher_pipeline_output] pipeline/output.go:145 Attempting to reconnect to backoff(elasticsearch(https://<indexer>:9200)) with 3 reconnect attempt(s)
2025-01-10T10:06:05.244Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2025-01-10T10:06:05.244Z INFO [publisher] pipeline/retry.go:223 done
2025-01-10T10:06:05.579Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2
2025-01-10T10:06:05.589Z INFO template/load.go:97 Template wazuh already exists and will not be overwritten.
2025-01-10T10:06:05.589Z INFO [index-management] idxmgmt/std.go:298 Loaded index template.
2025-01-10T10:06:05.591Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(elasticsearch(https://<indexer>:9200)) established

Server 3:

2024-12-11T06:39:46.111Z INFO instance/beat.go:645 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2024-12-11T06:39:46.111Z INFO instance/beat.go:653 Beat ID: 79df3468-852c-4d08-b616-7cb3134cd5c7
2024-12-11T06:39:46.111Z INFO [index-management] idxmgmt/std.go:184 Set output.elasticsearch.index to 'filebeat-7.10.2' as ILM is enabled.
2024-12-11T06:39:46.112Z INFO eslegclient/connection.go:99 elasticsearch url: https://<indexer>:9200
2024-12-11T06:39:46.139Z INFO [esclientleg] eslegclient/connection.go:314 Attempting to connect to Elasticsearch version 7.10.2

It seems there is a problem with server 2, is it related to the index template issue?

Thanks,

Sebastian

Stuti Gupta

unread,

Jan 16, 2025, 5:39:29 AM1/16/25

to Wazuh | Mailing List

Hi

It seems that filebeat is unable to reach wazuh-indexer, make sure the indexer is active and Also, check the filebeat status to ensure that it's able to communicate with the Wazuh indexer filebeat test output.

Ensure that the host value in /etc/filebeat/filebeat.yml is set to the list of Wazuh indexer nodes to connect to. The indexer IP should match the one specified in config.yml used for generating the certificates.

Please share the indexer logs cat /var/log/wazuh-indexer/wazuh-cluster.log

Hope to hear from you soon

m_alfo

unread,

Mar 26, 2025, 7:10:32 AM3/26/25

to Wazuh | Mailing List

Any updates?
I have the same problem, the same error only in the health-check, but when i go directly to the home all works fine, also changing to others tenant.

Thanks

Sebastian

unread,

Mar 26, 2025, 7:29:59 AM3/26/25

to Wazuh | Mailing List

Hi,

sadly i never heard back on this issue so i finally decided against going with the CCS Setup. After hours of debugging i was not able to find or fix the issue and since i also could not get any help here i decided it was too risky to pursue, as new updates could break it even more with an indexing change or something of that nature.