I've broken my wazuh cluster accidentally after fresh install
565 views
Skip to first unread message
wazuh
Jul 8, 2024, 3:52:42 AM7/8/24
to Wazuh | Mailing List
Hi I've seem to have broken my wazuh-cluster deployment after deleting the indices directory in both indexers from /var/lib/wazuh-indexer/nodes/0/ directory. (Previously I did a fresh re-install of node-1 indexer after i broke my cluster and had the same problem. deleting the indices folder in node-2 seemed to have fixed my issue at the time). I was stress testing my wazuh-indexer, which broke it, so i thought of deleting my indices and reinitializing the cluster. Currently this is the error I have from /var/log/wazuh-indexer/wazuh-cluster.log before initialization:
[2024-07-02T05:54:13,452][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@
1141244] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-02T05:54:13,452][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@ 1141244] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-02T05:54:15,055][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-07-02T05:54:15,058][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-07-02T05:54:13,452][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@ 1141244] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-02T05:54:15,055][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
[2024-07-02T05:54:15,058][ERROR][o.o.s.a.BackendRegistry ] [node-1] Not yet initialized (you may need to run securityadmin)
I've used curl -k -u admin:admin https://<WAZUH_INDEXER_IP_ADRESS>:9200 to check if cluster is initalized, it says it is not initialized so i tried using the indexer-security-init.sh.
when I run /usr/share/wazuh-indexer/bin/indexer-security-init.sh here is the following error I am stuck on:
Security Admin v7
Will connect to 10.4.6.4:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.10.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
* Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
* If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
Will connect to 10.4.6.4:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.10.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Cannot retrieve cluster state due to: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE]. This is not an error, will keep on trying ...
Root cause: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-2 [ACTIVE] (java.net.SocketTimeoutException/java.net.SocketTimeoutException)
* Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
* Make sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
* If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
* Add --accept-red-cluster to allow securityadmin to operate on a red cluster.
and this is the logs of when i start start my node-1 indexer service:
[2024-07-02T06:04:36,460][INFO ][o.o.s.s.t.SSLConfig ] [node-1] SSL dual mode is disabled
[2024-07-02T06:04:36,461][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /etc/wazuh-indexer
[2024-07-02T06:04:36,855][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3
[2024-07-02T06:04:36,859][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /etc/wazuh-indexer/, from there the key- and truststore files are resolved relatively
[2024-07-02T06:04:37,831][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Client Provider : JDK
[2024-07-02T06:04:37,831][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Server Provider : JDK
[2024-07-02T06:04:37,832][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS HTTP Provider : JDK
[2024-07-02T06:04:37,832][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2024-07-02T06:04:37,833][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2024-07-02T06:04:37,861][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Clustername: wazuh-cluster
[2024-07-02T06:04:37,871][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/certs/-key has insecure file permissions (should be 0600)
[2024-07-02T06:04:37,871][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/certs/-cert has insecure file permissions (should be 0600)
[2024-07-02T06:04:39,036][INFO ][o.o.p.c.c.PluginSettings ] [node-1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-07-02T06:04:39,841][INFO ][o.o.i.r.ReindexPlugin ] [node-1] ReindexPlugin reloadSPI called
[2024-07-02T06:04:39,848][INFO ][o.o.i.r.ReindexPlugin ] [node-1] Unable to find any implementation for RemoteReindexExtension
[2024-07-02T06:04:39,957][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-07-02T06:04:39,960][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-07-02T06:04:39,961][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-07-02T06:04:39,962][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-07-02T06:04:39,972][INFO ][o.o.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2024-07-02T06:04:39,972][INFO ][o.o.p.PluginsService ] [node-1] loaded module [analysis-common]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [geo]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-common]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-expression]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-painless]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [opensearch-dashboards]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [parent-join]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [percolator]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [rank-eval]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [reindex]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [repository-url]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [search-pipeline-common]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [systemd]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-alerting]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-anomaly-detection]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-asynchronous-search]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-cross-cluster-replication]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-custom-codecs]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-geospatial]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-index-management]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-job-scheduler]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-knn]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-ml]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-neural-search]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications-core]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-observability]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-performance-analyzer]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-reports-scheduler]
[2024-07-02T06:04:39,981][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-security]
[2024-07-02T06:04:39,981][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-security-analytics]
[2024-07-02T06:04:39,981][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-sql]
[2024-07-02T06:04:40,058][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-07-02T06:04:40,067][INFO ][o.o.e.ExtensionsManager ] [node-1] ExtensionsManager initialized
[2024-07-02T06:04:40,127][INFO ][o.o.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/mnt/indexer (/dev/sdb)]], net usable_space [9.2gb], net total_space [9.7gb], types [ext4]
[2024-07-02T06:04:40,130][INFO ][o.o.e.NodeEnvironment ] [node-1] heap size [128mb], compressed ordinary object pointers [true]
[2024-07-02T06:04:40,260][INFO ][o.o.n.Node ] [node-1] node name [node-1], node ID [yeraWVnORKeSSQOiEEmhOw], cluster name [wazuh-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-07-02T06:04:44,230][INFO ][o.o.n.p.NeuralSearch ] [node-1] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-07-02T06:04:44,603][WARN ][o.o.s.c.Salt ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-07-02T06:04:44,672][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2024-07-02T06:04:44,674][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
[2024-07-02T06:04:44,675][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Message routing enabled: false
[2024-07-02T06:04:44,711][INFO ][o.o.s.f.SecurityFilter ] [node-1] <NONE> indices are made immutable.
[2024-07-02T06:04:45,219][INFO ][o.o.a.b.ADCircuitBreakerService] [node-1] Registered memory breaker.
[2024-07-02T06:04:45,736][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML memory breaker.
[2024-07-02T06:04:45,736][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML disk breaker.
[2024-07-02T06:04:45,737][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML native memory breaker.
[2024-07-02T06:04:45,937][INFO ][o.r.Reflections ] [node-1] Reflections took 75 ms to scan 1 urls, producing 17 keys and 43 values
[2024-07-02T06:04:46,058][WARN ][o.o.s.p.SQLPlugin ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2024-07-02T06:04:47,001][INFO ][o.o.t.NettyAllocator ] [node-1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=128mb}]
[2024-07-02T06:04:47,168][INFO ][o.o.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2024-07-02T06:04:47,865][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-07-02T06:04:48,624][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node-1] PerformanceAnalyzer Enabled: true
[2024-07-02T06:04:48,687][INFO ][o.o.n.Node ] [node-1] initialized
[2024-07-02T06:04:48,688][INFO ][o.o.n.Node ] [node-1] starting ...
[2024-07-02T06:04:48,762][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [windows_logtype.json] log type
[2024-07-02T06:04:48,763][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [vpcflow_logtype.json] log type
[2024-07-02T06:04:48,764][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [test_windows_logtype.json] log type
[2024-07-02T06:04:48,768][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [s3_logtype.json] log type
[2024-07-02T06:04:48,769][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_web_logtype.json] log type
[2024-07-02T06:04:48,769][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_proxy_logtype.json] log type
[2024-07-02T06:04:48,770][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_macos_logtype.json] log type
[2024-07-02T06:04:48,771][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_compliance_logtype.json] log type
[2024-07-02T06:04:48,772][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_cloud_logtype.json] log type
[2024-07-02T06:04:48,772][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_apt_logtype.json] log type
[2024-07-02T06:04:48,773][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_application_logtype.json] log type
[2024-07-02T06:04:48,774][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [okta_logtype.json] log type
[2024-07-02T06:04:48,775][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [network_logtype.json] log type
[2024-07-02T06:04:48,776][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [netflow_logtype.json] log type
[2024-07-02T06:04:48,777][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [m365_logtype.json] log type
[2024-07-02T06:04:48,777][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [linux_logtype.json] log type
[2024-07-02T06:04:48,778][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [gworkspace_logtype.json] log type
[2024-07-02T06:04:48,779][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [github_logtype.json] log type
[2024-07-02T06:04:48,780][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [dns_logtype.json] log type
[2024-07-02T06:04:48,780][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [cloudtrail_logtype.json] log type
[2024-07-02T06:04:48,781][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [azure_logtype.json] log type
[2024-07-02T06:04:48,782][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [apache_access_logtype.json] log type
[2024-07-02T06:04:48,784][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [ad_ldap_logtype.json] log type
[2024-07-02T06:04:48,965][INFO ][o.o.t.TransportService ] [node-1] publish_address {10.4.6.4:9300}, bound_addresses {10.4.6.4:9300}
[2024-07-02T06:04:48,967][INFO ][o.o.t.TransportService ] [node-1] Remote clusters initialized successfully.
[2024-07-02T06:04:49,396][INFO ][o.o.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-07-02T06:04:49,409][INFO ][o.o.c.c.Coordinator ] [node-1] cluster UUID [3PAQtzA6R8Sxwu__yF2cpg]
[2024-07-02T06:04:50,572][INFO ][o.o.c.s.ClusterApplierService] [node-1] cluster-manager node changed {previous [], current [{node-2}{apW2rGUfRk2w6K3kaaOQUw}{yeZkVT0IRROVn0UScZR8Cw}{10.4.6.5}{10.4.6.5:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, added {{node-2}{apW2rGUfRk2w6K3kaaOQUw}{yeZkVT0IRROVn0UScZR8Cw}{10.4.6.5}{10.4.6.5:9300}{dimr}{shard_indexing_pressure_enabled=true}}, term: 19, version: 792, reason: ApplyCommitRequest{term=19, version=792, sourceNode={node-2}{apW2rGUfRk2w6K3kaaOQUw}{yeZkVT0IRROVn0UScZR8Cw}{10.4.6.5}{10.4.6.5:9300}{dimr}{shard_indexing_pressure_enabled=true}}
[2024-07-02T06:04:50,594][INFO ][o.o.c.s.ClusterSettings ] [node-1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2024-07-02T06:04:50,636][INFO ][o.o.a.c.HashRing ] [node-1] Node added: [apW2rGUfRk2w6K3kaaOQUw, yeraWVnORKeSSQOiEEmhOw]
[2024-07-02T06:04:50,647][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Cluster node changed, node removed: false, node added: true
[2024-07-02T06:04:50,649][INFO ][o.o.a.c.HashRing ] [node-1] AD version hash ring change is in progress. Can't build hash ring for node delta event.
[2024-07-02T06:04:50,649][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Hash ring build result: false
[2024-07-02T06:04:50,650][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-07-02T06:04:50,655][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: apW2rGUfRk2w6K3kaaOQUw
[2024-07-02T06:04:50,657][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: yeraWVnORKeSSQOiEEmhOw
[2024-07-02T06:04:50,658][INFO ][o.o.a.c.HashRing ] [node-1] All nodes with known AD version: {apW2rGUfRk2w6K3kaaOQUw=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}, yeraWVnORKeSSQOiEEmhOw=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}}
[2024-07-02T06:04:50,658][INFO ][o.o.a.c.HashRing ] [node-1] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 1
[2024-07-02T06:04:50,664][INFO ][o.o.a.c.HashRing ] [node-1] Build AD version hash ring successfully
[2024-07-02T06:04:50,667][INFO ][o.o.a.c.ADDataMigrator ] [node-1] Start migrating AD data
[2024-07-02T06:04:50,668][INFO ][o.o.a.c.ADDataMigrator ] [node-1] AD job index doesn't exist, no need to migrate
[2024-07-02T06:04:50,674][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Init AD version hash ring successfully
[2024-07-02T06:04:50,709][INFO ][o.o.m.a.MLModelAutoReDeployer] [node-1] Model auto reload configuration is false, not performing auto reloading!
[2024-07-02T06:04:50,728][INFO ][o.o.d.PeerFinder ] [node-1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node-1}{yeraWVnORKeSSQOiEEmhOw}{FjRsCFIyQ5uaEWOd6VMYow}{10.4.6.4}{10.4.6.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-07-02T06:04:50,752][INFO ][o.o.h.AbstractHttpServerTransport] [node-1] publish_address {10.4.6.4:9200}, bound_addresses {10.4.6.4:9200}
[2024-07-02T06:04:50,752][INFO ][o.o.n.Node ] [node-1] started
[2024-07-02T06:04:50,760][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started
[2024-07-02T06:04:50,761][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2024-07-02T06:04:50,780][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: []
[2024-07-02T06:04:50,792][INFO ][o.o.s.l.LogTypeService ] [node-1] Loading builtin types!
[2024-07-02T06:04:50,798][INFO ][o.o.s.l.LogTypeService ] [node-1] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-07-02T06:04:50,798][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2024-07-02T06:04:50,805][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: false
[2024-07-02T06:04:50,808][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-07-02T06:04:50,837][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@1141244] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
[2024-07-02T06:04:36,460][INFO ][o.o.s.s.t.SSLConfig ] [node-1] SSL dual mode is disabled
[2024-07-02T06:04:36,461][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /etc/wazuh-indexer
[2024-07-02T06:04:36,855][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3
[2024-07-02T06:04:36,859][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /etc/wazuh-indexer/, from there the key- and truststore files are resolved relatively
[2024-07-02T06:04:37,831][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Client Provider : JDK
[2024-07-02T06:04:37,831][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Server Provider : JDK
[2024-07-02T06:04:37,832][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS HTTP Provider : JDK
[2024-07-02T06:04:37,832][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2024-07-02T06:04:37,833][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2024-07-02T06:04:37,861][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Clustername: wazuh-cluster
[2024-07-02T06:04:37,871][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/certs/-key has insecure file permissions (should be 0600)
[2024-07-02T06:04:37,871][WARN ][o.o.s.OpenSearchSecurityPlugin] [node-1] File /etc/wazuh-indexer/certs/-cert has insecure file permissions (should be 0600)
[2024-07-02T06:04:39,036][INFO ][o.o.p.c.c.PluginSettings ] [node-1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2024-07-02T06:04:39,841][INFO ][o.o.i.r.ReindexPlugin ] [node-1] ReindexPlugin reloadSPI called
[2024-07-02T06:04:39,848][INFO ][o.o.i.r.ReindexPlugin ] [node-1] Unable to find any implementation for RemoteReindexExtension
[2024-07-02T06:04:39,957][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2024-07-02T06:04:39,960][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs
[2024-07-02T06:04:39,961][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2024-07-02T06:04:39,962][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: scheduler_geospatial_ip2geo_datasource, index: .scheduler-geospatial-ip2geo-datasource
[2024-07-02T06:04:39,972][INFO ][o.o.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2024-07-02T06:04:39,972][INFO ][o.o.p.PluginsService ] [node-1] loaded module [analysis-common]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [geo]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-common]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2024-07-02T06:04:39,973][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-expression]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-painless]
[2024-07-02T06:04:39,974][INFO ][o.o.p.PluginsService ] [node-1] loaded module [mapper-extras]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [opensearch-dashboards]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [parent-join]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [percolator]
[2024-07-02T06:04:39,975][INFO ][o.o.p.PluginsService ] [node-1] loaded module [rank-eval]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [reindex]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [repository-url]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [search-pipeline-common]
[2024-07-02T06:04:39,976][INFO ][o.o.p.PluginsService ] [node-1] loaded module [systemd]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-alerting]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-anomaly-detection]
[2024-07-02T06:04:39,977][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-asynchronous-search]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-cross-cluster-replication]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-custom-codecs]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-geospatial]
[2024-07-02T06:04:39,978][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-index-management]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-job-scheduler]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-knn]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-ml]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-neural-search]
[2024-07-02T06:04:39,979][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-notifications-core]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-observability]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-performance-analyzer]
[2024-07-02T06:04:39,980][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-reports-scheduler]
[2024-07-02T06:04:39,981][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-security]
[2024-07-02T06:04:39,981][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-security-analytics]
[2024-07-02T06:04:39,981][INFO ][o.o.p.PluginsService ] [node-1] loaded plugin [opensearch-sql]
[2024-07-02T06:04:40,058][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Disabled https compression by default to mitigate BREACH attacks. You can enable it by setting 'http.compression: true' in opensearch.yml
[2024-07-02T06:04:40,067][INFO ][o.o.e.ExtensionsManager ] [node-1] ExtensionsManager initialized
[2024-07-02T06:04:40,127][INFO ][o.o.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/mnt/indexer (/dev/sdb)]], net usable_space [9.2gb], net total_space [9.7gb], types [ext4]
[2024-07-02T06:04:40,130][INFO ][o.o.e.NodeEnvironment ] [node-1] heap size [128mb], compressed ordinary object pointers [true]
[2024-07-02T06:04:40,260][INFO ][o.o.n.Node ] [node-1] node name [node-1], node ID [yeraWVnORKeSSQOiEEmhOw], cluster name [wazuh-cluster], roles [ingest, remote_cluster_client, data, cluster_manager]
[2024-07-02T06:04:44,230][INFO ][o.o.n.p.NeuralSearch ] [node-1] Registering hybrid query phase searcher with feature flag [plugins.neural_search.hybrid_search_disabled]
[2024-07-02T06:04:44,603][WARN ][o.o.s.c.Salt ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2024-07-02T06:04:44,672][ERROR][o.o.s.a.s.SinkProvider ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2024-07-02T06:04:44,674][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
[2024-07-02T06:04:44,675][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Message routing enabled: false
[2024-07-02T06:04:44,711][INFO ][o.o.s.f.SecurityFilter ] [node-1] <NONE> indices are made immutable.
[2024-07-02T06:04:45,219][INFO ][o.o.a.b.ADCircuitBreakerService] [node-1] Registered memory breaker.
[2024-07-02T06:04:45,736][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML memory breaker.
[2024-07-02T06:04:45,736][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML disk breaker.
[2024-07-02T06:04:45,737][INFO ][o.o.m.b.MLCircuitBreakerService] [node-1] Registered ML native memory breaker.
[2024-07-02T06:04:45,937][INFO ][o.r.Reflections ] [node-1] Reflections took 75 ms to scan 1 urls, producing 17 keys and 43 values
[2024-07-02T06:04:46,058][WARN ][o.o.s.p.SQLPlugin ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2024-07-02T06:04:47,001][INFO ][o.o.t.NettyAllocator ] [node-1] creating NettyAllocator with the following configs: [name=unpooled, suggested_max_allocation_size=256kb, factors={opensearch.unsafe.use_unpooled_allocator=null, g1gc_enabled=true, g1gc_region_size=1mb, heap_size=128mb}]
[2024-07-02T06:04:47,168][INFO ][o.o.d.DiscoveryModule ] [node-1] using discovery type [zen] and seed hosts providers [settings]
[2024-07-02T06:04:47,865][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-07-02T06:04:48,624][INFO ][o.o.p.h.c.PerformanceAnalyzerConfigAction] [node-1] PerformanceAnalyzer Enabled: true
[2024-07-02T06:04:48,687][INFO ][o.o.n.Node ] [node-1] initialized
[2024-07-02T06:04:48,688][INFO ][o.o.n.Node ] [node-1] starting ...
[2024-07-02T06:04:48,762][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [windows_logtype.json] log type
[2024-07-02T06:04:48,763][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [vpcflow_logtype.json] log type
[2024-07-02T06:04:48,764][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [test_windows_logtype.json] log type
[2024-07-02T06:04:48,768][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [s3_logtype.json] log type
[2024-07-02T06:04:48,769][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_web_logtype.json] log type
[2024-07-02T06:04:48,769][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_proxy_logtype.json] log type
[2024-07-02T06:04:48,770][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_macos_logtype.json] log type
[2024-07-02T06:04:48,771][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_compliance_logtype.json] log type
[2024-07-02T06:04:48,772][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_cloud_logtype.json] log type
[2024-07-02T06:04:48,772][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_apt_logtype.json] log type
[2024-07-02T06:04:48,773][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [others_application_logtype.json] log type
[2024-07-02T06:04:48,774][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [okta_logtype.json] log type
[2024-07-02T06:04:48,775][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [network_logtype.json] log type
[2024-07-02T06:04:48,776][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [netflow_logtype.json] log type
[2024-07-02T06:04:48,777][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [m365_logtype.json] log type
[2024-07-02T06:04:48,777][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [linux_logtype.json] log type
[2024-07-02T06:04:48,778][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [gworkspace_logtype.json] log type
[2024-07-02T06:04:48,779][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [github_logtype.json] log type
[2024-07-02T06:04:48,780][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [dns_logtype.json] log type
[2024-07-02T06:04:48,780][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [cloudtrail_logtype.json] log type
[2024-07-02T06:04:48,781][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [azure_logtype.json] log type
[2024-07-02T06:04:48,782][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [apache_access_logtype.json] log type
[2024-07-02T06:04:48,784][INFO ][o.o.s.l.BuiltinLogTypeLoader] [node-1] Loaded [ad_ldap_logtype.json] log type
[2024-07-02T06:04:48,965][INFO ][o.o.t.TransportService ] [node-1] publish_address {10.4.6.4:9300}, bound_addresses {10.4.6.4:9300}
[2024-07-02T06:04:48,967][INFO ][o.o.t.TransportService ] [node-1] Remote clusters initialized successfully.
[2024-07-02T06:04:49,396][INFO ][o.o.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-07-02T06:04:49,409][INFO ][o.o.c.c.Coordinator ] [node-1] cluster UUID [3PAQtzA6R8Sxwu__yF2cpg]
[2024-07-02T06:04:50,572][INFO ][o.o.c.s.ClusterApplierService] [node-1] cluster-manager node changed {previous [], current [{node-2}{apW2rGUfRk2w6K3kaaOQUw}{yeZkVT0IRROVn0UScZR8Cw}{10.4.6.5}{10.4.6.5:9300}{dimr}{shard_indexing_pressure_enabled=true}]}, added {{node-2}{apW2rGUfRk2w6K3kaaOQUw}{yeZkVT0IRROVn0UScZR8Cw}{10.4.6.5}{10.4.6.5:9300}{dimr}{shard_indexing_pressure_enabled=true}}, term: 19, version: 792, reason: ApplyCommitRequest{term=19, version=792, sourceNode={node-2}{apW2rGUfRk2w6K3kaaOQUw}{yeZkVT0IRROVn0UScZR8Cw}{10.4.6.5}{10.4.6.5:9300}{dimr}{shard_indexing_pressure_enabled=true}}
[2024-07-02T06:04:50,594][INFO ][o.o.c.s.ClusterSettings ] [node-1] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2024-07-02T06:04:50,636][INFO ][o.o.a.c.HashRing ] [node-1] Node added: [apW2rGUfRk2w6K3kaaOQUw, yeraWVnORKeSSQOiEEmhOw]
[2024-07-02T06:04:50,647][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Cluster node changed, node removed: false, node added: true
[2024-07-02T06:04:50,649][INFO ][o.o.a.c.HashRing ] [node-1] AD version hash ring change is in progress. Can't build hash ring for node delta event.
[2024-07-02T06:04:50,649][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Hash ring build result: false
[2024-07-02T06:04:50,650][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-07-02T06:04:50,655][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: apW2rGUfRk2w6K3kaaOQUw
[2024-07-02T06:04:50,657][INFO ][o.o.a.c.HashRing ] [node-1] Add data node to AD version hash ring: yeraWVnORKeSSQOiEEmhOw
[2024-07-02T06:04:50,658][INFO ][o.o.a.c.HashRing ] [node-1] All nodes with known AD version: {apW2rGUfRk2w6K3kaaOQUw=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}, yeraWVnORKeSSQOiEEmhOw=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}}
[2024-07-02T06:04:50,658][INFO ][o.o.a.c.HashRing ] [node-1] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 1
[2024-07-02T06:04:50,664][INFO ][o.o.a.c.HashRing ] [node-1] Build AD version hash ring successfully
[2024-07-02T06:04:50,667][INFO ][o.o.a.c.ADDataMigrator ] [node-1] Start migrating AD data
[2024-07-02T06:04:50,668][INFO ][o.o.a.c.ADDataMigrator ] [node-1] AD job index doesn't exist, no need to migrate
[2024-07-02T06:04:50,674][INFO ][o.o.a.c.ADClusterEventListener] [node-1] Init AD version hash ring successfully
[2024-07-02T06:04:50,709][INFO ][o.o.m.a.MLModelAutoReDeployer] [node-1] Model auto reload configuration is false, not performing auto reloading!
[2024-07-02T06:04:50,728][INFO ][o.o.d.PeerFinder ] [node-1] setting findPeersInterval to [1s] as node commission status = [true] for local node [{node-1}{yeraWVnORKeSSQOiEEmhOw}{FjRsCFIyQ5uaEWOd6VMYow}{10.4.6.4}{10.4.6.4:9300}{dimr}{shard_indexing_pressure_enabled=true}]
[2024-07-02T06:04:50,752][INFO ][o.o.h.AbstractHttpServerTransport] [node-1] publish_address {10.4.6.4:9200}, bound_addresses {10.4.6.4:9200}
[2024-07-02T06:04:50,752][INFO ][o.o.n.Node ] [node-1] started
[2024-07-02T06:04:50,760][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Node started
[2024-07-02T06:04:50,761][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2024-07-02T06:04:50,780][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] 0 OpenSearch Security modules loaded so far: []
[2024-07-02T06:04:50,792][INFO ][o.o.s.l.LogTypeService ] [node-1] Loading builtin types!
[2024-07-02T06:04:50,798][INFO ][o.o.s.l.LogTypeService ] [node-1] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-07-02T06:04:50,798][WARN ][o.o.p.c.s.h.ConfigOverridesClusterSettingHandler] [node-1] Config override setting update called with empty string. Ignoring.
[2024-07-02T06:04:50,805][INFO ][o.o.s.c.ConfigurationRepository] [node-1] Background init thread started. Install default config?: false
[2024-07-02T06:04:50,808][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [node-1] Detected cluster change event for destination migration
[2024-07-02T06:04:50,837][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@1141244] retrieving configuration for [INTERNALUSERS, ACTIONGROUPS, CONFIG, ROLES, ROLESMAPPING, TENANTS, NODESDN, WHITELIST, ALLOWLIST, AUDIT] (index=.opendistro_security)
Nicolas Agustin Guevara Pihen
Jul 8, 2024, 2:05:43 PM7/8/24
to Wazuh | Mailing List
Hi,
I understand that you had a cluster, it failed due to a folder's deletion. After that, you reinstalled only one of the nodes and now you are not being able to run the security init script. Is that correct?
I understand that you had a cluster, it failed due to a folder's deletion. After that, you reinstalled only one of the nodes and now you are not being able to run the security init script. Is that correct?
wazuh
Jul 9, 2024, 4:27:26 AM7/9/24
to Wazuh | Mailing List
Hi,
I've managed to fix it. the issue of not being able to run the security init script was due to the _states folder still having some of the old cluster files. I've backed up my /var/lib/wazuh-indexer/nodes/0 folder and deleted it so I could make a fresh reinitialization of the cluster.
Nicolas Agustin Guevara Pihen
Jul 11, 2024, 1:42:37 PM7/11/24
to Wazuh | Mailing List
I'm glad to know that the problem is now fixed.
For future reference, you can delete indices from the Wazuh Indexer API (or the from the Dashboard in Dev Tools), and use wildcards. This way is cleaner than deleting the folder, and is less probable to produce an error.
For example :
DELETE wazuh-alerts-4.x-2024* will delete all the wazuh-alerts indices from 2024.
I hope you find this information helpful!
For future reference, you can delete indices from the Wazuh Indexer API (or the from the Dashboard in Dev Tools), and use wildcards. This way is cleaner than deleting the folder, and is less probable to produce an error.
For example :
DELETE wazuh-alerts-4.x-2024* will delete all the wazuh-alerts indices from 2024.
I hope you find this information helpful!
Reply all
Reply to author
Forward
0 new messages