4.7.2 version was working flawlessly. cron job performed its weekly `apt update` on all systems. This upgraded wazuh from 4.7.2 to 4.8. After which, wazuh-dashboard started throwing `ECONN REFUSED` errors. I checked for indexer service on port 9200. Nothing.
[2024-06-25T06:34:58,079][INFO ][o.o.h.AbstractHttpServerTransport] [scanner] publish_address {
192.168.100.16:9200}, bound_addresses {
192.168.100.16:9200}
[2024-06-25T06:34:58,080][INFO ][o.o.n.Node ] [scanner] started
[2024-06-25T06:34:58,085][INFO ][o.o.s.OpenSearchSecurityPlugin] [scanner] Node started
[2024-06-25T06:34:58,085][INFO ][o.o.s.c.ConfigurationRepository] [scanner] Will not attempt to create index .opendistro_security and default configs if they are absent. Use securityadmin to initialize cluster
[2024-06-25T06:34:58,097][INFO ][o.o.s.c.ConfigurationRepository] [scanner] Background init thread started. Install default config?: false
[2024-06-25T06:34:58,097][INFO ][o.o.s.c.ConfigurationRepository] [scanner] Wait for cluster to be available ...
[2024-06-25T06:34:58,102][INFO ][o.o.s.OpenSearchSecurityPlugin] [scanner] 0 OpenSearch Security modules loaded so far: []
[2024-06-25T06:34:58,738][INFO ][o.o.c.s.ClusterSettings ] [scanner] updating [plugins.index_state_management.template_migration.control] from [0] to [-1]
[2024-06-25T06:34:58,800][INFO ][o.o.a.c.HashRing ] [scanner] Node added: [NYwAnnogQQGj1f5CMYjJoQ]
[2024-06-25T06:34:58,804][INFO ][o.o.a.c.HashRing ] [scanner] Add data node to AD version hash ring: NYwAnnogQQGj1f5CMYjJoQ
[2024-06-25T06:34:58,806][INFO ][o.o.a.c.HashRing ] [scanner] All nodes with known AD version: {NYwAnnogQQGj1f5CMYjJoQ=ADNodeInfo{version=2.10.0, isEligibleDataNode=true}}
[2024-06-25T06:34:58,806][INFO ][o.o.a.c.HashRing ] [scanner] Rebuild AD hash ring for realtime AD with cooldown, nodeChangeEvents size 0
[2024-06-25T06:34:58,806][INFO ][o.o.a.c.HashRing ] [scanner] Build AD version hash ring successfully
[2024-06-25T06:34:58,807][INFO ][o.o.a.c.ADDataMigrator ] [scanner] Start migrating AD data
[2024-06-25T06:34:58,807][INFO ][o.o.a.c.ADDataMigrator ] [scanner] AD job index doesn't exist, no need to migrate
[2024-06-25T06:34:58,807][INFO ][o.o.a.c.ADClusterEventListener] [scanner] Init AD version hash ring successfully
[2024-06-25T06:34:58,811][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [scanner] Detected cluster change event for destination migration
[2024-06-25T06:34:58,833][INFO ][o.o.g.GatewayService ] [scanner] recovered [211] indices into cluster_state
[2024-06-25T06:34:58,835][INFO ][o.o.s.l.LogTypeService ] [scanner] Loading builtin types!
[2024-06-25T06:34:58,837][WARN ][o.o.o.i.ObservabilityIndex] [scanner] message: index [.opensearch-observability/NxSo7GFITU-77U-NQB8HJQ] already exists
[2024-06-25T06:34:58,837][INFO ][o.o.o.i.ObservabilityIntegrationsIndex] [scanner] observability:createMappingTemplate ss4o_metrics_template API called
[2024-06-25T06:34:58,840][INFO ][o.o.s.l.LogTypeService ] [scanner] Indexing [418] fieldMappingDocs from logTypes: 23
[2024-06-25T06:34:58,867][ERROR][o.o.b.Bootstrap ] [scanner] Exception
java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:560) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:493) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.10.0.jar:2.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) ~[?:?]
[2024-06-25T06:34:58,872][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [scanner] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:184) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.10.0.jar:2.10.0]
at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-2.10.0.jar:2.10.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103) ~[opensearch-2.10.0.jar:2.10.0]
Caused by: java.lang.IllegalArgumentException: index template [ss4o_metrics_template] has index patterns [ss4o_metrics-*-*] matching patterns from existing templates [ss4o_metric_template] with patterns (ss4o_metric_template => [ss4o_metrics-*-*]) that have the same priority [1], multiple index templates may not match during index creation, please use a different priority
at org.opensearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:560) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.metadata.MetadataIndexTemplateService$4.execute(MetadataIndexTemplateService.java:493) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:65) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService.executeTasks(MasterService.java:874) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:424) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService.runTasks(MasterService.java:295) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.MasterService$Batcher.run(MasterService.java:206) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:204) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:242) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:849) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedOpenSearchThreadPoolExecutor.java:282) ~[opensearch-2.10.0.jar:2.10.0]
at org.opensearch.common.util.concurrent.PrioritizedOpenSearchThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedOpenSearchThreadPoolExecutor.java:245) ~[opensearch-2.10.0.jar:2.10.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
[2024-06-25T06:34:58,878][WARN ][o.o.s.SecurityAnalyticsPlugin] [scanner] Failed to initialize LogType config index and builtin log types