Wazuh Primary Indexer Failed State

[Krishna Prasad Bhandary]

Hi all, we are currently facing an issue with our indexer. We did 2 things, one is reduce our Indexer Policy's retention period from 60 days down to 45 days. Second, we changed our passwords on the indexer using the wazuh-password-tool.sh bash file from the Wazuh documentation. We did not run the command on the Wazuh server. We updated the filebeat keystore to hold the new admin password generated and updated the kibanaserver password.
Originally we were getting errors of this kind

[2025-12-09T16:19:17,516][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:20:04,499][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012

So we restarted all components including our indexer. The dashboard and server work fine but the Wazuh Indexer is not restarting, it is in the failed state and we are getting the following errors.

[2025-12-09T16:20:45,882][ERROR][o.o.i.i.ManagedIndexRunner] [indexer-1] Failed to save ManagedIndexMetaData for [index=wazuh-alerts-4.x-2025.10.21]
[2025-12-09T16:20:45,883][ERROR][o.o.i.i.ManagedIndexRunner] [indexer-1] Failed to update ManagedIndexMetaData after executing the Step : attempt_transition_step
[2025-12-09T16:20:46,412][ERROR][o.o.j.s.u.LockService    ] [indexer-1] Exception occurred finding lock
[2025-12-09T16:20:46,430][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [indexer-1] uncaught exception in thread [DefaultDispatcher-worker-7]

For context we have a multi node cluster running with 2 indexers and 1 server with the dashboard running on the primary indexer. The secondary indexer is still running but when we try to make any cURL requests we get this

root@ip-wazuh-secondary-indexer-ip:/home/telpsa# curl -XGET https://wazuh-primary-indexer-ip:9200/_cluster/health?pretty -u 'admin:adminpwd' -k
curl: (7) Failed to connect to wazuh-primary-indexer-ip port 9200: Connection refused
root@ip-wazuh-secondary-indexer-ip:/home/telpsa# curl -XGET https://wazuh-secondary-indexer-ip:9200/_cluster/health?pretty -u 'admin:adminpwd' -k
{
  "error" : {
    "root_cause" : [
      {
        "type" : "cluster_manager_not_discovered_exception",
        "reason" : null
      }
    ],
    "type" : "cluster_manager_not_discovered_exception",
    "reason" : null
  },
  "status" : 503
}

And the output of the cluster log for the secondary indexer is below:

2025-12-10T11:37:30,281][WARN ][o.o.c.c.ClusterFormationFailureHelper] [indexer-2] cluster-manager not discovered or elected yet, an election requires a node with id [QU1TSbxUS86DnFrutS9O6w], have discovered [{indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [wazuh-primary-indexer-ip:9300] from hosts providers and [{indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}, {indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 108, last-accepted version 65684 in term 108
[2025-12-10T11:37:32,142][WARN ][r.suppressed             ] [indexer-2] path: /.kibana, params: {index=.kibana}
[2025-12-10T11:37:33,667][WARN ][o.o.p.c.u.JsonConverter  ] [indexer-2] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2025-12-10T11:37:38,667][WARN ][o.o.p.c.u.JsonConverter  ] [indexer-2] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2025-12-10T11:37:40,282][WARN ][o.o.c.c.ClusterFormationFailureHelper] [indexer-2] cluster-manager not discovered or elected yet, an election requires a node with id [QU1TSbxUS86DnFrutS9O6w], have discovered [{indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [wazuh-primary-indexer-ip:9300] from hosts providers and [{indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}, {indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 108, last-accepted version 65684 in term 108
[2025-12-10T11:37:43,668][WARN ][o.o.p.c.u.JsonConverter  ] [indexer-2] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2025-12-10T11:37:48,668][WARN ][o.o.p.c.u.JsonConverter  ] [indexer-2] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2025-12-10T11:37:50,283][WARN ][o.o.c.c.ClusterFormationFailureHelper] [indexer-2] cluster-manager not discovered or elected yet, an election requires a node with id [QU1TSbxUS86DnFrutS9O6w], have discovered [{indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [wazuh-primary-indexer-ip:9300] from hosts providers and [{indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}, {indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 108, last-accepted version 65684 in term 108
[2025-12-10T11:37:51,596][WARN ][o.o.s.a.BackendRegistry  ] [indexer-2] Authentication finally failed for admin from wazuh-server-ip:51340
[2025-12-10T11:37:53,669][WARN ][o.o.p.c.u.JsonConverter  ] [indexer-2] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2025-12-10T11:37:55,712][WARN ][o.o.c.NodeConnectionsService] [indexer-2] failed to connect to {indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true} (tried [6943] times)
[2025-12-10T11:37:58,669][WARN ][o.o.p.c.u.JsonConverter  ] [indexer-2] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])

I looked at the internal_users.yml (path: /etc/wazuh-indexer/opensearch-security/internal_users.yml) for both indexers and found that the hashes for the users dont match for any of the users, could this be the reason?

Would appreciate all the help we can get, thank you!

[Bony V John]

Hi,

Please allow me some time, I'm working on this and will get back to you with an update as soon as possible.

[Bony V John]

Hi,

From the curl output you shared, it seems that the Wazuh primary indexer is down. Because of this, the secondary indexer node is failing to connect to the primary node.

First, make sure that both Wazuh indexer servers have enough resources:

Disk usage: - df -h

Memory usage: free -h

CPU usage: top

If the server does not have enough resources, allocate resources properly to the indexer nodes.

Check the status of the Wazuh primary indexer node:

systemctl status wazuh-indexer

If it is inactive, restart it:

systemclt restart wazuh-indexer

If it still fails, disable the ILM policy plugin temporarily for troubleshooting:

Edit the file on the Wazuh indexer node: /etc/wazuh-indexer/opensearch.yml 

Add the following lines:  

plugins.index_state_management.enabled: false
plugins.index_state_management.history.enabled: false

This will help disable the plugin, as the primary indexer is failing to load ManagedIndexMetaData.

Then restart the indexer again and check if the primary node starts:

systemctl restart wazuh-indexer

If it starts successfully, the issue is likely related to the ILM policy configuration. In that case, verify and correct your ILM policies.

If it still does not start, share the Wazuh primary indexer log file with us: /var/log/wazuh-indexer/wazuh-cluster.log

First, we need to restore the Wazuh primary indexer to an active state. Once it is up, we can proceed to resolve the password issue.

[Krishna Prasad Bhandary]

Hi,

We added the lines regarding the index state management in the opensearch.yml file and restarted the primary indexer but it failed to restart. The whole cluster log file seems to be too big to share, attaching logs from the last 20 minutes before we applied the new policy to existing indices are given below. We are unable to attach the file due to our IT policy so we are attaching the raw logs

[2025-12-09T15:59:44,835][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:00:00,875][INFO ][o.o.c.m.MetadataUpdateSettingsService] [indexer-1] updating number_of_replicas to [0] for indices [wazuh-monitoring-2025.50w]
[2025-12-09T16:00:01,227][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:00:04,001][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 60242 miliseconds for next execution of job wazuh-monitoring-2025.49w
[2025-12-09T16:00:04,898][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:00:04,898][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:00:18,590][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 25124 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:00:19,247][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:00:19,247][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:00:19,318][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 12570 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:00:20,262][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:00:20,262][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:00:27,272][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 30472 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:00:28,104][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:00:28,104][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:00:34,379][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:00:39,149][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 37914 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:00:39,479][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:00:39,479][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:00:54,596][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 78548 miliseconds for next execution of job wazuh-monitoring-2025.50w
[2025-12-09T16:00:55,007][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 115744 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:00:55,538][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:00:55,538][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:00:55,538][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:00:55,538][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:00:56,633][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 164611 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:00:57,570][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:00:57,571][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:00:57,876][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 168500 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:00:58,605][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:00:58,605][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:01:10,044][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 119540 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:01:10,585][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 85000 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:01:10,636][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:01:10,636][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:01:11,448][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:01:11,448][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:01:14,550][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 46739 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:01:15,479][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:01:15,479][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:01:18,691][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 22579 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:01:19,694][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:01:19,694][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:01:24,209][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:01:27,180][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 160944 miliseconds for next execution of job wazuh-monitoring-2025.42w
[2025-12-09T16:01:27,771][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:01:27,771][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:01:29,582][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 99559 miliseconds for next execution of job wazuh-monitoring-2025.48w
[2025-12-09T16:01:29,800][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:01:29,800][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:01:46,374][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 123788 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:01:46,927][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:01:46,927][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:01:49,623][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 32210 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:01:49,956][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:01:49,956][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:01:57,545][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 165420 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:01:58,020][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:01:58,020][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:01:59,133][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:02:02,146][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 144840 miliseconds for next execution of job wazuh-statistics-2025.49w
[2025-12-09T16:02:03,049][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:02:03,049][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:02:23,265][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 167611 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:02:24,186][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:02:24,187][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:02:28,753][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 56183 miliseconds for next execution of job wazuh-monitoring-2025.46w
[2025-12-09T16:02:29,216][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:02:29,216][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:02:32,162][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 31717 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:02:33,158][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:02:33,158][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:02:36,794][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 126268 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:02:37,294][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:02:37,294][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:02:39,233][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:02:39,419][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 113147 miliseconds for next execution of job wazuh-statistics-2025.50w
[2025-12-09T16:02:40,322][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:02:40,322][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:02:44,149][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 12869 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:02:44,351][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:02:44,351][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:02:46,971][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 94743 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:02:47,380][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:02:47,380][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:02:47,718][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 56875 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:02:48,395][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:02:48,395][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:02:49,133][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 12275 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:02:49,429][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:02:49,429][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:02:58,990][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 129723 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:02:59,515][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:02:59,515][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:03:11,855][INFO ][o.o.j.s.JobSweeper       ] [indexer-1] Running full sweep
[2025-12-09T16:03:16,591][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 96583 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:03:17,565][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:03:17,565][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:03:19,194][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:03:52,117][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:04:31,999][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:05:12,802][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 167184 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:05:13,651][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:05:13,651][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:05:13,947][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 108527 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:05:14,666][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:05:14,666][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:05:15,157][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 51635 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:05:15,681][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:05:15,681][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:05:23,747][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 116170 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:05:24,747][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:05:24,747][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:05:25,297][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 166319 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:05:25,771][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:05:25,771][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:05:30,483][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:05:31,805][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 47811 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:05:32,817][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:05:32,817][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:05:33,409][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 12205 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:05:33,833][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:05:33,833][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:05:34,041][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 7899 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:05:34,848][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:05:34,848][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:05:38,310][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 84953 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:05:38,903][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:05:38,904][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:05:48,600][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 171433 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:05:48,981][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:05:48,981][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:05:52,406][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 36815 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:05:53,060][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:05:53,061][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:05:56,572][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 172165 miliseconds for next execution of job wazuh-monitoring-2025.46w
[2025-12-09T16:05:57,090][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:05:57,090][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:05:57,531][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 32863 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:05:58,109][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:05:58,109][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:06:00,732][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 155936 miliseconds for next execution of job wazuh-monitoring-2025.49w
[2025-12-09T16:06:01,163][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:06:01,163][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:06:13,460][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:06:19,071][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 92381 miliseconds for next execution of job wazuh-monitoring-2025.50w
[2025-12-09T16:06:19,306][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:06:19,306][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:06:25,512][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 114799 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:06:26,351][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:06:26,351][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:06:36,553][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 84442 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:06:37,415][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:06:37,416][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:06:53,637][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 54428 miliseconds for next execution of job wazuh-statistics-2025.50w
[2025-12-09T16:06:54,454][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:06:54,454][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:06:56,649][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 114032 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:06:57,482][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:06:57,483][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:07:00,534][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 121745 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:07:01,528][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:07:01,528][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:07:04,640][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 71759 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:07:05,583][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:07:05,583][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:07:08,244][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 40401 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:07:08,614][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:07:08,614][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:07:12,060][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 96369 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:07:12,342][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:07:12,645][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:07:12,646][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:07:26,270][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 141234 miliseconds for next execution of job wazuh-statistics-2025.49w
[2025-12-09T16:07:26,741][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:07:26,742][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:07:40,489][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 4835 miliseconds for next execution of job wazuh-monitoring-2025.48w
[2025-12-09T16:07:40,853][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:07:40,854][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:07:42,033][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 171285 miliseconds for next execution of job wazuh-monitoring-2025.42w
[2025-12-09T16:07:42,884][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:07:42,884][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:07:47,280][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 69316 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:07:47,946][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:07:47,946][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:07:49,552][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 151586 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:07:50,011][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:07:50,011][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:07:56,447][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:08:11,856][INFO ][o.o.j.s.JobSweeper       ] [indexer-1] Running full sweep
[2025-12-09T16:08:18,394][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 28947 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:08:19,187][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:08:19,187][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:08:38,615][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 99542 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:08:39,221][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:08:39,221][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:08:44,610][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 10799 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:08:45,271][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:08:45,271][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:08:47,915][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:09:44,412][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:10:09,731][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 169710 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:10:10,344][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:10:10,344][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:10:13,897][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 172760 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:10:14,439][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:10:14,440][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:10:23,912][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:10:33,519][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 114709 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:10:34,504][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:10:34,504][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:10:42,376][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 164321 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:10:42,571][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:10:42,571][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:10:49,145][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 106056 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:10:49,618][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:10:49,619][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:10:53,923][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 28391 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:10:54,651][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:10:54,651][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:10:54,919][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 93900 miliseconds for next execution of job wazuh-statistics-2025.50w
[2025-12-09T16:10:55,667][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:10:55,667][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:10:58,286][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:10:59,729][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 170567 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:11:00,757][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:11:00,757][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:11:05,766][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 17347 miliseconds for next execution of job wazuh-monitoring-2025.48w
[2025-12-09T16:11:06,661][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:11:06,661][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:11:11,176][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 130471 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:11:11,855][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:11:11,855][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:11:12,611][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 39385 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:11:12,871][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:11:12,871][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:11:25,349][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 86773 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:11:25,904][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:11:25,904][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:11:26,251][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 99715 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:11:26,920][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:11:26,920][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:11:32,904][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 2078 miliseconds for next execution of job wazuh-monitoring-2025.50w
[2025-12-09T16:11:33,804][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:11:33,805][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:11:36,001][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:11:38,705][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 36688 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:11:39,022][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:11:39,022][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:11:49,903][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 85402 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:11:50,917][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:11:50,917][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:11:54,937][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 126863 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:11:55,170][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:11:55,170][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:11:55,310][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 166449 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:11:56,186][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:11:56,187][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:11:57,338][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 32398 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:11:58,217][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:11:58,217][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:12:02,739][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 104693 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:12:03,249][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:12:03,249][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:12:22,665][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 135503 miliseconds for next execution of job wazuh-statistics-2025.49w
[2025-12-09T16:12:23,430][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:12:23,430][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:12:32,638][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 23824 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:12:32,936][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:12:33,493][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:12:33,493][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:12:36,425][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 122275 miliseconds for next execution of job wazuh-monitoring-2025.49w
[2025-12-09T16:12:37,335][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:12:37,335][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:12:46,493][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 94576 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:12:47,365][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:12:47,366][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:12:47,710][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 150721 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:12:48,620][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:12:48,620][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:12:52,374][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 7606 miliseconds for next execution of job wazuh-monitoring-2025.42w
[2025-12-09T16:12:52,554][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 52707 miliseconds for next execution of job wazuh-monitoring-2025.46w
[2025-12-09T16:12:52,687][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:12:52,687][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:12:53,293][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 46289 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:12:53,475][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:12:53,475][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:12:53,727][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:12:53,727][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:13:11,680][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [indexer-1] Starting housekeeping task for auto refresh streaming jobs.
[2025-12-09T16:13:11,681][INFO ][o.o.s.s.c.FlintStreamingJobHouseKeeperTask] [indexer-1] Finished housekeeping task for auto refresh streaming jobs.
[2025-12-09T16:13:11,856][INFO ][o.o.j.s.JobSweeper       ] [indexer-1] Running full sweep
[2025-12-09T16:13:15,632][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:13:18,593][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 115163 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:13:18,930][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:13:18,931][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:13:41,574][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 175609 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:13:41,967][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:13:41,967][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:14:08,324][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:14:40,554][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:15:00,560][INFO ][o.o.c.m.MetadataUpdateSettingsService] [indexer-1] updating number_of_replicas to [0] for indices [wazuh-monitoring-2025.50w]
[2025-12-09T16:15:02,601][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 31514 miliseconds for next execution of job wazuh-monitoring-2025.50w
[2025-12-09T16:15:03,015][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:15:03,015][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:15:08,695][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 27824 miliseconds for next execution of job wazuh-monitoring-2025.42w
[2025-12-09T16:15:09,086][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:15:09,086][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:15:13,541][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:15:24,877][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 83912 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:15:25,224][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:15:25,224][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.12
[2025-12-09T16:15:30,680][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 21514 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:15:31,260][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:15:31,260][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:15:33,565][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 88205 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:15:34,315][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:15:34,315][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.29
[2025-12-09T16:15:39,024][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 84038 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:15:39,378][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:15:39,378][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.16
[2025-12-09T16:15:40,237][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 43802 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:15:41,140][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:15:41,140][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:15:48,149][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 110350 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:15:48,432][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:15:48,433][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.15
[2025-12-09T16:15:53,096][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 10720 miliseconds for next execution of job wazuh-monitoring-2025.46w
[2025-12-09T16:15:53,510][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:15:53,511][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:16:03,666][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:16:18,278][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 75199 miliseconds for next execution of job wazuh-monitoring-2025.48w
[2025-12-09T16:16:18,715][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:16:18,715][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.48w
[2025-12-09T16:16:26,778][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 45034 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:16:27,169][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 93422 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:16:27,659][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:16:27,659][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.20
[2025-12-09T16:16:27,876][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:16:27,877][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.12
[2025-12-09T16:16:34,391][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 19201 miliseconds for next execution of job wazuh-statistics-2025.50w
[2025-12-09T16:16:34,749][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 139777 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:16:34,932][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:16:34,932][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:16:35,738][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:16:35,738][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.02
[2025-12-09T16:16:41,525][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 148101 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:16:41,967][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:16:41,967][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.30
[2025-12-09T16:16:45,687][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 123262 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:16:46,001][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:16:46,002][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.20
[2025-12-09T16:16:47,390][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 56170 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:16:48,036][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:16:48,036][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.17
[2025-12-09T16:16:50,501][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:16:55,364][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 168316 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:16:56,098][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:16:56,098][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.07
[2025-12-09T16:17:02,764][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 116944 miliseconds for next execution of job wazuh-monitoring-2025.49w
[2025-12-09T16:17:03,155][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:17:03,155][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.49w
[2025-12-09T16:17:07,767][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 97387 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:17:08,194][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:17:08,194][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.01
[2025-12-09T16:17:12,331][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 101122 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:17:13,270][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:17:13,270][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.13
[2025-12-09T16:17:16,934][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 80023 miliseconds for next execution of job wazuh-statistics-2025.49w
[2025-12-09T16:17:17,323][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:17:17,323][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.49w
[2025-12-09T16:17:28,144][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [indexer-1] Detected cluster change event for destination migration
[2025-12-09T16:17:28,144][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [indexer-1] Reset destination migration process.
[2025-12-09T16:17:31,248][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 175957 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:17:31,357][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:17:32,187][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:17:32,187][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.04
[2025-12-09T16:17:46,296][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 162587 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:17:46,960][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 138556 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:17:47,284][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:17:47,285][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.23
[2025-12-09T16:17:47,530][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:17:47,530][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.23
[2025-12-09T16:17:51,542][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 98642 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:17:52,319][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:17:52,319][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.21
[2025-12-09T16:17:54,452][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 9274 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:17:55,355][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:17:55,355][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:18:11,856][INFO ][o.o.j.s.JobSweeper       ] [indexer-1] Running full sweep
[2025-12-09T16:18:21,349][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 122015 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:18:21,687][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:18:21,687][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.24
[2025-12-09T16:18:23,116][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-09T16:18:28,144][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [indexer-1] Performing migration of destination data.
[2025-12-09T16:18:28,150][INFO ][k.j.i.ClassReference     ] [indexer-1] Need to migrate 0 email accounts, 0 email groups and 0 destinations (0 configs total)

[2025-12-09T16:19:17,516][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012

[2025-12-09T16:19:28,145][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [indexer-1] Cancel background destination migration process.
[2025-12-09T16:19:28,145][INFO ][o.o.a.u.d.DestinationMigrationCoordinator] [indexer-1] Performing migration of destination data.
[2025-12-09T16:19:28,145][INFO ][k.j.i.ClassReference     ] [indexer-1] Destination migration has finished.
[2025-12-09T16:19:57,640][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 70352 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:19:58,558][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:19:58,558][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.27
[2025-12-09T16:20:02,958][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 21151 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:20:03,887][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25
[2025-12-09T16:20:03,887][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.25

[2025-12-09T16:20:04,499][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012

[2025-12-09T16:20:10,966][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 100643 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:20:11,109][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 161672 miliseconds for next execution of job wazuh-monitoring-2025.46w
[2025-12-09T16:20:11,923][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:20:11,923][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:20:11,923][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.46w
[2025-12-09T16:20:11,923][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.22
[2025-12-09T16:20:19,693][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 79487 miliseconds for next execution of job wazuh-statistics-2025.50w
[2025-12-09T16:20:20,017][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:20:20,017][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-statistics-2025.50w
[2025-12-09T16:20:23,802][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 53129 miliseconds for next execution of job wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:20:24,796][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:20:24,796][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.11.05
[2025-12-09T16:20:28,913][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 32347 miliseconds for next execution of job wazuh-monitoring-2025.42w
[2025-12-09T16:20:29,853][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:20:29,853][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.42w
[2025-12-09T16:20:32,038][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 60275 miliseconds for next execution of job wazuh-monitoring-2025.50w
[2025-12-09T16:20:32,889][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:20:32,889][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-monitoring-2025.50w
[2025-12-09T16:20:44,654][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 39052 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:20:45,236][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:20:45,236][INFO ][o.o.i.i.ManagedIndexRunner] [indexer-1] Finished executing attempt_transition_step for wazuh-alerts-4.x-2025.10.21
[2025-12-09T16:20:45,864][INFO ][o.o.n.Node               ] [indexer-1] stopping ...
[2025-12-09T16:20:45,864][INFO ][o.o.s.a.r.AuditMessageRouter] [indexer-1] Closing AuditMessageRouter
[2025-12-09T16:20:45,864][INFO ][o.o.s.a.s.SinkProvider   ] [indexer-1] Closing DebugSink
[2025-12-09T16:20:45,869][INFO ][o.o.c.c.FollowersChecker ] [indexer-1] FollowerChecker{discoveryNode={indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} disconnected
[2025-12-09T16:20:45,870][INFO ][o.o.c.c.FollowersChecker ] [indexer-1] FollowerChecker{discoveryNode={indexer-2}{8c6Tf5IzQVGkPv2uY9ZBug}{jA_Qnzv1QFaHFpCs4G9NSw}{wazuh-secondary-indexer-ip}{wazuh-secondary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}, failureCountSinceLastSuccess=0, [cluster.fault_detection.follower_check.retry_count]=3} marking node as faulty

[2025-12-09T16:20:45,882][ERROR][o.o.i.i.ManagedIndexRunner] [indexer-1] Failed to save ManagedIndexMetaData for [index=wazuh-alerts-4.x-2025.10.21]

org.opensearch.node.NodeClosedException: node closed {indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$2.onClusterServiceClose(TransportReplicationAction.java:1192) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:276) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.service.ClusterApplierService.addTimeoutListener(ClusterApplierService.java:304) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:203) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:144) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.cluster.ClusterStateObserver.waitForNextChange(ClusterStateObserver.java:136) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase.retry(TransportReplicationAction.java:1184) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.replication.TransportReplicationAction$ReroutePhase$1.handleException(TransportReplicationAction.java:1163) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService$9.handleException(TransportService.java:1729) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor$RestoringTransportResponseHandler.handleException(SecurityInterceptor.java:404) ~[?:?]
        at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1515) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService$4.doRun(TransportService.java:374) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.13.0.jar:2.13.0]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
        at java.base/java.lang.Thread.run(Thread.java:1583) ~[?:?]

[2025-12-09T16:20:45,883][ERROR][o.o.i.i.ManagedIndexRunner] [indexer-1] Failed to update ManagedIndexMetaData after executing the Step : attempt_transition_step

[2025-12-09T16:20:46,411][INFO ][o.o.j.s.JobScheduler     ] [indexer-1] Will delay 151136 miliseconds for next execution of job wazuh-alerts-4.x-2025.10.20

[2025-12-09T16:20:46,412][ERROR][o.o.j.s.u.LockService    ] [indexer-1] Exception occurred finding lock

org.opensearch.transport.SendRequestTransportException: [indexer-1][wazuh-primary-indexer-ip:9300][indices:data/read/get[s]]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:844) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:280) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:239) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:303) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:841) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:280) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:231) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:125) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:78) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) [opensearch-security-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.support.AbstractClient.get(AbstractClient.java:572) [opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.findLock(LockService.java:247) [opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.lambda$acquireLockWithId$3(LockService.java:147) [opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) [opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.createLockIndex(LockService.java:81) [opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.acquireLockWithId(LockService.java:144) [opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.acquireLock(LockService.java:115) [opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1$lock$1.invoke(ManagedIndexRunner.kt:238) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1$lock$1.invoke(ManagedIndexRunner.kt:238) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:233) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend(ManagedIndexRunner.kt:238) [opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        ... 43 more

[2025-12-09T16:20:46,430][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [indexer-1] uncaught exception in thread [DefaultDispatcher-worker-7]

org.opensearch.transport.SendRequestTransportException: [indexer-1][wazuh-primary-indexer-ip:9300][indices:data/read/get[s]]
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:1002) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.transport.SecurityInterceptor.sendRequestDecorate(SecurityInterceptor.java:267) ~[?:?]
        at org.opensearch.security.OpenSearchSecurityPlugin$6$2.sendRequest(OpenSearchSecurityPlugin.java:847) ~[?:?]
        at org.opensearch.transport.TransportService.sendRequestAsync(TransportService.java:1750) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:885) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:844) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:280) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.onFailure(TransportSingleShardAction.java:239) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction$2.handleException(TransportSingleShardAction.java:303) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.transport.TransportService.sendRequest(TransportService.java:841) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.perform(TransportSingleShardAction.java:280) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction$AsyncSingleAction.start(TransportSingleShardAction.java:231) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:125) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.single.shard.TransportSingleShardAction.doExecute(TransportSingleShardAction.java:78) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:324) ~[?:?]
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165) ~[?:?]
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.client.support.AbstractClient.get(AbstractClient.java:572) ~[opensearch-2.13.0.jar:2.13.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.findLock(LockService.java:247) ~[opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.lambda$acquireLockWithId$3(LockService.java:147) ~[opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-2.13.0.jar:2.13.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.createLockIndex(LockService.java:81) ~[opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.acquireLockWithId(LockService.java:144) ~[opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.jobscheduler.spi.utils.LockService.acquireLock(LockService.java:115) ~[opensearch-job-scheduler-spi-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1$lock$1.invoke(ManagedIndexRunner.kt:238) ~[opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1$lock$1.invoke(ManagedIndexRunner.kt:238) ~[opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.opensearchapi.OpenSearchExtensionsKt.suspendUntil(OpenSearchExtensions.kt:233) ~[opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at org.opensearch.indexmanagement.indexstatemanagement.ManagedIndexRunner$runJob$1.invokeSuspend(ManagedIndexRunner.kt:238) ~[opensearch-index-management-2.13.0.0.jar:2.13.0.0]
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.8.21.jar:1.8.21-release-380(1.8.21)]
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:738) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665) [kotlinx-coroutines-core-jvm-1.3.9.jar:?]
Caused by: org.opensearch.node.NodeClosedException: node closed {indexer-1}{QU1TSbxUS86DnFrutS9O6w}{zc3cXJ6fRei4g0l-nO4WQQ}{wazuh-primary-indexer-ip}{wazuh-primary-indexer-ip:9300}{dimr}{shard_indexing_pressure_enabled=true}
        at org.opensearch.transport.TransportService.sendRequestInternal(TransportService.java:983) ~[opensearch-2.13.0.jar:2.13.0]
        ... 43 more
[2025-12-09T16:20:46,704][INFO ][o.o.n.Node               ] [indexer-1] stopped
[2025-12-09T16:20:46,704][INFO ][o.o.n.Node               ] [indexer-1] closing ...
[2025-12-09T16:20:46,709][INFO ][o.o.s.a.i.AuditLogImpl   ] [indexer-1] Closing AuditLogImpl
[2025-12-09T16:20:46,715][INFO ][o.o.n.Node               ] [indexer-1] closed

[Oscar Javier Alarcón Fierro]

help !!!!
Error: Error: An internal server error occurred. at WzRequest.genericReq (https://192.168.100.17/411003/bundles/plugin/wazuh/wazuh.plugin.js:1:496202) at async WzAuthentication.login (https://192.168.100.17/411003/bundles/plugin/wazuh/wazuh.plugin.js:1:508721) at async WzAuthentication.refresh (https://192.168.100.17/411003/bundles/plugin/wazuh/wazuh.plugin.js:1:509046)

IMPORTANTE: La información contenida en este mensaje y cualquier documento enviado al destinatario son propiedad del Servicio Geológico Colombiano, y se encuentra protegida por las normas atenientes a derechos de autor. Por tanto, el uso de esta información y sus anexos para propósitos ajenos al ejercicio del estudio o investigación para la cual fue solicitada, como su divulgación a personas a las cuales no se encuentre destinado este correo o su reproducción total o parcial o su distribución pública por cualquier medio, entre otros, no está autorizado por el Servicio Geológico Colombiano y constituye una violación a las normas de derechos de autor, ante lo cual el Servicio Geológico Colombiano estará legitimado para iniciar las acciones pertinentes previstas en la ley 23 de 1982, la ley 1915 de 2018 y demás normas concordantes, además de que la conducta del infractor podrá constituirse como un delito de conformidad con el Código Penal. Asimismo, el que ilícitamente eluda, evada, inutilice o suprima las medidas para evitar el uso no autorizado de esta información, estará sujeto a las sanciones penales correspondientes. Los servidores públicos que reciban este mensaje están obligados a custodiar y cuidar la documentación e información contenida o anexa al mismo, e impedir y evitar su sustracción, destrucción, ocultamiento o utilización indebidos de conformidad con lo previsto en el Código General Disciplinario. Si por error recibe este mensaje, le solicitamos enviarlo de vuelta al Servicio Geológico Colombiano a la dirección del emisor y borrarlo de sus archivos electrónicos o destruirlo. El receptor deberá verificar posibles virus informáticos que tenga el correo o cualquier anexo a él, razón por la cual el Servicio Geológico Colombiano no aceptará responsabilidad alguna por daños causados por cualquier virus transmitido en este correo.

El Servicio Geológico Colombiano, no asumirá responsabilidad, ni su institucionalidad se verá comprometida por información, opiniones o criterios emitidos por parte de terceros receptores del presente correo.

[Bony V John]

Hi Oscar, could you please create a new post by briefly mentioning your issue? This will help us keep the community threads clean and organized, and it will also allow us to focus on your issue separately. Thank you!

Hi Krishna,
The logs from before the issue occurred are not useful for finding the root cause. As mentioned before, we should focus on the primary indexer node.

So first, delete the current log file of the primary indexer node. This will help us avoid noisy old logs.

On the primary Wazuh indexer, run the below command to delete the old log file:

echo "" > /var/log/wazuh-indexer/wazuh-cluster.log

Then restart the Wazuh indexer service:

systemctl restart wazuh-indexer

Then share the new logs from the primary indexer with us:

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -iE "error|warn"

[Krishna Prasad Bhandary]

Hi, when we do this our log file does not output anything. (Output from terminal below). 
root@primary-indexer:/var/log/wazuh-indexer# cat /var/log/wazuh-indexer/wazuh-indexer-cluster.log

root@primary-indexer:/var/log/wazuh-indexer#

We have a snapshot of the indexer prior to updating and applying the policy to all indices. We will restore that and then try to bring back the communication between the Indexer-Server and Indexer-Dashboard by changing the passwords as mentioned in the first mail. Could we get some help on how to apply a policy where we reduce the retention period for the current and new indices as we feel that is the main cause of this error.

[Krishna Prasad Bhandary]

Hi Bony,
We dont see any logs added to the wazuh-cluster-log file on restart. We restored a snapshot from when the indexer policy was not applied but even that seems to give the same result. 

I had asked about this on the Slack as well and someone there mentioned to run the passwords-tool.sh file to reset the passwords. I am not sure how that could help but do we try that?

[Krishna Prasad Bhandary]

Hi Bony,

I wanted to provide an update. I think we have some errors to make sense of right now. I will be attaching the Cluster Logs (Error Only), Cluster Logs (Warning Only) and systemctl output. I have trimmed the logs to remove anything that was repeated a significant amount to make this more readable. Only the systemctl output remains unchanged.

The Cluster Logs (Error Only) has errors till 2nd December which was when we had taken our indexer snapshot which we restored back. Unsure if this will help but we are attaching the same, the Cluster Logs (Warning Only) is the Wazuh Server unable to establish a connection but on a port number we have not seen before, does this port need to be whitelisted?

Wazuh Indexer Error Logs

root@primary-indexer:/var/log/wazuh-indexer# cat wazuh-indexer-cluster.log | grep -i -E "error"
[2025-12-02T09:42:35,037][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [indexer-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
[2025-12-02T09:48:55,476][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [indexer-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
[2025-12-02T09:55:36,192][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [indexer-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
[2025-12-02T09:55:36,192][ERROR][o.o.s.s.t.SecuritySSLNettyTransport] [indexer-1] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
[2025-12-02T09:58:11,061][ERROR][o.o.a.a.AlertIndices     ] [indexer-1] info deleteOldIndices
[2025-12-02T09:58:11,065][ERROR][o.o.a.a.AlertIndices     ] [indexer-1] info deleteOldIndices

Wazuh Indexer Warn Logs
[2025-12-02T09:42:35,037][WARN ][o.o.t.TcpTransport       ] [indexer-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/primary-indexer-ip:9300, remoteAddress=/secondary-indexer-ip:36702}], closing connection
[2025-12-02T09:43:28,542][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T09:48:06,566][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T09:48:55,474][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T09:48:55,476][WARN ][o.o.t.TcpTransport       ] [indexer-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/primary-indexer-ip:9300, remoteAddress=/secondary-indexer-ip:42088}], closing connection
[2025-12-02T09:49:30,935][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T09:50:04,005][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T09:55:31,745][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T09:55:36,193][WARN ][o.o.t.TcpTransport       ] [indexer-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/primary-indexer-ip:9300, remoteAddress=/secondary-indexer-ip:36000}], closing connection
[2025-12-02T09:55:36,193][WARN ][o.o.t.TcpTransport       ] [indexer-1] exception caught on transport layer [Netty4TcpChannel{localAddress=/primary-indexer-ip:9300, remoteAddress=/secondary-indexer-ip:35990}], closing connection
[2025-12-02T09:56:18,146][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T10:31:48,841][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T10:32:28,580][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T10:32:38,694][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from secondary-indexer-ip:42168
[2025-12-02T10:32:50,062][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from secondary-indexer-ip:45818
[2025-12-02T10:32:58,829][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012
[2025-12-02T10:44:43,672][WARN ][r.suppressed             ] [indexer-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1764652457642, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2025-12-02T10:44:43,835][WARN ][r.suppressed             ] [indexer-1] path: /wazuh-alerts-*/_search, params: {ignore_unavailable=true, preference=1764652457642, index=wazuh-alerts-*, timeout=30000ms, track_total_hits=true}
[2025-12-02T10:45:11,039][WARN ][o.o.s.a.BackendRegistry  ] [indexer-1] Authentication finally failed for admin from wazuh-server-ip:59012

Wazuh Indexer systemctl output

root@primary-indexer:/etc/wazuh-indexer# systemctl status wazuh-indexer
● wazuh-indexer.service - wazuh-indexer
   Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/wazuh-indexer.service.d
           └─startup-timeout.conf
   Active: failed (Result: exit-code) since Thu 2025-12-11 13:36:00 IST; 19h ago
Dec 12 08:59:22 primary-indexer systemd-entrypoint[19817]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Dec 12 08:59:22 primary-indexer systemd-entrypoint[19817]: WARNING: System::setSecurityManager will be removed in a future release
Dec 12 08:59:23 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:23Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]: Dec 12, 2025 8:59:23 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]: WARNING: COMPAT locale provider will be removed in a future release
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/sun.nio.fs.UnixException.asIOException(UnixException.java:115)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:477)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/java.nio.file.Files.newDirectoryStream(Files.java:481)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:301)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at java.base/java.nio.file.Files.walkFileTree(Files.java:2820)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:227)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:146)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.cli.Command.main(Command.java:101)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)
Dec 12 08:59:23 primary-indexer systemd-entrypoint[19817]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-indexer-cluster.log
Dec 12 08:59:23 primary-indexer systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Dec 12 08:59:23 primary-indexer systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Dec 12 08:59:23 primary-indexer systemd[1]: Failed to start wazuh-indexer.
-- Subject: Unit wazuh-indexer.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit wazuh-indexer.service has failed.
--
-- The result is RESULT.
Dec 12 08:59:23 primary-indexer audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=wazuh-indexer comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Dec 12 08:59:25 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:25Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:28 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:28Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:30 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:30Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:33 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:33Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:35 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:35Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:38 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:38Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:40 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:40Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:43 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:43Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}
Dec 12 08:59:45 primary-indexer opensearch-dashboards[1182]: {"type":"log","@timestamp":"2025-12-12T03:29:45Z","tags":["error","opensearch","data"],"pid":1182,"message":"[ConnectionError]: connect ECONNREFUSED primary-indexer-ip:9200"}

[Bony V John]

Hi,

From the shared Wazuh Indexer log, it shows a file permission issue in the following log line:

Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup

It seems that the Wazuh Indexer service is trying to access the /etc/wazuh-indexer/backup directory, but due to permission issues it is unable to read it. Please check whether the wazuh-indexer user owns this directory.

Run the below command on the primary Wazuh Indexer node:

ls -l /etc/wazuh-indexer/

If the owner or group is different, run the following command to fix the permissions:  

sudo chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/backup

Then verify the permissions again using the ls -l command. The owner of the backup directory and its contents should match the rest of the Wazuh Indexer files.

After that, restart the Wazuh Indexer service:

systemctl restart wazuh-indexer

Then check whether the Wazuh Indexer service is up and running:

systemctl status wazuh-indexer

You can also refer to the Wazuh Google Groups discussion that is similar to this issue.

[Krishna Prasad Bhandary]

Hi Bony,

The Wazuh Indexer is running now but we have a massive amount of unassigned shards. Does removing the lock files solve this or do we have to do something else?

{
  "cluster_name" : "wazuh-indexer-cluster",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 72,
  "active_shards" : 94,
  "relocating_shards" : 0,
  "initializing_shards" : 8,
  "unassigned_shards" : 283,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 24.415584415584416

[Krishna Prasad Bhandary]

Will definitely look into this!

I also wanted to mention that I was prompting a bit with Copilot and it did recommend me to check if there are any lock files from when I applied the indexer policy, which I have found to be true. Listing below the output of the lock files. I am not sure if I should delete them and I am concerned we would get new lock files created when we restart the indexer again.

(Removed some indices for easier readability)

root@primary-indexer:/home/wazuhuser1# find /var/lib/wazuh-indexer/ -name "*.lock"
/var/lib/wazuh-indexer/nodes/0/_state/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/LuQJFHqNR7CLoGx1xgnxew/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/Pd3l2TxxT7OSlDpfz7arSw/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/Pd3l2TxxT7OSlDpfz7arSw/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/Pd3l2TxxT7OSlDpfz7arSw/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/J7Ev4UM9RnC2VWcpl0gLpA/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/J7Ev4UM9RnC2VWcpl0gLpA/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/J7Ev4UM9RnC2VWcpl0gLpA/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/oq9KcDtfQmqG2fqpLZ0nkg/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/ufpFxFQPT5qR3B5VR4dSGA/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/ufpFxFQPT5qR3B5VR4dSGA/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/EVL8LdlCTNanb_OoDa9XRA/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/EVL8LdlCTNanb_OoDa9XRA/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/EVL8LdlCTNanb_OoDa9XRA/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/5b35XD59Q6OvwGzC8vWGfQ/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/5b35XD59Q6OvwGzC8vWGfQ/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/4NV-K3V8RJGtGir0i_Ydhg/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/4NV-K3V8RJGtGir0i_Ydhg/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/nnyr7qfzQcihRcJo0xTiGQ/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/eR9CRmz0T0qJLHegUTdSmg/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/7RymLJQVT6aRtCUaJMjpLQ/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/7RymLJQVT6aRtCUaJMjpLQ/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/7RymLJQVT6aRtCUaJMjpLQ/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/-b-HFMnpSfGVOHPyMDDGXA/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/-b-HFMnpSfGVOHPyMDDGXA/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/-b-HFMnpSfGVOHPyMDDGXA/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/yCBVME-BQZ6A7b6X28oFjA/1/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/yCBVME-BQZ6A7b6X28oFjA/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/yCBVME-BQZ6A7b6X28oFjA/2/index/write.lock
/var/lib/wazuh-indexer/nodes/0/indices/uduJxyxmSFiWWmGDS3ukzw/0/index/write.lock
/var/lib/wazuh-indexer/nodes/0/node.lock
root@primary-indexer:/home/wazuhuser1# ls -l /var/lib/wazuh-indexer/
total 20
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  5 May 27  2025 batch_metrics_enabled.conf
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  5 May 27  2025 logging_enabled.conf
drwxr-xr-x 3 wazuh-indexer wazuh-indexer 15 Apr 14  2023 nodes
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  4 May 27  2025 performance_analyzer_enabled.conf
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  4 May 27  2025 rca_enabled.conf
-rw-r--r-- 1 wazuh-indexer wazuh-indexer  5 May 27  2025 thread_contention_monitoring_enabled.conf
root@primary-indexer:/home/wazuhuser1# ps aux | grep opensearch
root     21254  0.0  0.0  14860  1124 pts/0    S+   15:28   0:00 grep --color=auto opensearch

[Krishna Prasad Bhandary]

Hi Bony,

I would like to provide an update. The cluster has reached the green state! The changing of the permissions ultimately led to solving the issue so thank you!

{
  "cluster_name" : "wazuh-indexer-cluster",
  "status" : "green",

  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,

  "active_primary_shards" : 161,
  "active_shards" : 316,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,

  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,

  "active_shards_percent_as_number" : 100.0
}

I wanted to know if I could get help on one final issue. We had changed the indexer passwords sometime in June but that had not fully completed due to the tool not being able to create a backup, however after looking into it in the last 2 weeks we found that the intern_users.yml had changed and we did have the changed passwords. I assume this permission issue was the cause for it not fully completing.

We ran the securityadmin.sh script on November 24th and this resulted in the passwords being applied and communications with the Dashboard and Server stopping. We have about 3 weeks of logs now and are unsure that when we add the updated passwords or fully reset it again if the old logs will get indexed.

If they DO get indexed, we do not have enough space and thus wanted to change our retention policy from 60 days to 45 days to allow for the new logs to be added. How do I apply this policy for existing indices? Do I have to manually delete the older indices? I tried to modify the policy for the indices currently and set it to 45 days. Since nothing happened I selected the indices that the policy should apply to and on clicking apply, the issue at the start was created. Would like some help on this if possible. Once again thank you for your swift responses, they have been of great help to my team and I.

[Krishna Prasad Bhandary]

If needed I can raise the above as a new request and you can close this thread.