how to migrate Wazuh old logs to new server

[Ashish Masih]

Hello,

I need urgent help because my previous server failed. I was able to retrieve the data through file recovery, but I now need to restore it to my new Wazuh 4.11 server.

My previous server configuration included Elastic Stack Basic, Wazuh 4.4.5, Elasticsearch 7.17.9, and Kibana. 7.17.9

All agents are up and running on the new server; however, I need help moving the previous two years logs to the new server. 

[Javier Medeot]

Hi Ashish Masih.

You can extract and reinject old alert logs from your /var/ossec/logs/alerts/ compressed log files into a temporary file to replay historical events. Take a look at this guide which will allow you to repopulate your indices in a controlled manner and filtering by a time-window:

- Restoring old logs

[Ashish Masih]

Thanks, Javier for the knowledge sharing.

I have the old data in the old server directory /var/ossec/logs/alerts (screenshot attached).

[Javier Medeot]

Great, then follow the Restoring old logs guide I shared earlier. You would need to specify the min and max dates specific to your case. Let me know if you encounter any issues when creating and executing the script so I can try to assist.

[Ashish Masih]

Hello Javier,

It will be helpful if you can give me the command in steps, as all the logs are in another disk, which I have mounted on the new server. 

The reference article is talking about the temp folder, and this much of the logs moved to the temp folder is a little difficult.

I need to move the last one year's logs, which are showing up month-wise, and each month's folder has more than 60 files. in .log.sum and .json.sum extensions.

old log mounted on = /recover/var/ossec/logs/alerts/2025

[Javier Medeot]

Ashish, the script allows to specify a different path for your logs location with the -w parameter. In your case you need to add -w /recover/var/ossec/ to the command.

The script reads directly from your mounted directory /recover/var/ossec/ and decompresses all the old logs and stores them in the recovery.json file in /tmp directory. If /tmp has limited space, you can replace the -o /tmp/recovery.json parameter with another location. You'll have to update Step 5 accordingly, though.

The script only processes the .json.gz files (gzipped JSON logs) and you don't have to manually move them. The .sum files are checksums that can be ignored for this process.

The updated command in Step 4 will look like this to retrieve your January 2024's recovered logs.

nohup ./recovery.py -w /recover/var/ossec/ -eps 500 -min 2024-01-01T00:00:00 -max 2024-01-31T23:59:59 -o /tmp/recovery.json -log ./recovery.log -sz 2.5 &

Let me know if this is what you need. Thanks.

[Ashish Masih]

Hello Javier,

Thanks for the command; I am able to restore the logs, and they are visible in Wazuh. 

How can I check these logs in the FIM and Threat Hunting modules?

[Jogendar Sagar]

Hi

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wazuh/f71df371-6672-44da-b66a-f4cf58b24712n%40googlegroups.com.

[Javier Medeot]

Ashish, maybe old alerts use a different manager.name value. You can see that Threat hunting view filters alerts by your current manager name only. If this is the case, you can update the manager.name field in your old injected alert log indices as a workaround.

Backup the indices before proceeding to update them to avoid accidental loss by taking a snapshot. Next, to update the old manager.name fields, go to Indexer management > Dev tools  and run the following request replacing <OLD_MANAGER_NAME> and <NEW_MANAGER_NAME>.

POST wazuh-alerts-*/_update_by_query
{
  "script": {
    "source": "ctx._source.manager.name = '<NEW_MANAGER_NAME>'",
    "lang": "painless"
  },
  "query": {
    "term": {
      "manager.name": "<OLD_MANAGER_NAME>"  // Target only old manager alerts
    }
  }
}

[Ashish Masih]

Hello Javier,

Can I use two separate manager names, one for old logs and a second one for new ones?

And few of the months' logs showing 

2025-04-30 17:38:30 wazuh-reinjection: Extracted 90652 alerts from day 20-Dec-2024
2025-04-30 17:38:30 wazuh-reinjection: Reading file: /recover/var/ossec/logs/alerts/2024/Dec/ossec-alerts-21.json.gz
2025-04-30 17:44:11 wazuh-reinjection: Extracted 79289 alerts from day 21-Dec-2024
2025-04-30 17:44:11 wazuh-reinjection: Reading file: /recover/var/ossec/logs/alerts/2024/Dec/ossec-alerts-22.json.gz
2025-04-30 17:45:45 wazuh-reinjection: Output file reached max size, setting it to zero and restarting
2025-04-30 17:49:50 wazuh-reinjection: Extracted 78237 alerts from day 22-Dec-2024
2025-04-30 17:49:50 wazuh-reinjection: Reading file: /recover/var/ossec/logs/alerts/2024/Dec/ossec-alerts-23.json.gz
2025-04-30 17:53:37 wazuh-reinjection: Output file reached max size, setting it to zero and restarting
2025-04-30 17:54:57 wazuh-reinjection: Extracted 70631 alerts from day 23-Dec-2024
2025-04-30 17:54:57 wazuh-reinjection: Reading file: /recover/var/ossec/logs/alerts/2024/Dec/ossec-alerts-24.json.gz

what need to do here

[Ashish Masih]

Hello Javior, 

One more thing happen today during restoration data current dashboard logs showing stopped, pelase find the below reference logs 

after check logs found error in in cluster logs

[2025-05-01T14:33:17,652][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms4g, -Xmx4g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/lib/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=2147483648, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2025-05-01T14:33:58,038][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2025-05-01T14:34:26,564][ERROR][o.o.s.l.BuiltinLogTypeLoader] [node-1] Failed loading builtin log types from disk!
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.16.0.jar:2.16.0]
[2025-05-01T14:34:38,236][ERROR][o.o.s.a.BackendRegistry  ] [node-1] Not yet initialized (you may need to run securityadmin)
[2025-05-01T14:34:38,265][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@11c24699] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-05-01T14:34:38,265][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@11c24699] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-05-01T14:34:38,265][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@11c24699] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-05-01T14:34:38,265][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@11c24699] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)
[2025-05-01T14:34:38,265][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [node-1] Failure No shard available for [org.opensearch.action.get.MultiGetShardRequest@11c24699] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security)

I am getting similar errors in journalctl -u wazuh-dashboard

{"type":"log","@timestamp":"2025-05-01T00:15:01Z","tags":["error","plugins","wazuh","monitoring"],"pid":13207,"message":"validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [0] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}

["error","opensearch","data"],"pid":851,"message":"[search_phase_execution_exception]: all shards failed"}

Cluster Health

{
  "cluster_name" : "wazuh-cluster",
  "status" : "yellow",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 1001,
  "active_shards" : 1001,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 3,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 99.7011952191235
}

shards count file attached 

 /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200 ... done
Connected as "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
OpenSearch Version: 2.16.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: wazuh-cluster
Clusterstate: YELLOW
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Populate config from /etc/wazuh-indexer/opensearch-security/
Will update '/config' with /etc/wazuh-indexer/opensearch-security/config.yml
   SUCC: Configuration for 'config' created or updated
Will update '/roles' with /etc/wazuh-indexer/opensearch-security/roles.yml
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /etc/wazuh-indexer/opensearch-security/roles_mapping.yml
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /etc/wazuh-indexer/opensearch-security/internal_users.yml
   SUCC: Configuration for 'internalusers' created or updated
Will update '/actiongroups' with /etc/wazuh-indexer/opensearch-security/action_groups.yml
   SUCC: Configuration for 'actiongroups' created or updated
Will update '/tenants' with /etc/wazuh-indexer/opensearch-security/tenants.yml
   SUCC: Configuration for 'tenants' created or updated
Will update '/nodesdn' with /etc/wazuh-indexer/opensearch-security/nodes_dn.yml
   SUCC: Configuration for 'nodesdn' created or updated
Will update '/whitelist' with /etc/wazuh-indexer/opensearch-security/whitelist.yml
   SUCC: Configuration for 'whitelist' created or updated
Will update '/audit' with /etc/wazuh-indexer/opensearch-security/audit.yml
   SUCC: Configuration for 'audit' created or updated
Will update '/allowlist' with /etc/wazuh-indexer/opensearch-security/allowlist.yml
   SUCC: Configuration for 'allowlist' created or updated
SUCC: Expected 10 config types for node {"updated_config_types":["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"],"updated_config_size":10,"message":null} is 10 (["allowlist","tenants","rolesmapping","nodesdn","audit","roles","whitelist","actiongroups","config","internalusers"]) due to: null
Done with success

Wazuh Dasbhoard service status 

● wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: enabled)
     Active: active (running) since Thu 2025-05-01 14:32:59 UTC; 3h 39min ago
   Main PID: 851 (node)
      Tasks: 11 (limit: 19147)
     Memory: 220.8M (peak: 1.2G)
        CPU: 57.589s
     CGroup: /system.slice/wazuh-dashboard.service
             └─851 /usr/share/wazuh-dashboard/node/bin/node --no-warnings --max-http-header-size=65536 --unhandled-rejections=warn /usr/share/wazuh-dashboard/src/cli/dist

May 01 17:41:39 SIEM opensearch-dashboards[851]: {"type":"response","@timestamp":"2025-05-01T17:41:37Z","tags":[],"pid":851,"method":"post","statusCode":200,"req":{"url":"/internal/search/opensearch-with-long-numerals","method":"post","headers":{"host":"siem.citixsys.co.in","connection":"keep-alive","osd-version":"2.16.0","sec-ch-ua-platform":"\"Windows\"","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","sec-ch-ua":"\"Google Chrome\";v=\"135\", \"Not-A.Brand\";v=\"8\", \"Chromium\";v=\"135\"","content-type":"application/json","sec-ch-ua-mobile":"?0","osd-xsrf":"osd-fetch","accept":"*/*","origin":"https://siem.citixsys.co.in","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://siem.citixsys.co.in/app/data-explorer/discover","accept-language":"en-US,en;q=0.9","accept-encoding":"gzip, deflate, zstd","content-length":"2698"},"remoteAddress":"40.88.32.228","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","referer":"https://siem.citixsys.co.in/app/data-explorer/discover"},"res":{"statusCode":200,"responseTime":2088,"contentLength":9},"message":"POST /internal/search/opensearch-with-long-numerals 200 2088ms - 9.0B"}
May 01 17:41:50 SIEM opensearch-dashboards[851]: {"type":"response","@timestamp":"2025-05-01T17:41:47Z","tags":[],"pid":851,"method":"post","statusCode":200,"req":{"url":"/internal/search/opensearch-with-long-numerals","method":"post","headers":{"host":"siem.citixsys.co.in","connection":"keep-alive","osd-version":"2.16.0","sec-ch-ua-platform":"\"Windows\"","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","sec-ch-ua":"\"Google Chrome\";v=\"135\", \"Not-A.Brand\";v=\"8\", \"Chromium\";v=\"135\"","content-type":"application/json","sec-ch-ua-mobile":"?0","osd-xsrf":"osd-fetch","accept":"*/*","origin":"https://siem.citixsys.co.in","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://siem.citixsys.co.in/app/data-explorer/discover","accept-language":"en-US,en;q=0.9","accept-encoding":"gzip, deflate, zstd","content-length":"2698"},"remoteAddress":"40.88.32.228","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","referer":"https://siem.citixsys.co.in/app/data-explorer/discover"},"res":{"statusCode":200,"responseTime":2421,"contentLength":9},"message":"POST /internal/search/opensearch-with-long-numerals 200 2421ms - 9.0B"}
May 01 17:41:54 SIEM opensearch-dashboards[851]: {"type":"response","@timestamp":"2025-05-01T17:41:52Z","tags":[],"pid":851,"method":"post","statusCode":200,"req":{"url":"/internal/search/opensearch-with-long-numerals","method":"post","headers":{"host":"siem.citixsys.co.in","connection":"keep-alive","osd-version":"2.16.0","sec-ch-ua-platform":"\"Windows\"","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","sec-ch-ua":"\"Google Chrome\";v=\"135\", \"Not-A.Brand\";v=\"8\", \"Chromium\";v=\"135\"","content-type":"application/json","sec-ch-ua-mobile":"?0","osd-xsrf":"osd-fetch","accept":"*/*","origin":"https://siem.citixsys.co.in","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://siem.citixsys.co.in/app/data-explorer/discover","accept-language":"en-US,en;q=0.9","accept-encoding":"gzip, deflate, zstd","content-length":"2698"},"remoteAddress":"40.88.32.228","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","referer":"https://siem.citixsys.co.in/app/data-explorer/discover"},"res":{"statusCode":200,"responseTime":2047,"contentLength":9},"message":"POST /internal/search/opensearch-with-long-numerals 200 2047ms - 9.0B"}
May 01 17:42:01 SIEM opensearch-dashboards[851]: {"type":"response","@timestamp":"2025-05-01T17:41:59Z","tags":[],"pid":851,"method":"post","statusCode":200,"req":{"url":"/internal/search/opensearch-with-long-numerals","method":"post","headers":{"host":"siem.citixsys.co.in","connection":"keep-alive","osd-version":"2.16.0","sec-ch-ua-platform":"\"Windows\"","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","sec-ch-ua":"\"Google Chrome\";v=\"135\", \"Not-A.Brand\";v=\"8\", \"Chromium\";v=\"135\"","content-type":"application/json","sec-ch-ua-mobile":"?0","osd-xsrf":"osd-fetch","accept":"*/*","origin":"https://siem.citixsys.co.in","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://siem.citixsys.co.in/app/data-explorer/discover","accept-language":"en-US,en;q=0.9","accept-encoding":"gzip, deflate, zstd","content-length":"2698"},"remoteAddress":"40.88.32.228","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","referer":"https://siem.citixsys.co.in/app/data-explorer/discover"},"res":{"statusCode":200,"responseTime":2053,"contentLength":9},"message":"POST /internal/search/opensearch-with-long-numerals 200 2053ms - 9.0B"}
May 01 17:42:08 SIEM opensearch-dashboards[851]: {"type":"response","@timestamp":"2025-05-01T17:42:06Z","tags":[],"pid":851,"method":"post","statusCode":200,"req":{"url":"/internal/search/opensearch-with-long-numerals","method":"post","headers":{"host":"siem.citixsys.co.in","connection":"keep-alive","osd-version":"2.16.0","sec-ch-ua-platform":"\"Windows\"","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","sec-ch-ua":"\"Google Chrome\";v=\"135\", \"Not-A.Brand\";v=\"8\", \"Chromium\";v=\"135\"","content-type":"application/json","sec-ch-ua-mobile":"?0","osd-xsrf":"osd-fetch","accept":"*/*","origin":"https://siem.citixsys.co.in","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","sec-fetch-dest":"empty","referer":"https://siem.citixsys.co.in/app/data-explorer/discover","accept-language":"en-US,en;q=0.9","accept-encoding":"gzip, deflate, zstd","content-length":"2698"},"remoteAddress":"40.88.32.228","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/135.0.0.0 Safari/537.36","referer":"https://siem.citixsys.co.in/app/data-explorer/discover"},"res":{"statusCode":200,"responseTime":2034,"contentLength":9},"message":"POST /internal/search/opensearch-with-long-numerals 200 2034ms - 9.0B"}
May 01 17:45:00 SIEM opensearch-dashboards[851]: {"type":"log","@timestamp":"2025-05-01T17:45:00Z","tags":["error","opensearch","data"],"pid":851,"message":"[validation_exception]: Validation Failed: 1: this action would add [0] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}
May 01 17:45:00 SIEM opensearch-dashboards[851]: {"type":"log","@timestamp":"2025-05-01T17:45:00Z","tags":["error","plugins","wazuh","monitoring"],"pid":851,"message":"validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [0] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}
May 01 17:48:45 SIEM opensearch-dashboards[851]: {"type":"response","@timestamp":"2025-05-01T17:48:45Z","tags":[],"pid":851,"method":"get","statusCode":401,"req":{"url":"/.git/index","method":"get","headers":{"host":"172.178.15.120","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36","accept-charset":"utf-8","accept-encoding":"gzip","connection":"close"},"remoteAddress":"196.251.70.87","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.109 Safari/537.36"},"res":{"statusCode":401,"responseTime":3,"contentLength":9},"message":"GET /.git/index 401 3ms - 9.0B"}
May 01 18:00:00 SIEM opensearch-dashboards[851]: {"type":"log","@timestamp":"2025-05-01T18:00:00Z","tags":["error","opensearch","data"],"pid":851,"message":"[validation_exception]: Validation Failed: 1: this action would add [0] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}
May 01 18:00:00 SIEM opensearch-dashboards[851]: {"type":"log","@timestamp":"2025-05-01T18:00:00Z","tags":["error","plugins","wazuh","monitoring"],"pid":851,"message":"validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [0] total shards, but this cluster currently has [1004]/[1000] maximum shards open;"}

[Javier Medeot]

Ashish, the following error message:

Validation Failed: 1: this action would add [0] total shards, but this cluster currently has [1004]/[1000] maximum shards open 

seems to be the reason new operations are getting blocked. You can scale up your Wazuh indexer cluster to increase capacity by adding new indexer nodes as explained here:

• Adding Wazuh indexer nodes

You can also remove older indices no longer needed. Implement an index retention policy to handle removal of older indices automatically as explained here:

• Index retention policy

Depending on your cluster structure, you can configure an optimal number of shards and replicas as explained here:

• Shards and replicas

[Ashish Masih]

Thanks, Javier, I am able to recover all the indexes, and all services are running.

Thanks again for your efforts.