Wazuh-indexer upgrade to 4.13 from 4.13 fail

[M. Audureau]

Hello wazuh community,

I have a test wazuh stack 4.12 in single node mode on debian 12 with indexer / dashboard / manager, when I installed wazuh, I have followed the documentation "Step-by-step installation".
I have tried to upgrade to 4.13 version, unfortunatly after upgrading wazuh-indexer via "apt-get install wazuh-indexer" the indexer service doesn't start.
I search in forums and github issues but I doesn't find user with same problem.

See below my "/var/wazuh-indexer/wazuh-cluster.log" log file :

[2025-09-22T21:52:25,030][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms8192m, -Xmx8192m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-5553860509695503458, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, --add-modules=jdk.incubator.vector, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=4294967296, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=deb, -Dopensearch.bundled_jdk=true]
[2025-09-22T21:52:29,485][WARN ][stderr                   ] [node-1] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-09-22T21:52:29,486][WARN ][stderr                   ] [node-1] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-09-22T21:52:29,486][WARN ][stderr                   ] [node-1] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-09-22T21:52:32,939][WARN ][o.o.s.c.Salt             ] [node-1] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2025-09-22T21:52:32,976][ERROR][o.o.s.a.s.SinkProvider   ] [node-1] Default endpoint could not be created, auditlog will not work properly.
[2025-09-22T21:52:32,977][WARN ][o.o.s.a.r.AuditMessageRouter] [node-1] No default storage available, audit log may not work properly. Please check configuration.
[2025-09-22T21:52:33,835][WARN ][o.o.s.p.SQLPlugin        ] [node-1] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2025-09-22T21:52:35,064][WARN ][o.o.g.DanglingIndicesState] [node-1] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-09-22T21:52:36,884][WARN ][o.o.c.m.MetadataIndexUpgradeService] [node-1] [.opensearch-sap-correlation-history-2025.09.15-000006/p86X76GrRWWlV64X4MuURw] ignoring unknown index setting: [index.correlation] with value [true]; archiving
[2025-09-22T21:52:36,886][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.19.2.jar:2.19.2]
[2025-09-22T21:52:36,894][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.19.2.jar:2.19.2]


My wazuh-indexer configuration "/etc/wazuh-indexer/opensearch.yml" :

network.host: "0.0.0.0"
node.name: "node-1"
#cluster.initial_master_nodes:
#- "node-1"
cluster.name: "wazuh-cluster"
discovery.type: single-node

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer

plugins.security.ssl.http.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.http.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.http.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.transport.pemcert_filepath: /etc/wazuh-indexer/certs/wazuh-indexer.pem
plugins.security.ssl.transport.pemkey_filepath: /etc/wazuh-indexer/certs/wazuh-indexer-key.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: /etc/wazuh-indexer/certs/root-ca.pem
plugins.security.ssl.http.enabled: true
plugins.security.ssl.transport.enforce_hostname_verification: false
plugins.security.ssl.transport.resolve_hostname: false
plugins.security.ssl.http.enabled_ciphers:
  - "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384"
  - "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256"
  - "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384"
plugins.security.ssl.http.enabled_protocols:
  - "TLSv1.2"
plugins.security.authcz.admin_dn:
- "CN=admin,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.check_snapshot_restore_write_privileges: true
plugins.security.enable_snapshot_restore_privilege: true
plugins.security.nodes_dn:
- "CN=indexer,OU=Wazuh,O=Wazuh,L=California,C=US"
plugins.security.restapi.roles_enabled:
- "all_access"
- "security_rest_api_access"

plugins.security.system_indices.enabled: true
plugins.security.system_indices.indices: [".opendistro-alerting-config", ".opendistro-alerting-alert*", ".opendistro-anomaly-results*", ".opendistro-anomaly-detector*", ".opendistro-anomaly-checkpoints", ".opendistro-anomal>

### Option to allow Filebeat-oss 7.10.2 to work ###
compatibility.override_main_response_version: true

As I write these lines, I have rollback to 4.12 thanks to a snapshot created on the vm before the update began.
What do I need to correct/check before trying the update again?
Note: Before the update, the cluster was green.

Can you help me solve this problem?

[Facundo Dalmau]

Hi. 

I will try to replicate it, but in the meantime, could you describe the upgrade process you have carried out? Was the one described in the official documentation (https://documentation.wazuh.com/current/upgrade-guide/upgrading-central-components.html#upgrading-the-wazuh-indexer)? Do you occasionally have the complete stack trace of the error?

[M. Audureau]

Hi,

I have follow the upgrade process as describe in the official documentation :

curl -X PUT "https://127.0.0.1:9200/_cluster/settings" \
-u <USERNAME>:<PASSWORD> -k -H "Content-Type: application/json" -d '
{
   "persistent": {
      "cluster.routing.allocation.enable": "primaries"
   }
}'

{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":{}}


curl -X POST "https://127.0.0.1:9200/_flush" -u <USERNAME>:<PASSWORD> -k

{"_shards":{"total":421,"successful":421,"failed":0}}

systemctl stop wazuh-manager -> I am in single node
systemctl stop wazuh-indexer
cp /etc/wazuh-indexer/jvm.options /etc/wazuh-indexer/jvm.options.old

apt-get install wazuh-indexer

Fichier de configuration « /etc/wazuh-indexer/jvm.options »
 ==> Modifié (par vous ou par un script) depuis l'installation.
 ==> Le distributeur du paquet a fourni une version mise à jour.
   Que voulez-vous faire ? Vos options sont les suivantes :
    Y ou I  : installer la version du responsable du paquet
    N ou O  : garder votre version actuellement installée
      D     : afficher les différences entre les versions
      Z     : suspendre ce processus pour examiner la situation
 L'action par défaut garde votre version actuelle.
*** jvm.options (Y/I/N/O/D/Z) [défaut=N] ? Y
Installation de la nouvelle version du fichier de configuration /etc/wazuh-indexer/jvm.options ...

I re-applied the 8Go of RAM instead of 1Go by default.

nano /etc/wazuh-indexer/jvm.options
-Xms8192m
-Xmx8192m

systemctl daemon-reload
systemctl enable wazuh-indexer.service
systemctl start wazuh-indexer.service

After that the wazuh-indexer service not start with the error I have posted in my previous post.

What do you mean by "complete stack trace of the error" ?

[M. Audureau]

Hi,

I solved my problem by deleting the indexes which caused the errors through "Dev Tools" on dashboard prior to make the update to 4.13.1 .

After that wazuh-indexer successfuly start in 4.13.1.

DELETE .opensearch-sap-correlation-history-2025.08.16-000005

DELETE .opensearch-sap-correlation-history-2025.09.15-000006

DELETE .opensearch-sap-correlation-metadata

I haven't found much information on the usefulness of these indexes...

[Facundo Dalmau]

Hi.

Glad to read you solved it. Those indices are related to the Security Analytics Plugin, which was removed in Wazuh v4.13.0.
For future cases, what I referred to with `complete stack trace of the error` was the complete content of the /var/wazuh-indexer/wazuh-cluster.log file showing the error, since it may have offered more information related to it.

[M. Audureau]

Hi,

Here is the log file “/var/wazuh-indexer/wazuh-cluster.log.”.

I have only included the part from the service startup to the error I encountered.

It might be a good idea to add a note to the upgrade procedure.