Adding new indexer node to existing 3 indexer node cluster
harish grandhi
We have inhouse wazuh setup with distributed deployment(1 dash board, 3 wazuh manager and worker nodes, 3 wazuh indexer nodes).
No we are trying to add one additional wazuh indexer node. We followed the steps as per documentation, after completing all the steps unable to start the wazuh indexer service on newly configured indexer node.
[@lonpimb940wzi02 ~]# journalctl -xe |grep wazuh
Jan 22 17:00:04 lonpimb940wzi02.abc.xyz.com systemd[1]: Found device /dev/mapper/vg1-var_lib_wazuh_indexer.
-- Subject: Unit dev-mapper-vg1\x2dvar_lib_wazuh_indexer.device has finished start-up
-- Unit dev-mapper-vg1\x2dvar_lib_wazuh_indexer.device has finished starting up.
Jan 22 17:00:04 lonpimb940wzi02.abc.xyz.com systemd[1]: Mounting /var/lib/wazuh-indexer...
-- Subject: Unit var-lib-wazuh\x2dindexer.mount has begun start-up
-- Unit var-lib-wazuh\x2dindexer.mount has begun starting up.
Jan 22 17:00:05 lonpimb940wzi02.abc.xyz.com systemd[1]: Mounted /var/lib/wazuh-indexer.
-- Subject: Unit var-lib-wazuh\x2dindexer.mount has finished start-up
-- Unit var-lib-wazuh\x2dindexer.mount has finished starting up.
Jan 22 17:00:08 lonpimb940wzi02.abc.xyz.com systemd[1]: Starting wazuh-indexer...
-- Subject: Unit wazuh-indexer.service has begun start-up
-- Unit wazuh-indexer.service has begun starting up.
Jan 22 17:00:10 lonpimb940wzi02.abc.xyz.com systemd-entrypoint[1153]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Jan 22 17:00:12 lonpimb940wzi02.abc.xyz.com systemd-entrypoint[1153]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.16.0.jar)
Jan 22 17:00:22 lonpimb940wzi02.abc.xyz.com systemd-entrypoint[1153]: ERROR: OpenSearch did not exit normally - check the logs at /var/log/wazuh-indexer/lon-wazuh-cluster.log
Jan 22 17:00:22 lonpimb940wzi02.abc.xyz.com systemd-entrypoint[1153]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /var/log/wazuh-indexer/tmp/jna9656403466518432966.tmp: /var/log/wazuh-indexer/tmp/jna9656403466518432966.tmp: failed to map segment from shared object [in thread "main"]
Jan 22 17:00:22 lonpimb940wzi02.abc.xyz.com systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Jan 22 17:00:22 lonpimb940wzi02.abc.xyz.com systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
-- The unit wazuh-indexer.service has entered the 'failed' state with result 'exit-code'.
Jan 22 17:00:22 lonpimb940wzi02.abc.xyz.com systemd[1]: Failed to start wazuh-indexer.
-- Subject: Unit wazuh-indexer.service has failed
-- Unit wazuh-indexer.service has failed.
[@lonpimb940wzi02 ~]#
****************************************
111]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/l>
111]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
111]: WARNING: System::setSecurityManager will be removed in a future release
111]: Jan 22, 2025 5:56:04 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
111]: WARNING: COMPAT locale provider will be removed in a future release
111]: WARNING: A terminally deprecated method in java.lang.System has been called
111]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib>
111]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security
111]: WARNING: System::setSecurityManager will be removed in a future release
111]: ERROR: [1] bootstrap checks failed
111]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at you>
111]: ERROR: OpenSearch did not exit normally - check the logs at /var/log/wazuh-indexer/lon-wazuh-cluster.log
111]: fatal error in thread [Thread-3], exiting
111]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
111]: at org.opensearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:48)
111]: at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)
111]: at org.opensearch.systemd.Libsystemd.<clinit>(Libsystemd.java:47)
111]: at org.opensearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126)
111]: at org.opensearch.systemd.SystemdPlugin.close(SystemdPlugin.java:152)
111]: at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
111]: at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
111]: at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)
111]: at org.opensearch.node.Node.close(Node.java:1791)
111]: at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)
111]: at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)
****************************************************************
java.lang.UnsatisfiedLinkError: /var/log/wazuh-indexer/tmp/jna6307739035583799470.tmp: /var/log/wazuh-indexer/tmp/jna6307739035583799470.tmp: failed to map segment from shared object
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.16.0.jar:2.16.0]
[2025-01-22T17:56:05,109][WARN ][o.o.b.Natives ] [lonpimb940wzi02] cannot check if running as root because JNA is not available
[2025-01-22T17:56:05,109][WARN ][o.o.b.Natives ] [lonpimb940wzi02] cannot install system call filter because JNA is not available
[2025-01-22T17:56:05,110][WARN ][o.o.b.Natives ] [lonpimb940wzi02] cannot register console handler because JNA is not available
[2025-01-22T17:56:05,110][WARN ][o.o.b.Natives ] [lonpimb940wzi02] cannot getrlimit RLIMIT_NPROC because JNA is not available
[2025-01-22T17:56:05,111][WARN ][o.o.b.Natives ] [lonpimb940wzi02] cannot getrlimit RLIMIT_AS because JNA is not available
[2025-01-22T17:56:05,111][WARN ][o.o.b.Natives ] [lonpimb940wzi02] cannot getrlimit RLIMIT_FSIZE because JNA is not available
[2025-01-22T17:56:05,244][INFO ][o.o.n.Node ] [lonpimb940wzi02] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms12g, -Xmx12g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=6442450944, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2025-01-22T17:56:05,356][WARN ][o.a.l.i.v.VectorizationProvider] [lonpimb940wzi02] Java vector incubator module is not readable. For optimal vector performance, pass '--add-modules jdk.incubator.vector' to enable Vector API.
[2025-01-22T17:56:07,599][WARN ][stderr ] [lonpimb940wzi02] WARNING: A restricted method in java.lang.foreign.Linker has been called
[2025-01-22T17:56:07,600][WARN ][stderr ] [lonpimb940wzi02] WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
[2025-01-22T17:56:07,600][WARN ][stderr ] [lonpimb940wzi02] WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
[2025-01-22T17:56:09,901][WARN ][o.o.s.c.Salt ] [lonpimb940wzi02] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes
[2025-01-22T17:56:09,936][ERROR][o.o.s.a.s.SinkProvider ] [lonpimb940wzi02] Default endpoint could not be created, auditlog will not work properly.
[2025-01-22T17:56:09,937][WARN ][o.o.s.a.r.AuditMessageRouter] [lonpimb940wzi02] No default storage available, audit log may not work properly. Please check configuration.
[2025-01-22T17:56:10,707][WARN ][o.o.s.p.SQLPlugin ] [lonpimb940wzi02] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information
[2025-01-22T17:56:11,801][WARN ][o.o.g.DanglingIndicesState] [lonpimb940wzi02] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2025-01-22T17:56:12,256][ERROR][o.o.p.c.j.GCMetrics ] [lonpimb940wzi02] MX bean missing: G1 Concurrent GC
[2025-01-22T17:56:12,427][ERROR][o.o.s.l.BuiltinLogTypeLoader] [lonpimb940wzi02] Failed loading builtin log types from disk!
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.16.0.jar:2.16.0]
[2025-01-22T17:56:12,643][ERROR][o.o.b.Bootstrap ] [lonpimb940wzi02] node validation exception
[2025-01-22T17:56:12,660][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [lonpimb940wzi02] fatal error in thread [Thread-3], exiting
java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /var/log/wazuh-indexer/tmp/jna6307739035583799470.tmp: /var/log/wazuh-indexer/tmp/jna6307739035583799470.tmp: failed to map segment from shared object [in thread "main"]
at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.16.0.jar:2.16.0]
[@lonpimb940wzi02 ~]#
Stuti Gupta
It seems that the wazuh-indexer is not installed properly. Make sure you meet the minumun hardware requirement. https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/index.html#hardware-recommendations. I will recommended to reinstall it creafully and make sure to follow all the steps mentioned here: https://documentation.wazuh.com/current/user-manual/wazuh-indexer-cluster/add-wazuh-indexer-nodes.html
Hope this helps
harish grandhi
now we are into another issue. post the service is up and running , we have initialized the cluster. but the new node coming out of the cluster when i check the command "curl -k -u <USERNAME>:<PASSWORD> https://<WAZUH_INDEXER_IP_ADDRESS>:9200/_cat/nodes?v" .
if we wait for couple of minutes then again its all good even GUI dashboard. Then again it is repeating.
some errors from new wazuh indexer closter log
very will continue using [172.27.2.73:9300, 172.27.2.75:9300, 172.27.2.135:9300] from hosts providers and [{lonpimb940wzi01}{diK2ayT6TrmlAWuCZ2g_jg}{2NZyvHRdRo2w64rGiCYlBg}{172.27.2.135}{172.27.2.135:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpimb940wzi02}{iCJNDB8mQwuLvkBakN9BdA}{9w-tK5nbSvyOd_9mB__XMw}{172.27.2.136}{172.27.2.136:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpima940wzi01}{1Oe_M2D7TNqtqQ4boPLWfA}{90zMQtKxQvWlDTGeIwlgGg}{172.27.2.73}{172.27.2.73:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpima940wzi02}{I77qo6i3SgOvd33Bz4AEsQ}{YNmmClRLQdmTOLFLQP1Iqg}{172.27.2.75}{172.27.2.75:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 66, last-accepted version 53337 in term 66
[2025-01-30T06:27:12,042][WARN ][o.o.c.c.ClusterFormationFailureHelper] [lonpimb940wzi02] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [diK2ayT6TrmlAWuCZ2g_jg, I77qo6i3SgOvd33Bz4AEsQ, 1Oe_M2D7TNqtqQ4boPLWfA], have discovered [{lonpimb940wzi02}{iCJNDB8mQwuLvkBakN9BdA}{9w-tK5nbSvyOd_9mB__XMw}{172.27.2.136}{172.27.2.136:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpimb940wzi01}{diK2ayT6TrmlAWuCZ2g_jg}{2NZyvHRdRo2w64rGiCYlBg}{172.27.2.135}{172.27.2.135:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpima940wzi01}{1Oe_M2D7TNqtqQ4boPLWfA}{90zMQtKxQvWlDTGeIwlgGg}{172.27.2.73}{172.27.2.73:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpima940wzi02}{I77qo6i3SgOvd33Bz4AEsQ}{YNmmClRLQdmTOLFLQP1Iqg}{172.27.2.75}{172.27.2.75:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is a quorum; discovery will continue using [172.27.2.73:9300, 172.27.2.75:9300, 172.27.2.135:9300] from hosts providers and [{lonpimb940wzi01}{diK2ayT6TrmlAWuCZ2g_jg}{2NZyvHRdRo2w64rGiCYlBg}{172.27.2.135}{172.27.2.135:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpimb940wzi02}{iCJNDB8mQwuLvkBakN9BdA}{9w-tK5nbSvyOd_9mB__XMw}{172.27.2.136}{172.27.2.136:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpima940wzi01}{1Oe_M2D7TNqtqQ4boPLWfA}{90zMQtKxQvWlDTGeIwlgGg}{172.27.2.73}{172.27.2.73:9300}{dimr}{shard_indexing_pressure_enabled=true}, {lonpima940wzi02}{I77qo6i3SgOvd33Bz4AEsQ}{YNmmClRLQdmTOLFLQP1Iqg}{172.27.2.75}{172.27.2.75:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 66, last-accepted version 53337 in term 66
[2025-01-30T06:27:18,470][INFO ][o.o.j.s.JobSweeper ] [lonpimb940wzi02] Running full sweep
[2025-01-30T06:27:22,042][WARN
][o.o.c.c.ClusterFormationFailureHelper] [lonpimb940wzi02] cluster-manager not
discovered or elected yet, an election requires at least 2 nodes with ids from
[diK2ayT6TrmlAWuCZ2g_jg, I77qo6i3SgOvd33Bz4AEsQ, 1Oe_M2D7TNqtqQ4boPLWfA], have
discovered
[{lonpimb940wzi02}{iCJNDB8mQwuLvkBakN9BdA}{9w-tK5nbSvyOd_9mB__XMw}{172.27.2.136}{172.27.2.136:9300}{dimr}{shard_indexing_pressure_enabled=true},
{lonpimb940wzi01}{diK2ayT6TrmlAWuCZ2g_jg}{2NZyvHRdRo2w64rGiCYlBg}{172.27.2.135}{172.27.2.135:9300}{dimr}{shard_indexing_pressure_enabled=true},
{lonpima940wzi01}{1Oe_M2D7TNqtqQ4boPLWfA}{90zMQtKxQvWlDTGeIwlgGg}{172.27.2.73}{172.27.2.73:9300}{dimr}{shard_indexing_pressure_enabled=true},
{lonpima940wzi02}{I77qo6i3SgOvd33Bz4AEsQ}{YNmmClRLQdmTOLFLQP1Iqg}{172.27.2.75}{172.27.2.75:9300}{dimr}{shard_indexing_pressure_enabled=true}]
which is a quorum; discovery will continue using [172.27.2.73:9300,
172.27.2.75:9300, 172.27.2.135:9300] from hosts providers and
[{lonpimb940wzi01}{diK2ayT6TrmlAWuCZ2g_jg}{2NZyvHRdRo2w64rGiCYlBg}{172.27.2.135}{172.27.2.135:9300}{dimr}{shard_indexing_pressure_enabled=true},
{lonpimb940wzi02}{iCJNDB8mQwuLvkBakN9BdA}{9w-tK5nbSvyOd_9mB__XMw}{172.27.2.136}{172.27.2.136:9300}{dimr}{shard_indexing_pressure_enabled=true},
{lonpima940wzi01}{1Oe_M2D7TNqtqQ4boPLWfA}{90zMQtKxQvWlDTGeIwlgGg}{172.27.2.73}{172.27.2.73:9300}{dimr}{shard_indexing_pressure_enabled=true},
{lonpima940wzi02}{I77qo6i3SgOvd33Bz4AEsQ}{YNmmClRLQdmTOLFLQP1Iqg}{172.27.2.75}{172.27.2.75:9300}{dimr}{shard_indexing_pressure_enabled=true}]
from last-known cluster state; node term 66, last-accepted version 53337 in
term 66
filebeat output is fine:
[root@lonpima940wzm01 ~]# filebeat test output
elasticsearch: https://172.27.2.73:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.27.2.73
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.10.2
elasticsearch: https://172.27.2.75:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.27.2.75
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.10.2
elasticsearch: https://172.27.2.135:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.27.2.135
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.10.2
elasticsearch: https://172.27.2.136:9200...
parse url... OK
connection...
parse host... OK
dns lookup... OK
addresses: 172.27.2.136
dial up... OK
TLS...
security: server's certificate chain verification is enabled
handshake... OK
TLS version: TLSv1.3
dial up... OK
talk to server... OK
version: 7.10.2
[root@lonpima940wzm01 ~]#
