Error in setting up wazuh indexer

306 views
Skip to first unread message

Gary Ng

unread,
Oct 18, 2024, 12:17:16 AM10/18/24
to Wazuh | Mailing List
Sorry for the newbie question... We are facing issue when starting up wazuh indexer, anyone can help? Thanks

Oct 18 10:42:48 wazuh-sys systemd[1]: Starting wazuh-indexer...

Oct 18 10:42:50 wazuh-sys systemd-entrypoint[2974]: WARNING: A terminally deprecated method in java.lang.System has been called

Oct 18 10:42:50 wazuh-sys systemd-entrypoint[2974]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)

Oct 18 10:42:50 wazuh-sys systemd-entrypoint[2974]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch

Oct 18 10:42:50 wazuh-sys systemd-entrypoint[2974]: WARNING: System::setSecurityManager will be removed in a future release

Oct 18 10:42:52 wazuh-sys systemd-entrypoint[2974]: Oct 18, 2024 10:42:52 AM sun.util.locale.provider.LocaleProviderAdapter <clinit>

Oct 18 10:42:52 wazuh-sys systemd-entrypoint[2974]: WARNING: COMPAT locale provider will be removed in a future release

Oct 18 10:42:52 wazuh-sys systemd-entrypoint[2974]: WARNING: A terminally deprecated method in java.lang.System has been called

Oct 18 10:42:52 wazuh-sys systemd-entrypoint[2974]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.Security (file:/usr/share/wazuh-indexer/lib/opensearch-2.13.0.jar)

Oct 18 10:42:52 wazuh-sys systemd-entrypoint[2974]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.Security

Oct 18 10:42:52 wazuh-sys systemd-entrypoint[2974]: WARNING: System::setSecurityManager will be removed in a future release

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: ERROR: [1] bootstrap checks failed

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: [1]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: ERROR: OpenSearch did not exit normally - check the logs at /var/log/wazuh-indexer/wazuh-cluster.log

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: fatal error in thread [Thread-3], exiting

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.systemd.Libsystemd.lambda$static$0(Libsystemd.java:48)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.security.AccessController.doPrivileged(AccessController.java:319)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.systemd.Libsystemd.<clinit>(Libsystemd.java:47)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.systemd.SystemdPlugin.sd_notify(SystemdPlugin.java:126)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.systemd.SystemdPlugin.close(SystemdPlugin.java:152)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:114)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.node.Node.close(Node.java:1690)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:89)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:131)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.common.util.io.IOUtils.close(IOUtils.java:81)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.UnsatisfiedLinkError: /var/log/wazuh-indexer/tmp/jna11988429801313569869.tmp: /var/log/wazuh-indexer/tmp/jna11988429801313569869.tmp: failed to map segment from shared object [in thread "main"]

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:331)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:197)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:139)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2418)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.lang.Runtime.load0(Runtime.java:852)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.lang.System.load(System.java:2025)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:1045)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:1015)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at com.sun.jna.Native.<clinit>(Native.java:221)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.lang.Class.forName0(Native Method)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.lang.Class.forName(Class.java:421)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at java.base/java.lang.Class.forName(Class.java:412)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.Natives.<clinit>(Natives.java:60)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:123)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:191)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:181)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:172)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.cli.Command.main(Command.java:101)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:138)

Oct 18 10:43:05 wazuh-sys systemd-entrypoint[2974]: #011at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:104)

Oct 18 10:43:05 wazuh-sys systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE

Oct 18 10:43:05 wazuh-sys systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.

Oct 18 10:43:05 wazuh-sys systemd[1]: Failed to start wazuh-indexer.

hasitha.u...@wazuh.com

unread,
Oct 18, 2024, 1:19:32 AM10/18/24
to Wazuh | Mailing List

Hi Gary,

If this is a fresh installation, I recommend reinstalling it on a fresh operating system, ensuring you follow the resource and OS requirements:

If you're installing using the quickstart method, you can deploy everything on a single server. Make sure you have the proper resources and Wazuh requirements.

If the issue persists, please share the cluster logs for further investigation by running the following command:
cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn|crit"

I hope this helps! Let me know if you need further assistance.

Regards,
Hasitha Upekshitha

Gary Ng

unread,
Oct 18, 2024, 3:54:24 AM10/18/24
to Wazuh | Mailing List
Running the command i got the following errors

[2024-10-18T10:38:36,333][WARN ][o.o.b.Natives            ] [192.168.12.11] unable to load JNA native support library, native methods will be disabled.

java.lang.UnsatisfiedLinkError: /var/log/wazuh-indexer/tmp/jna6578779806595086664.tmp: /var/log/wazuh-indexer/tmp/jna6578779806595086664.tmp: failed to map segment from shared object

        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.13.0.jar:2.13.0]

[2024-10-18T10:38:36,383][WARN ][o.o.b.Natives            ] [192.168.12.11] cannot check if running as root because JNA is not available

[2024-10-18T10:38:36,383][WARN ][o.o.b.Natives            ] [192.168.12.11] cannot install system call filter because JNA is not available

[2024-10-18T10:38:36,384][WARN ][o.o.b.Natives            ] [192.168.12.11] cannot register console handler because JNA is not available

[2024-10-18T10:38:36,407][WARN ][o.o.b.Natives            ] [192.168.12.11] cannot getrlimit RLIMIT_NPROC because JNA is not available

[2024-10-18T10:38:36,407][WARN ][o.o.b.Natives            ] [192.168.12.11] cannot getrlimit RLIMIT_AS because JNA is not available

[2024-10-18T10:38:36,408][WARN ][o.o.b.Natives            ] [192.168.12.11] cannot getrlimit RLIMIT_FSIZE because JNA is not available

[2024-10-18T10:38:37,554][INFO ][o.o.n.Node               ] [192.168.12.11] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.security.manager=allow, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/var/log/wazuh-indexer/tmp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/wazuh-indexer, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.security.manager=allow, -Djava.util.concurrent.ForkJoinPool.common.threadFactory=org.opensearch.secure_sm.SecuredForkJoinWorkerThreadFactory, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///etc/wazuh-indexer/opensearch-performance-analyzer/opensearch_security.policy, --add-opens=jdk.attach/sun.tools.attach=ALL-UNNAMED, -XX:MaxDirectMemorySize=536870912, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]

[2024-10-18T10:39:13,712][ERROR][o.o.p.c.j.GCMetrics      ] [192.168.12.11] MX bean missing: G1 Concurrent GC

[2024-10-18T10:39:29,863][WARN ][o.o.s.c.Salt             ] [192.168.12.11] If you plan to use field masking pls configure compliance salt e1ukloTsQlOgPquJ to be a random string of 16 chars length identical on all nodes

[2024-10-18T10:39:30,164][ERROR][o.o.s.a.s.SinkProvider   ] [192.168.12.11] Default endpoint could not be created, auditlog will not work properly.

[2024-10-18T10:39:30,166][WARN ][o.o.s.a.r.AuditMessageRouter] [192.168.12.11] No default storage available, audit log may not work properly. Please check configuration.

[2024-10-18T10:39:36,670][WARN ][o.o.s.p.SQLPlugin        ] [192.168.12.11] Master key is a required config for using create and update datasource APIs. Please set plugins.query.datasources.encryption.masterkey config in opensearch.yml in all the cluster nodes. More details can be found here: https://github.com/opensearch-project/sql/blob/main/docs/user/ppl/admin/datasources.rst#master-key-config-for-encrypting-credential-information


hasitha.u...@wazuh.com

unread,
Oct 25, 2024, 1:31:13 AM10/25/24
to Wazuh | Mailing List
Hi Gary,

I suggest you to follow the Hardware requirements and OS requirements again.

Hardware requirements: https://documentation.wazuh.com/current/quickstart.html#hardware
OS requirements: https://documentation.wazuh.com/current/quickstart.html#operating-system

Follow the steps regarding this HeapDumpOnOutOfMemoryError issue.

I suggest you expand your RAM and configure the heap memory accordingly.
Here are some key points to keep in mind to configure heap memory:
Use no more than 50% of your available RAM.
Don’t set the heap size over 32 GB.
Start by checking your memory with:
free -h

Then, update the heap size in the /etc/wazuh-indexer/jvm.options file. For example, if your server has 12 GB of RAM, set the heap size to 6 GB as shown below:
-Xms6g
-Xmx6g
After making these changes, restart the Wazuh indexer for them to take effect:
systemctl restart wazuh-indexer

Most probably not enough RAM in the server

Additionally please share the output of following commands.
free -h
top

Also, provide the OS and the OS version you are using.

Let me know the update on this.

Regards,
Hasitha Upekshitha
Reply all
Reply to author
Forward
0 new messages