wazuh 4.9 Indexer errors

leon appel

unread,

Sep 11, 2024, 12:51:02 PM9/11/24

to Wazuh | Mailing List

Hi

I have upgraded my wazuh server from 4.8 to 4.9 and since then have been getting these errors in the cluster logs. Has anyone got any insight on these errors

Thanks in advance

errors.jpg

Julio Gasco

unread,

Sep 13, 2024, 7:44:57 AM9/13/24

to Wazuh | Mailing List

HI Leon,

Sorry for the delay. The first errors regarding Managed Index seems to be a failure when applying an ISM policy. I would recommend recreating your ISM index management policy to see if this fixes the issue. Also you can share it with us and I can see if there might be a problem with it. It might be due to a replica that can't be created because there are missing nodes ? I would like to have some details from the ISM policies if you are using them and some details on the infrastructure (number of indexers, specs,etc.)

Regarding the MX bean this is an opensearch bug that is not affecting Wazuh functioning we have opened a public issue for opensearch to get more info and details on a workaround: [BUG] MX bean missing: G1 Concurrent GC error message · Issue #14974 · opensearch-project/OpenSearch (github.com)

Regards!

leon appel

unread,

Sep 13, 2024, 9:40:31 AM9/13/24

to Wazuh | Mailing List

Hi Julio

I have delete all my indexes and created the policies again and im no longer seeing some of those errors, however I did find that my primary indexer has failed overnight. I was able to start it ok though

node1 - Indexer Failed
root@WAZUH:~# systemctl status wazuh-indexer
× wazuh-indexer.service - wazuh-indexer
Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/wazuh-indexer.service.d
└─wazuh-indexer.conf
Active: failed (Result: exit-code) since Fri 2024-09-13 02:24:40 BST; 12h ago
Docs: https://documentation.wazuh.com
Process: 911 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=128)
Main PID: 911 (code=exited, status=128)
CPU: 15min 43.071s

Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at org.opensearch.ingest.IngestService.executePipelines(IngestService.java:601)
Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at org.opensearch.ingest.IngestService$3.doRun(IngestService.java:563)
Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913)
Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52)
Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Sep 13 02:24:39 DC1-WAZ01-VLL systemd-entrypoint[911]: at java.base/java.lang.Thread.run(Thread.java:1583)
Sep 13 02:24:40 DC1-WAZ01-VLL systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=128/n/a
Sep 13 02:24:40 DC1-WAZ01-VLL systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Sep 13 02:24:40 DC1-WAZ01-VLL systemd[1]: wazuh-indexer.service: Consumed 15min 43.071s CPU time.

Node1 cluster log
[2024-09-13T02:24:37,670][WARN ][o.o.p.c.u.JsonConverter ] [node-1] Json Mapping Error: Cannot invoke "java.lang.Long.longValue()" because "this.cacheMaxSize" is null (through reference chain: org.opensearch.performanceanalyzer.collectors.CacheConfigMetricsCollector$CacheMaxSizeStatus["Cache_MaxSize"])
[2024-09-13T02:24:39,718][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] fatal error in thread [opensearch[node-1][write][T#1]], exiting
java.lang.InternalError: a fault occurred in an unsafe memory access operation
at com.maxmind.db.Decoder.decodeInteger(Decoder.java:236) ~[?:?]
at com.maxmind.db.Decoder.decodeInteger(Decoder.java:231) ~[?:?]
at com.maxmind.db.Decoder.decodeInteger(Decoder.java:227) ~[?:?]
at com.maxmind.db.Decoder.decodeUint16(Decoder.java:207) ~[?:?]
at com.maxmind.db.Decoder.decodeByType(Decoder.java:184) ~[?:?]
at com.maxmind.db.Decoder.decode(Decoder.java:151) ~[?:?]
at com.maxmind.db.Decoder.decode(Decoder.java:76) ~[?:?]
at com.maxmind.db.Reader.resolveDataPointer(Reader.java:411) ~[?:?]
at com.maxmind.db.Reader.getRecord(Reader.java:185) ~[?:?]
at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:280) ~[?:?]
at com.maxmind.geoip2.DatabaseReader.getCity(DatabaseReader.java:365) ~[?:?]
at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:348) ~[?:?]
at org.opensearch.ingest.geoip.GeoIpProcessor.lambda$retrieveCityGeoData$0(GeoIpProcessor.java:227) ~[?:?]
at org.opensearch.ingest.geoip.IngestGeoIpPlugin$GeoIpCache.putIfAbsent(IngestGeoIpPlugin.java:206) ~[?:?]
at org.opensearch.ingest.geoip.GeoIpProcessor.lambda$retrieveCityGeoData$1(GeoIpProcessor.java:225) ~[?:?]
at java.base/java.security.AccessController.doPrivileged(AccessController.java:319) ~[?:?]
at org.opensearch.ingest.geoip.GeoIpProcessor.retrieveCityGeoData(GeoIpProcessor.java:224) ~[?:?]
at org.opensearch.ingest.geoip.GeoIpProcessor.getGeoData(GeoIpProcessor.java:175) ~[?:?]
at org.opensearch.ingest.geoip.GeoIpProcessor.execute(GeoIpProcessor.java:137) ~[?:?]
at org.opensearch.ingest.Processor.execute(Processor.java:68) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:150) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:182) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:155) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:182) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.Processor.execute(Processor.java:73) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:150) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:182) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:155) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:182) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.Processor.execute(Processor.java:73) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:150) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.lambda$innerExecute$1(CompoundProcessor.java:182) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.Processor.execute(Processor.java:73) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.innerExecute(CompoundProcessor.java:164) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:150) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.Pipeline.execute(Pipeline.java:133) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.IngestDocument.executePipeline(IngestDocument.java:804) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.IngestService.innerExecute(IngestService.java:754) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.IngestService.executePipelines(IngestService.java:601) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.ingest.IngestService$3.doRun(IngestService.java:563) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:913) ~[opensearch-2.13.0.jar:2.13.0]
at org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:52) ~[opensearch-2.13.0.jar:2.13.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]

Kind Regards

Reply all

Reply to author

Forward