Cannot Start wazuh-indexer service

John

unread,

Aug 5, 2023, 10:59:03 AM8/5/23

to Wazuh mailing list

Hi,

I have an all-in-one instance of Wazuh that failed a couple of days ago. I am unable to start the wazuh-indexer service. Any help is much apprecaited!

```
[root@wazuh wazuh-indexer]# systemctl status wazuh-indexer
● wazuh-indexer.service - Wazuh-indexer
Loaded: loaded (/usr/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/wazuh-indexer.service.d
└─override.conf
Active: failed (Result: exit-code) since Sat 2023-08-05 14:45:30 UTC; 6min ago
Docs: https://documentation.wazuh.com
Process: 9189 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 9189 (code=exited, status=1/FAILURE)

Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:147)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.cli.Command.main(Command.java:101)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Aug 05 14:45:30 wazuh systemd-entrypoint[9189]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
```

- Here's an excerpt from my /var/log/wazuh-indexer/wazuh-cluster.log:

[2023-08-05T03:36:10,757][WARN ][stderr ] [node-1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-08-05T03:36:10,757][WARN ][stderr ] [node-1] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-08-05T03:36:10,757][WARN ][stderr ] [node-1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-08-05T03:36:10,784][INFO ][o.o.s.s.t.SSLConfig ] [node-1] SSL dual mode is disabled
[2023-08-05T03:36:10,785][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] OpenSearch Config path is /etc/wazuh-indexer
[2023-08-05T03:36:12,395][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] JVM supports TLSv1.3
[2023-08-05T03:36:12,396][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Config directory is /etc/wazuh-indexer/, from there the key- and truststore files are resolved relatively
[2023-08-05T03:36:15,109][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Client Provider : JDK
[2023-08-05T03:36:15,109][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS Transport Server Provider : JDK
[2023-08-05T03:36:15,109][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] TLS HTTP Provider : JDK
[2023-08-05T03:36:15,109][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for transport layer : [TLSv1.3, TLSv1.2]
[2023-08-05T03:36:15,110][INFO ][o.o.s.s.DefaultSecurityKeyStore] [node-1] Enabled TLS protocols for HTTP layer : [TLSv1.3, TLSv1.2]
[2023-08-05T03:36:15,119][INFO ][o.o.s.OpenSearchSecurityPlugin] [node-1] Clustername: wazuh-cluster
[2023-08-05T03:36:57,427][INFO ][o.o.p.c.PluginSettings ] [node-1] Trying to create directory /dev/shm/performanceanalyzer/.
[2023-08-05T03:36:57,427][INFO ][o.o.p.c.PluginSettings ] [node-1] Config: metricsLocation: /dev/shm/performanceanalyzer/, metricsDeletionInterval: 1, httpsEnabled: false, cleanup-metrics-db-files: true, batch-metrics-retention-period-minutes: 7, rpc-port: 9650, webservice-port 9600
[2023-08-05T03:37:20,319][INFO ][o.o.i.r.ReindexPlugin ] [node-1] ReindexPlugin reloadSPI called
[2023-08-05T03:37:20,320][INFO ][o.o.i.r.ReindexPlugin ] [node-1] Unable to find any implementation for RemoteReindexExtension
[2023-08-05T03:37:21,271][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro_anomaly_detector, index: .opendistro-anomaly-detector-jobs[2023-08-05T03:37:23,369][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: reports-scheduler, index: .opendistro-reports-definitions
[2023-08-05T03:37:23,371][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: opendistro-index-management, index: .opendistro-ism-config
[2023-08-05T03:37:26,275][INFO ][o.o.j.JobSchedulerPlugin ] [node-1] Loaded scheduler extension: observability, index: .opensearch-observability-job
[2023-08-05T03:37:26,452][INFO ][o.o.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2023-08-05T03:37:26,452][INFO ][o.o.p.PluginsService ] [node-1] loaded module [analysis-common]
[2023-08-05T03:37:26,452][INFO ][o.o.p.PluginsService ] [node-1] loaded module [geo]
[2023-08-05T03:37:26,452][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-common]
[2023-08-05T03:37:26,453][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-geoip]
[2023-08-05T03:37:26,453][INFO ][o.o.p.PluginsService ] [node-1] loaded module [ingest-user-agent]
[2023-08-05T03:37:26,453][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-expression]
[2023-08-05T03:37:26,453][INFO ][o.o.p.PluginsService ] [node-1] loaded module [lang-mustache]

...

org.opensearch.node.NodeClosedException: node closed {node-1}{IZn6RY3ATXyqjj9yqUH4tQ}{j0t_SppCRyCst4KhhH5Daw}{127.0.0.1}{127.0.0.1:9300}{dimr}{shard_indexing_pressure_enabled=true}
at org.opensearch.action.admin.cluster.health.TransportClusterHealthAction$3.onClusterServiceClose(TransportClusterHealthAction.java:311) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.cluster.ClusterStateObserver$ContextPreservingListener.onClusterServiceClose(ClusterStateObserver.java:387) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onClose(ClusterStateObserver.java:276) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.cluster.service.ClusterApplierService.doStop(ClusterApplierService.java:195) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.cluster.service.ClusterService.doStop(ClusterService.java:141) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.common.component.AbstractLifecycleComponent.stop(AbstractLifecycleComponent.java:97) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.node.Node.stop(Node.java:1361) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.node.Node.close(Node.java:1387) [opensearch-2.6.0.jar:2.6.0]
at org.opensearch.core.internal.io.IOUtils.close(IOUtils.java:89) [opensearch-core-2.6.0.jar:2.6.0]
at org.opensearch.core.internal.io.IOUtils.close(IOUtils.java:131) [opensearch-core-2.6.0.jar:2.6.0]
at org.opensearch.core.internal.io.IOUtils.close(IOUtils.java:81) [opensearch-core-2.6.0.jar:2.6.0]
at org.opensearch.bootstrap.Bootstrap$4.run(Bootstrap.java:206) [opensearch-2.6.0.jar:2.6.0]
[2023-08-05T14:15:48,996][INFO ][o.o.n.Node ] [node-1] stopped
[2023-08-05T14:15:48,997][INFO ][o.o.n.Node ] [node-1] closing ...
[2023-08-05T14:15:49,037][INFO ][o.o.s.a.i.AuditLogImpl ] [node-1] Closing AuditLogImpl
[2023-08-05T14:15:49,041][INFO ][o.o.n.Node ] [node-1] closed

Olusegun Adenrele Oyebo

unread,

Aug 6, 2023, 5:56:39 AM8/6/23

to John, Wazuh mailing list

Dear John,

Thank you for using wazuh and also sharing the logs.

For an all-in-one deployment, there is minimum specification recommended when trying to set up your server:

Monitoring 1-25 agents: 4 vCPU, 8 GiB of RAM, 50GB of storage
Monitoring 25-50 agents: 8 vCPU, 8 GiB of RAM, 100GB of storage
Monitoring 50-100 agents: 8 vCPU, 8 GiB of RAM, 200GB of storage

The above specifications are subject to 90 days of queryable/indexed alert data:
https://documentation.wazuh.com/current/quickstart.html#Requirements

Kindly confirm that you followed the steps as described in the official documentation in deploying the components on the server.

Can you also confirm the current resource utilization and specification on the server. You can get utilization and specs by running the below commands:

df -H (free storage)
top (memory/cpu utilization)
lscpu (CPU specification)
cat /proc/meminfo (RAM specification)

After reviewing the wazun-cluster.log file, it shows that there seems to be an issue with initializing the cluster for the wazuh indexer. You can try to run the below command for cluster initialization:
/usr/share/wazuh-indexer/bin/indexer-security-init.sh
https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/step-by-step.html#cluster-initialization

We hope this was helpful. Do not hesitate to contact us further if you have any other query.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/852b34c9-911d-432f-8a69-2606cb20e149n%40googlegroups.com.

John

unread,

Aug 6, 2023, 7:25:36 AM8/6/23

to Wazuh mailing list

Hi there.

Output of the indexer init:

[root@wazuh wazuh-user]# /usr/share/wazuh-indexer/bin/indexer-security-init.sh
**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755 **
**************************************************************************
Security Admin v7
Will connect to 127.0.0.1:9200
ERR: Seems there is no OpenSearch running on 127.0.0.1:9200 - Will exit

Test the cluster (here the user/password is masked):

[root@wazuh wazuh-user]# curl -k -u <user>:<pass> https://192.168.1.33:9200
curl: (7) Failed connect to 192.168.1.33:9200; Connection refused

On the manager gui:

Wazuh dashboard server is not ready yet

thank you!

Olusegun Adenrele Oyebo

unread,

Aug 6, 2023, 12:07:07 PM8/6/23

to John, Wazuh mailing list

Dear John,

Thank you for your response.

Kindly revert with the above information as stated earlier. Current system utilization and specifications:

df -H (free storage)
top (memory/cpu utilization)
lscpu (CPU specification)
cat /proc/meminfo (RAM specification)

Also help with the below additional information and logs by running the below commands:

journalctl -u wazuh-indexer
journalctl -u wazuh-manager
journalctl -u filebeat
journalctl -u wazuh-dashboard
ossec.log file. You can get it in the directory /var/ossec/logs/ossec.log

We will be expecting your feedback so as to assist you accordingly.

Best regards.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/76089805-1d17-4a70-b3f0-87e4ce575b3en%40googlegroups.com.

Message has been deleted

Olusegun Adenrele Oyebo

unread,

Aug 26, 2023, 3:38:10 AM8/26/23

to John, Wazuh mailing list

Dear John,

Thanks for the confirmation. Do not hesitate to engage us further in case you have any questions or concerns.

Best regards.

On Thu, Aug 24, 2023 at 2:24 PM John <gal...@gmail.com> wrote:

Issue resolved. Thank you!

You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/63c13927-5e2a-499f-95fa-1bc8f70046c7n%40googlegroups.com.

Reply all

Reply to author

Forward

Message has been deleted