Unable to start Wazuh-Indexer [curl: (7) Failed to connect to XXXX port 9200: Connection refused]

[Shiva Gujjanti]

Hi Team,

Need help Urgently

I'm facing issue to starting the Wazuh-Indexer services and cant access the console, However i can see Wazuh-dashboard, Wazuh-manager is up and running fine. Please help me to access the console and start the Wazuh-indexer.

Currently we are using Wazuh 4.3

Below is the error messages when i run systemctl status wazuh-indexer,  filebeat test output and attached journalctl -u wazuh-indexer output.

root@SOC Server:/# systemctl status wazuh-indexer.service
● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-12-25 14:57:50 UTC; 13s ago
       Docs: https://documentation.wazuh.com
    Process: 7828 ExecStart=/usr/share/wazuh-indexer/bin/systemd-entrypoint -p ${PID_DIR}/wazuh-indexer.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 7828 (code=exited, status=1/FAILURE)

Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]: Error: Could not create the Java Virtual Machine.
Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]: Error: A fatal exception has occurred. Program will exit.
Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]:         at org.opensearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:139)
Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]:         at org.opensearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:101)
Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]:         at org.opensearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:72)
Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]:         at org.opensearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:152)
Dec 25 14:57:50 SOC Server systemd-entrypoint[7955]:         at org.opensearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:110)
Dec 25 14:57:50 SOC Server systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Dec 25 14:57:50 SOC Server systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Dec 25 14:57:50 SOC Server systemd[1]: Failed to start Wazuh-indexer.

root@SOC Server:/# filebeat test output
elasticsearch: https://X.X.X.X:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: X.X.X.X
    dial up... ERROR dial tcp X.X.X.X:9200: connect: connection refused

curl: (7) Failed to connect to X.X.X.X port 9200: Connection refused

[Shiva Gujjanti]

Hi Team,

Can someone help me rectify this issue at the earliest! 

[Marcos Darío Buslaiman]

Hi, 

Thanks for using Wazuh!

Please, could you share some information about your environment and the scenario of this issue?

Do you have an all-in-one installation? in case not, How many nodes have you installed for indexer?
Have you followed some installation guides?
According to the log, this issue could be related to permissions or misconfiguration of Java.

Could you check the version of java with the following command:
java -version
And the memory available on your server
free -h 
In the following file, you have the memory assigned to the Wazuh indexer, there we can check the values "Xms" and "Xmx"
cat /etc/wazuh-indexer/jvm.options.

The recommendation for these values should be the following:

• Use no more than 50% of available RAM.

• Use no more than 32 GB.

Please when you gather this information share it with us, so we can find out the cause of the issue.
Thanks!

[Shiva Gujjanti]

Hi Marcos,

We have followed this guide to deploy 4.3 https://documentation.wazuh.com/current/installation-guide/index.html  Wherein central components are deployed with assistance and all the components on a single host with only one node.

root@SOC-infra:~# java -version
openjdk version "11.0.17" 2022-10-18
OpenJDK Runtime Environment (build 11.0.17+8-post-Ubuntu-1ubuntu220.04)
OpenJDK 64-Bit Server VM (build 11.0.17+8-post-Ubuntu-1ubuntu220.04, mixed mode, sharing)

root@SOC-infra:~# free -h
              total        used        free      shared  buff/cache   available
Mem:           31Gi       2.1Gi       282Mi        20Mi        29Gi        28Gi
Swap:            0B          0B          0B


## -Xms8g
## -Xmx8g

Quick responses will help us.

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/duZRTrH36TQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/ee978241-ac86-4d71-81fc-e80cf153addfn%40googlegroups.com.

[Marcos Darío Buslaiman]

We will need to verify the following:
Please, check the version of Java on the wazuh indexer path 
/usr/share/wazuh-indexer/jdk/bin/java -version

Please execute the following command and share the output
grep -iE 'WARN|ERR' /var/log/wazuh-indexer/wazuh-cluster.log

Have you installed this from scratch or its an upgrade from an elasticsearch previous installation?
Have you had disk space problems recently?
Have you performed some changes before this issue occurs?

Thanks!

[Shiva Gujjanti]

Here you go,

root@SOC-infra:/# /usr/share/wazuh-indexer/jdk/bin/java -version
openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)

grep -iE 'WARN|ERR' /var/log/wazuh-indexer/wazuh-cluster.log

I wasn't able to locate these cluster logs, I have checked in /var/log and couldn't find thi cluster

Have you installed this from scratch or is it an upgrade from an elasticsearch previous installation? 
Yes, we have 4.2, recently I have erased everything and deployed 4.3 

Have you had disk space problems recently? 

NO

Have you performed some changes before this issue occurs?

Not really

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/9823a6e1-8a52-484c-99dd-8ca4024376a5n%40googlegroups.com.

[Marcos Darío Buslaiman]

Hi,

We will need to keep performing some checks, please, could you send the output of the following commands:
According to your SO, run this in order to verify the versions installed.
yum list installed | grep -i "wazuh\|elastic\|kibana"
or
apt list --installed | grep -i  "wazuh\|elastic\|kibana"

The wazuh-indexer configuration file, (please remove the IPs and other sensitive information)
cat /etc/wazuh-indexer/opensearch.yml

And this command:
journalctl -u wazuh-indexer -f -n 200

[Shiva Gujjanti]

Hi Marcos,

root@SOC-infra:/# apt list --installed | grep -i  "wazuh\|elastic\|kibana"

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
elasticsearch-oss/stable,now 7.10.2 amd64 [installed]
wazuh-dashboard/stable,now 4.3.10-1 amd64 [installed]
wazuh-indexer/stable,now 4.3.10-1 amd64 [installed]
wazuh-manager/stable,now 4.3.10-1 amd64 [installed]

cat /etc/wazuh-indexer/opensearch.yml

Attached txt file

journalctl -u wazuh-indexer -f -n 200

Dec 26 20:59:28 SOC-infra systemd[1]: Failed to start Wazuh-indexer.
Dec 27 07:39:46 SOC-infra systemd[1]: Starting Wazuh-indexer...
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: output:
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: [0.000s][error][logging] Initialization of output 'file=/var/log/wazuh-indexer/gc.log' using options 'filecount=32,filesize=64m' failed.
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: error:
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: Error: Could not create the Java Virtual Machine.
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]: Error: A fatal exception has occurred. Program will exit.
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]:         at org.opensearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:139)
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]:         at org.opensearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:101)
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]:         at org.opensearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:72)
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]:         at org.opensearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:152)
Dec 27 07:39:47 SOC-infra systemd-entrypoint[73449]:         at org.opensearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:110)
Dec 27 07:39:47 SOC-infra systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Dec 27 07:39:47 SOC-infra systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Dec 27 07:39:47 SOC-infra systemd[1]: Failed to start Wazuh-indexer.

[Marcos Darío Buslaiman]

Hi, seems that could be a problem with the permissions or something misconfigured related to java.
Please could you verify if the following folder has been created and which permission it has?
ls -ltrh /var/log/wazuh-indexer/

Then we'll need to verify the java files and folder
ls -ltrh /usr/share/wazuh-indexer/jdk/bin/java

ls -ltrh /usr/share/wazuh-indexer/jdk/bin/java*

Then which processes are running 
ps -fea | grep "java\|elastic\|wazuh"

And also, we need to verify the file  /etc/wazuh-indexer/jvm.options so please send me the output of these:

ls -ltrh  /etc/wazuh-indexer
ls -ltrh  /etc/wazuh-indexer/jvm.options
cat /etc/wazuh-indexer/jvm.options

Thanks!

[Shiva Gujjanti]

Hi Marcos,

I have shared all the logs separately, Did you get a chance to check what's going wrong and please let me know when we should connect to rectify this issue at the earliest. 

A quick solution will really help us.

[Marcos Darío Buslaiman]

Hi,

Yes, I have been checking the logs and the outputs of the commands and I think that could be related to some permission of the Java process or maybe some misconfiguration of it, for that reason, I think with the last few commands I asked you to run, we could validate some issues with Java.

Please could you send me the output of the following commands:
To verify if the folder has been created and which permission it has?

ls -ltrh /var/log/wazuh-indexer/

Then we'll need to verify the java files and folder
ls -ltrh /usr/share/wazuh-indexer/jdk/bin/java

ls -ltrh /usr/share/wazuh-indexer/jdk/bin/java*

Then which processes are running 
ps -fea | grep "java\|elastic\|wazuh"

And also, we need to verify the file /etc/wazuh-indexer/jvm.options so please send me the output of these:
ls -ltrh  /etc/wazuh-indexer
ls -ltrh  /etc/wazuh-indexer/jvm.options
cat /etc/wazuh-indexer/jvm.options

Thanks