wazuh-indexer not starting

[Paulo Fernandes]

Hello all,

I installed Wazuh in Ubuntu server. I added 3 agents and everything seem to be working fine.

Today I restarted the server and now wazuh-indexer is not starting, I tried to restart the service, but without success.

I also tried to run "bash wazuh-install.sh --start-cluster", but without success, wazuh-indexer wont start.

Any ideas of what the problem could be?

Thanks in advance for your help.

Here is the Journal report:

░░ The process' exit code is 'exited' and its exit status is 1.
Sep 13 10:50:16 3cpo systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit wazuh-indexer.service has entered the 'failed' state with result 'exit-code'.
Sep 13 10:50:16 3cpo systemd[1]: Failed to start Wazuh-indexer.
░░ Subject: A start job for unit wazuh-indexer.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit wazuh-indexer.service has finished with a failure.
░░
░░ The job identifier is 171 and the job result is failed.
Sep 13 10:50:16 3cpo systemd[1]: wazuh-indexer.service: Consumed 1.909s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit wazuh-indexer.service completed and consumed the indicated resources.
Sep 13 10:54:56 3cpo systemd[1]: Starting Wazuh-indexer...
░░ Subject: A start job for unit wazuh-indexer.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit wazuh-indexer.service has begun execution.
░░
░░ The job identifier is 586.
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: Exception in thread "main" java.lang.RuntimeException: starting java failed with [1]
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: output:
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: [0.000s][error][logging] Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: [0.000s][error][logging] Initialization of output 'file=/var/log/wazuh-indexer/gc.log' using options 'filecount=32,filesize=64m' failed.
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: error:
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: Invalid -Xlog option '-Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m', see error log for details.
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: Error: Could not create the Java Virtual Machine.
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]: Error: A fatal exception has occurred. Program will exit.
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]:         at org.opensearch.tools.launchers.JvmErgonomics.flagsFinal(JvmErgonomics.java:125)
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]:         at org.opensearch.tools.launchers.JvmErgonomics.finalJvmOptions(JvmErgonomics.java:87)
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]:         at org.opensearch.tools.launchers.JvmErgonomics.choose(JvmErgonomics.java:70)
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]:         at org.opensearch.tools.launchers.JvmOptionsParser.jvmOptions(JvmOptionsParser.java:150)
Sep 13 10:54:57 3cpo systemd-entrypoint[1429]:         at org.opensearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:108)
Sep 13 10:54:57 3cpo systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit wazuh-indexer.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Sep 13 10:54:57 3cpo systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit wazuh-indexer.service has entered the 'failed' state with result 'exit-code'.
Sep 13 10:54:57 3cpo systemd[1]: Failed to start Wazuh-indexer.
░░ Subject: A start job for unit wazuh-indexer.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit wazuh-indexer.service has finished with a failure.
░░
░░ The job identifier is 586 and the job result is failed.
Sep 13 10:54:57 3cpo systemd[1]: wazuh-indexer.service: Consumed 1.824s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit wazuh-indexer.service completed and consumed the indicated resources.

Thanks,

Best regards,

Paulo Fernandes

[Md. Nazmur Sakib]

Hi Paulo Fernandes,

Hope you are doing well. Thanks for using Wazuh!

As I can see in the screenshot you shared that JVM memory is not able to allocate that why it is not able to start. In order to start it please check these options below and restart it accordingly:

You can tune these options to increase the size of memory available for the indexer in the file: /etc/wazuh-indexer/jvm.options. There, add the number of Gigabytes of RAM you want to allocate to the Wazuh-indexer's heap. It is recommendable to set it to half of the available RAM with a maximum of 32GB. So for example, if you have 8GB of memory, you would allocate 4 as follows:

# Xms represents the initial size of total heap space 

# Xmx represents the maximum size of total heap space 

 -Xms4g 

-Xmx4g

Next, restart Wazuh-indexer:

systemctl daemon-reload

systemctl restart wazuh-indexer

For correct installation of Wazuh-Indexer, Wazuh-Dashboard & Wazuh-manager also please check out this: https://documentation.wazuh.com/current/installation-guide/

Please let me know if this solves your issue or if you need any further help.

Regards

Md. Nazmur Sakib

[Paulo Fernandes]

Hi Nazmur Sabik,

First of all, thank you very much for your help.

I tried your suggestion, but without luck. I had 3000m, and updated jvm.options to 4g, like you suggested. Restarted the Wazuh-indexer and still got the same error.

On this system I have 8GB of RAM, I also tried to increase to 6GB the jvm.options, but with the same result.

At the installation I followed the Wazuh documentation. I used the Wazuh installation assistant.

Any other idea? Or should I try to do a fresh install?

For now this machine is dedicated to Wazuh.

Thanks,

Best regards,

Paulo Fernandes

[Md. Nazmur Sakib]

Hi Paulo Fernandes,

Sorry to hear that the issue is not resolved yet. 

Yes, you can do a fresh install if you do not have any dependencies and if you do not spend more time troubleshooting. 

But if you want we can also look into the issue.

For that I need you to send the status and current log of your wazuh-indexer after making changes in jvm option. Also, make sure not to keep empty space before the line:

-Xms4g 

-Xmx4g

Empty space will cause misconfiguration as it is case-sensitive.

Restart the wazuh-indexer and wazuh-manager after changing.

systemctl restart wazuh-indexer

systemctl restart wazuh-manager

Check the status.

If you still see the error 

Error opening log file '/var/log/wazuh-indexer/gc.log': No such file or directory

Create a gc.log file.

touch /var/log/wazuh-indexer/gc.log

chown wazuh-indexer:wazuh-indexer /var/log/wazuh-indexer/gc.log

chmod 644 /var/log/wazuh-indexer/gc.log

Restart the wazuh-indexer and wazuh-manager after changing.

systemctl restart wazuh-indexer

systemctl restart wazuh-manager

The problem you are having here is clearly for jvm.options. If you are doing a fresh install. I would recommend checking the java jdk version before installing if it is 9 or above.

java --version

Also sharing the installation document for your reference:

https://documentation.wazuh.com/current/installation-guide/wazuh-indexer/installation-assistant.html

Please let me know if this helps. 

Regards

[Paulo Fernandes]

Hi Nazmur Sakib,

It work's, it's alive.

Creating the "gc.log" file manually solved the issue, I followed every step and Wazuh started working right away. 

But now I have a new problem, after I rebooted Wazuh he is not responding on the browser. It's giving some errors.

Here is one, it also gave me an API error:

Error: An internal server error occurred. at createError (https://192.168.1.11/45202/bundles/plugin/wazuh/wazuh.plugin.js:2:28658) at settle (https://192.168.1.11/45202/bundles/plugin/wazuh/wazuh.plugin.js:8:19613) at XMLHttpRequest.onloadend (https://192.168.1.11/45202/bundles/plugin/wazuh/wazuh.plugin.js:2:26451)

I noticed that this is related to Wazuh-manager service that is not coming up after the reboot. If I start him manually (systemctl start wazuh-manager) everything starts working fine again.

Also when I run "java --version" I did not have any install, so I installed version 19:

openjdk 19.0.2 2023-01-17
OpenJDK Runtime Environment (build 19.0.2+7-Ubuntu-0ubuntu322.04)
OpenJDK 64-Bit Server VM (build 19.0.2+7-Ubuntu-0ubuntu322.04, mixed mode, sharing)

Could this be the problem?

Thanks,

Best regards,

Paulo Fernandes

[Md. Nazmur Sakib]

Hi Paulo Fernandes,

Can you try restarting the manager and dashboard and check if the you can access the user interface.

systemctl restart wazuh-manager

systemctl restart wazuh-dashboard

If not check if there is any error log in ossec

cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

tail /var/ossec/logs/ossec.log

Please let me know the findings.

[Paulo Fernandes]

Hi Nazmur Sakib,

I run the:

systemctl restart wazuh-manager

systemctl restart wazuh-dashboard

After this, I can access the system via the user interface without any issues.

Here is the error logs:

~$ sudo cat /var/ossec/logs/ossec.log | grep -i -E "error|warn"

2023/09/14 10:00:46 wazuh-authd: WARNING: Duplicate name 'guardserver', rejecting enrollment. Agent '001' can't be replaced since it is not disconnected.
~$ sudo tail /var/ossec/logs/ossec.log

2023/09/14 11:28:15 wazuh-modulesd:task-manager: INFO: (8200): Module Task Manager started.
2023/09/14 11:28:15 wazuh-modulesd:download: INFO: Module started.
2023/09/14 11:28:15 wazuh-modulesd:database: INFO: Module started.
2023/09/14 11:28:15 wazuh-modulesd:control: INFO: Starting control thread.
2023/09/14 11:28:15 wazuh-modulesd:syscollector: INFO: Module started.
2023/09/14 11:28:15 wazuh-modulesd:syscollector: INFO: Starting evaluation.
2023/09/14 11:28:15 sca: INFO: Starting evaluation of policy: '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml'
2023/09/14 11:28:15 wazuh-modulesd:syscollector: INFO: Evaluation finished.
2023/09/14 11:28:20 sca: INFO: Evaluation finished for policy '/var/ossec/ruleset/sca/cis_ubuntu22-04.yml'
2023/09/14 11:28:20 sca: INFO: Security Configuration Assessment scan finished. Duration: 5 seconds.

[Paulo Fernandes]

Hi Nazmur,

Some more information about the previous message, after reboot the system I will have the same problem, for Wazuh to run, I will have to restart the "wazuh-manager" and "wazuh-dashboard".

Thanks,

Best regards,

Paulo Fernandes