Wazuh-indexer failed (result: exit-code)

[Gavin Wallace]

Good afternoon,

On thursday of last week I upgraded my Wazuh installation, all in one, and since then I have not been able to access the wazuh-dashboard. 

I have been searching through the google group and have tried a few different troubleshooting methods and I believe I have found the issue but I need some help fixing the problem. I believe the issue is with permissions related to the indexer.pem file. Running the command:

sudo cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

I get the following output:

[2023-04-04T12:14:43,525][INFO ][o.o.n.Node               ] [node-1] JVM arguments [-Xshare:auto, -Dopensearch.networkaddress.cache.ttl=60, -Dopensearch.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms64415m, -Xmx64415m, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/opensearch-15632558769880172476, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=/var/log/wazuh-indexer/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/wazuh-indexer/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Dclk.tck=100, -Djdk.attach.allowAttachSelf=true, -Djava.security.policy=file:///usr/share/wazuh-indexer/plugins/opendistro-performance-analyzer/pa_config/es_security.policy, -XX:MaxDirectMemorySize=33772535808, -Dopensearch.path.home=/usr/share/wazuh-indexer, -Dopensearch.path.conf=/etc/wazuh-indexer, -Dopensearch.distribution.type=rpm, -Dopensearch.bundled_jdk=true]
[2023-04-04T12:14:44,369][WARN ][stderr                   ] [node-1] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[2023-04-04T12:14:44,369][WARN ][stderr                   ] [node-1] SLF4J: Defaulting to no-operation (NOP) logger implementation
[2023-04-04T12:14:44,369][WARN ][stderr                   ] [node-1] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[2023-04-04T12:14:44,674][ERROR][o.o.b.Bootstrap          ] [node-1] Exception
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) [opensearch-cli-2.4.1.jar:2.4.1]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]
[2023-04-04T12:14:44,682][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
        at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-2.4.1.jar:2.4.1]
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]

Can anyone help me correct the error here? Thank you for developing such a wonderful product, and thanks in advance for any help,

Gavin

[Guido Iván García]

Hello Gavin, thanks for posting in the community!

Based on the log output you provided, it seems that the Wazuh installation is encountering an error related to the indexer.pem file. Specifically, the error message suggests that the file is not readable due to incorrect permissions or does not exist.

To resolve this issue, you may want to try the following:

First, verify that the file exists with the following command:
ls -la /etc/wazuh-indexer/certs/
If it does not exist and there is a file with the name "wazuh-indexer.pem", try to replace it with "indexer.pem".

If it exists, check the permissions of the file using the following command:
ls -l /etc/wazuh-indexer/certs/indexer.pem

Make sure that the file is readable by the user running the Wazuh service. If the permissions are incorrect, you can change them using the following command:
chmod 644 /etc/wazuh-indexer/certs/indexer.pem

This will give read permission to the owner and group of the file, and read permission to others.

Once you have updated the permissions, you may need to restart the Wazuh service for the changes to take effect:
systemctl restart wazuh-manager

After the service has been restarted, check the logs again using the same command you used before:

sudo cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -i -E "error|warn"

If the issue has been resolved, you should not see any errors related to the indexer.pem file.

I hope this helps. Let me know if you have any specific questions or if there is anything else I can do to help

Regards,
Guido

[Gavin Wallace]

Guido,

Thank you for the reply!

The filed indexer.pem did not exist so I used the cp command to copy wazuh-indexer.pem and renamed the copy indexer.pem. Please let me know if I should use another command to better make the change.

After creating the indexer.pem file I altered the permission so that they read -rw-r--r-- using the command provided, then I restarted the wazuh-manager. Unfortunately the indexer log still proivides the same error of which I copied the relavent bit below. 

Gavin

Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: OpenSearchException[Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath]

[Gavin Wallace]

Alternatively,

Is there a simple way to point Opensearch at wazuh-indexer.pem? Thank you for all your help so far,

Gavin

[DFIR AU]

This is a workaround/fix for a default install of 4.4.0 following an upgrade from 4.3.10 - assuming you don't have any custom certs/keys defined.

Wazuh-indexer

Apr 06 13:07:57 wuzah systemd-entrypoint[4065]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Apr 06 13:07:57 wuzah systemd[1]: wazuh-indexer.service: Main process exited, code=exited, status=1/FAILURE
Apr 06 13:07:57 wuzah systemd[1]: wazuh-indexer.service: Failed with result 'exit-code'.
Apr 06 13:07:57 wuzah systemd[1]: Failed to start Wazuh-indexer.

Caused by: org.opensearch.OpenSearchException: Unable to read /etc/wazuh-indexer/certs/indexer.pem (/etc/wazuh-indexer/certs/indexer.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemcert_filepath
Caused by: org.opensearch.OpenSearchException: Unable to read /etc/wazuh-indexer/certs/indexer-key.pem (/etc/wazuh-indexer/certs/indexer-key.pem). Please make sure this files exists and is readable regarding to permissions. Property: plugins.security.ssl.transport.pemkey_filepath

Wazuh-dashboard

 × wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-04-06 13:17:46 ACST; 1min 38s ago
    Process: 814 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards -c /etc/wazuh-dashboard/opensearch_dashboards.yml (code=exited, status=1/FAILURE)
   Main PID: 814 (code=exited, status=1/FAILURE)
        CPU: 14.766s

[814]:  FATAL  Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard-key.pem'
[4104]:  FATAL  Error: ENOENT: no such file or directory, open '/etc/wazuh-dashboard/certs/dashboard.pem'

Workaround / solution

 You could copy certs and keys to the new/expected location, however then you have another location to manage/update. Alternatively, create a symbolic link (instructions below for Linux) pointing the expected location to the original/existing location.

ln -s /etc/wazuh-indexer/certs/wazuh-indexer.pem /etc/wazuh-indexer/certs/indexer.pem
ln -s /etc/wazuh-indexer/certs/wazuh-indexer-key.pem /etc/wazuh-indexer/certs/indexer-key.pem
ln -s /etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem /etc/wazuh-dashboard/certs/dashboard-key.pem
ln -s /etc/wazuh-dashboard/certs/wazuh-dashboard.pem /etc/wazuh-dashboard/certs/dashboard.pem