Wazuh-indexer not starting

[Roye R]

Installation of all roles on one server.  Numerous agents running for half a year without issue.  Suddenly couldn't login and checked the server and the indexer won't start.  Error in journalctl is:

systemd-entrypoint[180836]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup/internal_users.yml

 systemd-entrypoint[180836]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup/internal_users.yml

I've checked the directory and the permissions were set to root.  I changed it to the wazuh-indexer and still get the error.

Nothing else is failed on the server.  

Any help would be appreciated.  This is maddening.

[Felix Bocco]

Hello Roye,

We will assume that the backup directory has the minimum necessary permissions so Wazuh indexer service can be restarted.

Did you try to get health check data from the API?
From the server use:
curl -k -u <user>:<password> https://<WAZUH_INDEXER_IP>:9200/_cluster/health

Where,
user: the user to connect to Wazuh. Ex: admin.
password: user's password.
WAZUH_INDEXER_IP: The IP server address where the indexer is installed.

Had you check the disk space used?
This can be done with df -h .

Had you check the indexer log? Did you find any ERROR/WARNING messages regarding this issue?
To check indexer log, go to path /var/log/wazuh-indexer/<CLUSTER_NAME>.log (where CLUSTER_NAME is the name of the cluster, by default is wazuh-cluster.log)

We suggest you to run a grep command as follows:

grep -Ei "error|warn" /var/log/wazuh-indexer/<CLUSTER_NAME>.log


Please let us know your findings.

[Roye R]

The ls from the backup folder:

drw-------  2 wazuh-indexer wazuh-indexer 4.0K Oct 10 18:06 .
drwxr-x--- 11 wazuh-indexer wazuh-indexer 4.0K Oct 10 20:21 ..
-rw-------  1 wazuh-indexer wazuh-indexer   50 Oct 10 18:12 action_groups.yml
-rw-------  1 wazuh-indexer wazuh-indexer 2.0K Oct 10 18:12 allowlist.yml
-rw-------  1 wazuh-indexer wazuh-indexer 2.5K Oct 10 18:12 audit.yml
-rw-------  1 wazuh-indexer wazuh-indexer 9.8K Oct 10 18:12 config.yml
-rw-rw-rw-  1 wazuh-indexer wazuh-indexer 1.3K Oct 10 18:12 internal_users.yml
-rw-------  1 wazuh-indexer wazuh-indexer  154 Oct 10 18:12 nodes_dn.yml
-rw-------  1 wazuh-indexer wazuh-indexer  13K Oct 10 18:12 opensearch.yml.example
-rw-------  1 wazuh-indexer wazuh-indexer 1.5K Oct 10 18:12 roles_mapping.yml
-rw-------  1 wazuh-indexer wazuh-indexer 4.6K Oct 10 18:12 roles.yml
-rw-------  1 wazuh-indexer wazuh-indexer  170 Oct 10 18:12 tenants.yml
-rw-------  1 wazuh-indexer wazuh-indexer 2.0K Oct 10 18:12 whitelist.yml

Connection refused on the curl command.  

Disk space isn't an issue, 88 Gb free.

Nothing in the logs current.  Last entry was from last week when it was still running.  I renamed it to see if the log file would be created but nothing yet.

[Roye R]

More info to make this more strange; I'm still getting email alerts from Wazuh.  So it would seem that something is running, but the Indexer and Dashboard are unavailable.

[Felix Bocco]

Hello Roye,


Can you please check that there is no file/directory which could need execution permissions. Also check if the wazuh-indexer group could be needing read-write permissions. Take as reference the main wazuh-indexer directory:

ls -la  wazuh-indexer

total 48
drwxr-x---.  10 wazuh-indexer wazuh-indexer  4096 Oct 17 14:46 .
drwxr-xr-x. 130 root          root           8192 Oct 17 14:44 ..
dr-x------.   2 wazuh-indexer wazuh-indexer   117 Oct 17 14:30 certs
-rw-rw----.   1 wazuh-indexer wazuh-indexer  2735 Oct 17 14:30 jvm.options
drwxr-x---.   2 wazuh-indexer wazuh-indexer     6 Oct  5 16:39 jvm.options.d
-rw-rw----.   1 wazuh-indexer wazuh-indexer 14808 Oct  5 16:39 log4j2.properties
-rw-rw----.   1 wazuh-indexer wazuh-indexer   196 Oct 17 14:30 opensearch.keystore
-rw-------.   1 wazuh-indexer wazuh-indexer    73 Oct 17 14:30 .opensearch.keystore.initial_md5sum
drwxr-x---.   2 wazuh-indexer wazuh-indexer    31 Oct 17 14:30 opensearch-notifications
drwxr-x---.   2 wazuh-indexer wazuh-indexer    36 Oct 17 14:30 opensearch-notifications-core
drwxr-x---.   2 wazuh-indexer wazuh-indexer    31 Oct 17 14:30 opensearch-observability
drwxr-x---.   2 wazuh-indexer wazuh-indexer   263 Oct 17 14:30 opensearch-performance-analyzer
drwxr-x---.   2 wazuh-indexer wazuh-indexer    35 Oct 17 14:30 opensearch-reports-scheduler
drwxr-x---.   2 wazuh-indexer wazuh-indexer   245 Oct 17 14:30 opensearch-security
-rw-rw----.   1 wazuh-indexer wazuh-indexer  2188 Oct 17 14:30 opensearch.yml


Can you send us the output for the following command?
systemctl status wazuh-indexer

Can you send us the log indexer from the following path?
You can pick it from here(by default): /var/log/wazuh-indexer/wazuh-cluster.log

Can you send us the configuration indexer from the following path?
You can pick it from here: etc/wazuh index/opensearch.yml

Regards,
Félix.

[Roye R]

We're going to replace Wazuh with something that doesn't break for no reason.

Thanks for trying to help.

[Felix Bocco]

Hi Roye,

Sorry that things are not working out. We would like to help you with the issue you are having but without further information is hard for us to give a possible workaround and/or understand what is going on. Our target is to help you with this.If you are still interested in getting a solution for this, please send us the following:

The output for the following command:
systemctl status wazuh-indexerThe log indexer from the following path:
You can pick it from here(by default): /var/log/wazuh-indexer/wazuh-cluster.logThe configuration indexer from the following path:

You can pick it from here: etc/wazuh index/opensearch.ymlAlso have you tried checking that there is no file/directory which could need execution permissions and that the wazuh-indexer group could be needing read-write permissions as asked in the previous reply?Lastly, which curl command have you run?

Please try to give it a try to this and if it doesn't work, we can understand your choice.

Expecting to hear from you!

[Danish Ibrar]

  Everything was working fine, Had to restart indexer but its not starting I get this error while trying to restart the indexer Service 

journalctl -xe

Nov 03 13:17:13 tpl-siem systemd[1]: Starting Wazuh-indexer...
-- Subject: A start job for unit wazuh-indexer.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit wazuh-indexer.service has begun execution.
--
-- The job identifier is 378769.
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: add missing path
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: add missing path
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: failed to get udev uid: Invalid argument
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: failed to get sysfs uid: Invalid argument
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: failed to get sgio uid: No such file or directory
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: add missing path
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: add missing path
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: failed to get udev uid: Invalid argument
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: failed to get sysfs uid: Invalid argument
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: failed to get sgio uid: No such file or directory
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs/indexer-key.pem
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs/indexer-key.pem
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.Files.readAttributes(Files.java:1843)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.Files.walkFileTree(Files.java:2840)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:232)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:142)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)

root@tpl-siem:/# du -h --max-depth=1 /var/lib/wazuh-indexer/
2.1T    /var/lib/wazuh-indexer/nodes
2.1T    /var/lib/wazuh-indexer/

root@tpl-siem:/# df -kh
Filesystem                         Size  Used Avail Use% Mounted on
udev                                36G     0   36G   0% /dev
tmpfs                              7.1G  1.9M  7.1G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  3.0T  2.3T  604G  80% /
tmpfs                               36G  3.6G   32G  11% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                               36G     0   36G   0% /sys/fs/cgroup
/dev/loop3                          92M   92M     0 100% /snap/lxd/24061
/dev/loop2                          92M   92M     0 100% /snap/lxd/23991
/dev/sda2                          1.5G  207M  1.2G  15% /boot
tmpfs                              7.1G     0  7.1G   0% /run/user/1000
/dev/sdb                           2.0T  193G  1.7T  11% /home/siem/backup-logs
/dev/loop6                          64M   64M     0 100% /snap/core20/1974
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/28dc8dacb7c4772a0f810be00c30607ae9b9deee5e20ecee1ac043881d48a0cc/merged
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/da986ba7a966f16232f6227092182171d17e8deb3f303c022d1f4bd84684f11d/merged
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/9a421d9b01fad627d4edacb1816fe896eb0780e25eb55ddf3901ad3df36e0f1b/merged
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/4adac82c85145343d69f7088a80cd549af5d3a56b4f85c18b8e6a3f04be9fa76/merged

[Danish Ibrar]

Can anyone help?

[RusFM]

Just started getting this as well.  I have not figured it out yet.