Wazuh-indexer failed to start

[Danish Ibrar]

Everything was working fine, Had to restart indexer but its not starting I get this error while trying to restart the indexer Service 

journalctl -xe

Nov 03 13:17:13 tpl-siem systemd[1]: Starting Wazuh-indexer...
-- Subject: A start job for unit wazuh-indexer.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit wazuh-indexer.service has begun execution.
--
-- The job identifier is 378769.
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: add missing path
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 13:17:16 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: add missing path
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: failed to get udev uid: Invalid argument
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: failed to get sysfs uid: Invalid argument
Nov 03 13:17:18 tpl-siem multipathd[883]: sdb: failed to get sgio uid: No such file or directory
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: add missing path
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 13:17:21 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: add missing path
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: failed to get udev uid: Invalid argument
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: failed to get sysfs uid: Invalid argument
Nov 03 13:17:23 tpl-siem multipathd[883]: sdb: failed to get sgio uid: No such file or directory
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs/indexer-key.pem
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs/indexer-key.pem
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.Files.readAttributes(Files.java:1843)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at java.base/java.nio.file.Files.walkFileTree(Files.java:2840)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:232)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:142)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
Nov 03 13:17:24 tpl-siem systemd-entrypoint[2249031]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)

[Gonzalo Membrillo Solbes]

Hello,

This looks like the partition you are using to store your Indexer data has ran out of space. In order to verify this, would you be able to run the following instructions?

You can check the size of your current Filesystem partitions running following command:

# df -kh

Can you check the path.data parameter in your /etc/wazuh-indexer/opensearch.yml file?

path.data: /var/lib/wazuh-indexer

Then check the usage of the path.data

# du -h --max-depth=1 /var/lib/wazuh-indexer/

You should be sure that partition where path.data resides has more than 15% free space according to how you created the filesystem or the indexer won't be able to create any new indices, causing it to stop working.

If this is the case, you may need to add more storage space to the indexer's partition or delete data manually.

Regards,

Gonzalo

[Danish Ibrar]

root@tpl-siem:/# du -h --max-depth=1 /var/lib/wazuh-indexer/
2.1T    /var/lib/wazuh-indexer/nodes
2.1T    /var/lib/wazuh-indexer/

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/b3b78abb-2091-4010-9632-b3d0322b1910n%40googlegroups.com.

[Danish Ibrar]

root@tpl-siem:/# df -kh
Filesystem                         Size  Used Avail Use% Mounted on
udev                                36G     0   36G   0% /dev
tmpfs                              7.1G  1.9M  7.1G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv  3.0T  2.4T  538G  82% /
tmpfs                               36G  3.6G   32G  11% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                               36G     0   36G   0% /sys/fs/cgroup
/dev/loop3                          92M   92M     0 100% /snap/lxd/24061
/dev/loop2                          92M   92M     0 100% /snap/lxd/23991
/dev/sda2                          1.5G  207M  1.2G  15% /boot
tmpfs                              7.1G     0  7.1G   0% /run/user/1000
/dev/sdb                           2.0T  127G  1.7T   7% /home/siem/backup-logs
/dev/loop6                          64M   64M     0 100% /snap/core20/1974
overlay                            3.0T  2.4T  538G  82% /var/lib/docker/overlay2/28dc8dacb7c4772a0f810be00c30607ae9b9deee5e20ecee1ac043881d48a0cc/merged
overlay                            3.0T  2.4T  538G  82% /var/lib/docker/overlay2/da986ba7a966f16232f6227092182171d17e8deb3f303c022d1f4bd84684f11d/merged
overlay                            3.0T  2.4T  538G  82% /var/lib/docker/overlay2/9a421d9b01fad627d4edacb1816fe896eb0780e25eb55ddf3901ad3df36e0f1b/merged
overlay                            3.0T  2.4T  538G  82% /var/lib/docker/overlay2/4adac82c85145343d69f7088a80cd549af5d3a56b4f85c18b8e6a3f04be9fa76/merged
/dev/loop4                          64M   64M     0 100% /snap/core20/2015
/dev/loop7                          41M   41M     0 100% /snap/snapd/20092
/dev/loop8                          41M   41M     0 100% /snap/snapd/20290

[Danish Ibrar]

This is after clearing some storage.

root@tpl-siem:/# df -kh
Filesystem                         Size  Used Avail Use% Mounted on
udev                                36G     0   36G   0% /dev
tmpfs                              7.1G  1.9M  7.1G   1% /run

/dev/mapper/ubuntu--vg-ubuntu--lv  3.0T  2.3T  604G  80% /

tmpfs                               36G  3.6G   32G  11% /dev/shm
tmpfs                              5.0M     0  5.0M   0% /run/lock
tmpfs                               36G     0   36G   0% /sys/fs/cgroup
/dev/loop3                          92M   92M     0 100% /snap/lxd/24061
/dev/loop2                          92M   92M     0 100% /snap/lxd/23991
/dev/sda2                          1.5G  207M  1.2G  15% /boot
tmpfs                              7.1G     0  7.1G   0% /run/user/1000

/dev/sdb                           2.0T  193G  1.7T  11% /home/siem/backup-logs

/dev/loop6                          64M   64M     0 100% /snap/core20/1974

overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/28dc8dacb7c4772a0f810be00c30607ae9b9deee5e20ecee1ac043881d48a0cc/merged
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/da986ba7a966f16232f6227092182171d17e8deb3f303c022d1f4bd84684f11d/merged
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/9a421d9b01fad627d4edacb1816fe896eb0780e25eb55ddf3901ad3df36e0f1b/merged
overlay                            3.0T  2.3T  604G  80% /var/lib/docker/overlay2/4adac82c85145343d69f7088a80cd549af5d3a56b4f85c18b8e6a3f04be9fa76/merged

[Danish Ibrar]

Still getting this

Nov 03 15:06:21 tpl-siem systemd[1]: Starting Wazuh-indexer...

-- Subject: A start job for unit wazuh-indexer.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit wazuh-indexer.service has begun execution.
--

-- The job identifier is 379423.
Nov 03 15:06:21 tpl-siem multipathd[883]: sda: add missing path
Nov 03 15:06:21 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 15:06:21 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 15:06:21 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 15:06:23 tpl-siem multipathd[883]: sdb: add missing path
Nov 03 15:06:23 tpl-siem multipathd[883]: sdb: failed to get udev uid: Invalid argument
Nov 03 15:06:23 tpl-siem multipathd[883]: sdb: failed to get sysfs uid: Invalid argument
Nov 03 15:06:23 tpl-siem multipathd[883]: sdb: failed to get sgio uid: No such file or directory
Nov 03 15:06:26 tpl-siem multipathd[883]: sda: add missing path
Nov 03 15:06:26 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 15:06:26 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 15:06:26 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 15:06:28 tpl-siem multipathd[883]: sdb: add missing path
Nov 03 15:06:28 tpl-siem multipathd[883]: sdb: failed to get udev uid: Invalid argument
Nov 03 15:06:28 tpl-siem multipathd[883]: sdb: failed to get sysfs uid: Invalid argument
Nov 03 15:06:28 tpl-siem multipathd[883]: sdb: failed to get sgio uid: No such file or directory
Nov 03 15:06:31 tpl-siem multipathd[883]: sda: add missing path
Nov 03 15:06:31 tpl-siem multipathd[883]: sda: failed to get udev uid: Invalid argument
Nov 03 15:06:31 tpl-siem multipathd[883]: sda: failed to get sysfs uid: Invalid argument
Nov 03 15:06:31 tpl-siem multipathd[883]: sda: failed to get sgio uid: No such file or directory
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs/indexer-key.pem
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/certs/indexer-key.pem
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/java.nio.file.Files.readAttributes(Files.java:1843)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:225)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:373)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at java.base/java.nio.file.Files.walkFileTree(Files.java:2840)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:232)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:142)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:169)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:100)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.cli.Command.main(Command.java:101)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:135)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:101)
Nov 03 15:06:32 tpl-siem systemd-entrypoint[2255305]: For complete error details, refer to the log at /var/log/wazuh-indexer/elasticsearch.log

[Danish Ibrar]

Anyone???

[Gonzalo Membrillo Solbes]

Hello again,

I noticed that, additionally, you have an error in the indexer key permissions. Could you check who owns the indexer-key.pem file?
The correct permissions for the certificates should be like this:

chmod 500 /etc/wazuh-indexer/certs

chmod 400 /etc/wazuh-indexer/certs/*

chown -R wazuh-indexer:wazuh-indexer /etc/wazuh-indexer/certs

While you have cleared some space, the directory in which the Wazuh Indexer stores its data is still at 80% capacity which could cause some problems in the future. I'd recommend clearing up some additional space.

Regards,

Gonzalo