Unable to start wazuh-indexer

[Vaughn Hawk]

Checking filebeat, I get the follwing:

filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... ERROR dial tcp 127.0.0.1:9200: connect: connection refused

Looking at journal -xe

 journalctl -xe
Sep 18 21:25:45 wazuh-server opensearch-dashboards[449]: {"type":"log","@timestamp":"2023-09-18T21:25:45Z","tags":["error","opensearch","data"],"pid":449,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 18 21:25:48 wazuh-server opensearch-dashboards[449]: {"type":"log","@timestamp":"2023-09-18T21:25:48Z","tags":["error","opensearch","data"],"pid":449,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 18 21:25:50 wazuh-server opensearch-dashboards[449]: {"type":"log","@timestamp":"2023-09-18T21:25:50Z","tags":["error","opensearch","data"],"pid":449,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 18 21:25:53 wazuh-server opensearch-dashboards[449]: {"type":"log","@timestamp":"2023-09-18T21:25:53Z","tags":["error","opensearch","data"],"pid":449,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"}
Sep 18 21:25:55 wazuh-server opensearch-dashboards[449]: {"type":"log","@timestamp":"2023-09-18T21:25:55Z","tags":["error","opensearch","data"],"pid":449,"message":"[ConnectionError]: connect ECONNREFUSED 127.0.0.1:9200"

And if I look in the cluster logs, I get a bunch of these errors:

[2023-09-18T00:01:54,535][INFO ][o.o.j.s.JobSweeper       ] [node-1] Running full sweep
[2023-09-18T00:01:55,569][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.5gb[4.3%], all indices on this node will be marked read-only
[2023-09-18T00:02:25,577][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.5gb[4.3%], all indices on this node will be marked read-only
[2023-09-18T00:02:55,586][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.4gb[4.2%], all indices on this node will be marked read-only
[2023-09-18T00:03:25,593][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.3gb[4.2%], all indices on this node will be marked read-only
[2023-09-18T00:03:55,599][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.3gb[4.2%], all indices on this node will be marked read-only
[2023-09-18T00:04:25,608][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.2gb[4.2%], all indices on this node will be marked read-only
[2023-09-18T00:04:55,614][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21.1gb[4.2%], all indices on this node will be marked read-only
[2023-09-18T00:05:25,622][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21gb[4.2%], all indices on this node will be marked read-only
[2023-09-18T00:05:55,627][WARN ][o.o.c.r.a.DiskThresholdMonitor] [node-1] flood stage disk watermark [95%] exceeded on [Rc_D7aKbS3qQZ9_KGRU7PQ][node-1][/var/lib/wazuh-indexer/nodes/0] free: 21gb[4.2%], all indices on this node will be marked read-only

Looking in that directory, everything is from today, so I have no clue what I can delete, or if this will help at all:

ncdu /var/lib/wazuh-indexer/nodes/0

ncdu 1.18 ~ Use the arrow keys to navigate, press ? for help
--- /var/lib/wazuh-indexer/nodes/0 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  385.9 GiB [#######################################] /indices
  944.0 KiB [                                       ] /_state
    0.0   B [                                       ]  node.lock

So looking in indices, as I mentioned, everything is from today:

drwxr-xr-x.   6 wazuh-indexer wazuh-indexer   47 Sep 18 19:32 KhKMpzEDRzmrynVtpWCNhA
drwxr-xr-x.   6 wazuh-indexer wazuh-indexer   47 Sep 18 19:32 kSYxoyy_Q_yjsInjKEyBtQ
drwxr-xr-x.   4 wazuh-indexer wazuh-indexer   29 Sep 18 19:32 dvOJ5XDATMaLmHRbRqaSDw
drwxr-xr-x.   4 wazuh-indexer wazuh-indexer   29 Sep 18 19:32 dfv0kggqQ1GCrszwDbck9w
drwxr-xr-x.   4 wazuh-indexer wazuh-indexer   29 Sep 18 19:32 WulL98N3T2SyvQ3C-vMkDQ
drwxr-xr-x.   6 wazuh-indexer wazuh-indexer   47 Sep 18 19:32 fOVO0HtqSyayavjRvf7iBg
drwxr-xr-x.   4 wazuh-indexer wazuh-indexer   29 Sep 18 19:32 z1h755jOQA2oPBHpiYcjSg
drwxr-xr-x.   4 wazuh-indexer wazuh-indexer   29 Sep 18 19:32 bl7jywZ0ScW7eoMDq9vfrQ
drwxr-xr-x.   6 wazuh-indexer wazuh-indexer   47 Sep 18 19:32 pEYzQRh9Q3SuMYEKpI22fA

Any advice would be helpful, we're pretty new at using wazuh and I can't find anything that has solved this issue.

Thanks,

Vaughn Hawk

[suricata]

Hello,

I solved that problem recently by doing the following:

#nano /etc/wazuh-indexer/jvm.options

(In my case 4g)

#systemctl daemon-reload

#systemctl restart wazuh-indexer

#systemctl restart wazuh-dashoboard

Regards,

[Wazuh | Mailing List]

Hi,

yes, as suricata mentioned, sometimes you need to validate if resources requirements are being met (even if at the begining the cluster was working, the app needs can change over time).

Another points to validate on this can be found here: https://groups.google.com/g/wazuh/c/fOGa1WHb2Kk

[suricata]

Thanks ;-)

[Vaughn Hawk]

So I did change the values you recommended in the /etc/wazuh-indexer/jvm.options config file, that made no difference, wazuh-indexer will still not start.

filebeat test output still shows:

elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... ERROR dial tcp 127.0.0.1:9200: connect: connection refused

journalctl -xe shows:

Sep 20 17:23:01 wazuh-server dbus[443]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.servSep 20 17:23:01 wazuh-server dbus[443]: [system] Successfully activated service 'org.freedesktop.hostname1'
Sep 20 17:26:03 wazuh-server polkitd[469]: Registered Authentication Agent for unix-process:780:8320348 (system bus name :1.137 [/usr/bin/pkttyagent --notiSep 20 17:26:03 wazuh-server polkitd[469]: Unregistered Authentication Agent for unix-process:780:8320348 (system bus name :1.137, object path /org/freedesSep 20 17:26:12 wazuh-server polkitd[469]: Registered Authentication Agent for unix-process:805:8321214 (system bus name :1.138 [/usr/bin/pkttyagent --notiSep 20 17:26:15 wazuh-server systemd-entrypoint[813]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 20 17:26:15 wazuh-server systemd-entrypoint[813]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usrSep 20 17:26:15 wazuh-server systemd-entrypoint[813]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 20 17:26:15 wazuh-server systemd-entrypoint[813]: WARNING: System::setSecurityManager will be removed in a future release
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: Exception in thread "main" org.opensearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedExcSep 20 17:26:17 wazuh-server systemd-entrypoint[813]: Likely root cause: java.nio.file.AccessDeniedException: /etc/wazuh-indexer/backup/action_groups.yml
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/java.nio.file.Files.readAttributes(Files.java:1851)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:226)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/java.nio.file.FileTreeWalker.next(FileTreeWalker.java:374)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at java.base/java.nio.file.Files.walkFileTree(Files.java:2845)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:237)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.common.logging.LogConfigurator.configure(LogConfigurator.java:147)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:373)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.cli.Command.main(Command.java:101)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Sep 20 17:26:17 wazuh-server systemd-entrypoint[813]: For complete error details, refer to the log at /var/log/wazuh-indexer/wazuh-cluster.log
Sep 20 17:26:18 wazuh-server polkitd[469]: Unregistered Authentication Agent for unix-process:805:8321214 (system bus name :1.138, object path /org/freedesSep 20 17:30:03 wazuh-server sudo[2007]: madisons : TTY=pts/1 ; PWD=/home/madisons ; USER=root ; COMMAND=/bin/bash
Sep 20 17:30:03 wazuh-server sudo[2007]: pam_unix(sudo-i:session): session opened for user root by madisons(uid=0)

None of this seems particularly useful other than "AccessDenied", so maybe it's a password expired thing? I've tried resetting the admin password but that made no difference; when I try to reset ALL passwords using the instructions for the wazuh-passwords-tool.sh tool, and that fails, every time.

[Mario Andres Ruiz Hernandez]

It's not like there is no difference with the inicial error you reported, the log error is very different now :) So we have some kind of "advance". I'll ask internal wazuh team to see what we can do on this. Stay tuned.

[Vaughn Hawk]

Thank you :)

[Mario Andres Ruiz Hernandez]

Still asking internally.

Can you try some of the troubleshooting process described in the official documentation so so we can see how it goes?

https://documentation.wazuh.com/current/user-manual/elasticsearch/troubleshooting.html 

[Vaughn Hawk]

Hey Mario,

So I think the main crux of the issue is filebeat:

filebeat test output
elasticsearch: https://127.0.0.1:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 127.0.0.1
    dial up... ERROR dial tcp 127.0.0.1:9200: connect: connection refused

I'm not sure what is causing that, but that's the main issue why everything isn't working. I've tried going through the troubleshooting guides provided by Wazuh and still having this specific problem.

[Mario Andres Ruiz Hernandez]

Hey Vaughn, where you able to solve?