Wazuh Dashboard Server Not Ready

[Roye R]

New install on a different server. 4.3.10.

All services started (indexer, manager, filebeat, dashboard).

Only hint of a problem is dashboard status shows "response error" for opensearch-dashboards.  Other test commands return ok.

What do I do to narrow down the cause?

[Norberto Cesar Vicchi]

Hello Roye!
I am glad you are using Wazuh!
Could you please share the content of the dashboard logs? You can obtain them by executing:

journalctl -u wazuh-dashboard #
cat /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log | grep -i -E "error|warn"

Regards,

Norberto

[Roye R]

Here's the end of the journalctl command, all the same entry:

Mar 30 15:14:05 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:05Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:08 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:08Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:10 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:10Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:13 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:13Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:15 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:15Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:18 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:18Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:20 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:20Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}
Mar 30 15:14:23 lbco-monitor opensearch-dashboards[781]: {"type":"log","@timestamp":"2023-03-30T15:14:23Z","tags":["error","opensearch","data"],"pid":781,"message":"[ResponseError]: Response Error"}

Also, that log file doesn't exist:

cat: /usr/share/wazuh-dashboard/data/wazuh/logs/wazuhapp.log: No such file or directory
heliogenit@lbco-monitor:~$ sudo ls -hal /usr/share/wazuh-dashboard/data/wazuh/logs
total 8.0K
drwxr-xr-x 2 wazuh-dashboard wazuh-dashboard 4.0K Mar 28 20:39 .
drwxr-xr-x 5 wazuh-dashboard wazuh-dashboard 4.0K Mar 28 20:39 ..
heliogenit@lbco-monitor:~$

[Norberto Cesar Vicchi]

I am afraid it doesn't give relevant information.

The message "Wazuh-Dashboard server is not ready yet" can be produced for one of the following reasons:
   * Your service or Wazuh-dashboard has some error that causes it to constantly reboot.
   * Your Indexer service is not up or has some error

Did you attempt to restart all exisiting components? You can do so with:

  systemctl restart wazuh-indexer
  systemctl restart wazuh-manager
  systemctl restart wazuh-dashboard

In a situation where the wazuh-indexer  is not active and you are unable to restart it please note that Wazuh-indexer might give timeout when starting the service. This can happen if the service does not get to start in the time linux is expecting it to happen, we can extend this time with the following steps:

Execute systemctl edit wazuh-indexer and add the following lines

[Service]
TimeoutStartSec=180

Save the file and execute systemctl daemon-reload and then systemctl restart wazuh-indexer
This appears to be the case from the error output provided.
After extending the timeout period the service should start without problems.

Regards,

Norberto

[Roye R]

Tried restarting many times.  The services are all running and the indexer isn't restarting as the PID hasn't changed much unless I manually restart the service.

I made the change you suggested, to no effect.  Also when I start the services, the only one showing any type of errors is the dashboard service.  The message I posted originally is the only errors I see.  The wazuh-manager shows no errors.  The indexer shows a warning but I get that warning on my other instance of Wazuh as well and its running fine.

● wazuh-indexer.service - Wazuh-indexer
     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/wazuh-indexer.service.d
             └─override.conf
     Active: active (running) since Fri 2023-03-31 20:54:13 UTC; 10s ago
       Docs: https://documentation.wazuh.com
   Main PID: 180109 (java)
      Tasks: 76 (limit: 6985)
     Memory: 3.3G
     CGroup: /system.slice/wazuh-indexer.service
             └─180109 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.hea>

Mar 31 20:53:46 lbco-monitor systemd[1]: Starting Wazuh-indexer...
Mar 31 20:54:10 lbco-monitor systemd-entrypoint[180109]: WARNING: An illegal reflective access operation has occurred
Mar 31 20:54:10 lbco-monitor systemd-entrypoint[180109]: WARNING: Illegal reflective access by io.protostuff.runtime.PolymorphicThrowableSchema (file:/usr/share/wazuh-indexer/plugins/opensearch-anomaly-det>
Mar 31 20:54:10 lbco-monitor systemd-entrypoint[180109]: WARNING: Please consider reporting this to the maintainers of io.protostuff.runtime.PolymorphicThrowableSchema
Mar 31 20:54:10 lbco-monitor systemd-entrypoint[180109]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Mar 31 20:54:10 lbco-monitor systemd-entrypoint[180109]: WARNING: All illegal access operations will be denied in a future release
Mar 31 20:54:13 lbco-monitor systemd[1]: Started Wazuh-indexer.

Thanks for the assistance,

R.

[Norberto Cesar Vicchi]

The issue could be related to the max_map_count value. Could you please execute the following command and restart the Wazuh Indexer service?
sysctl -w vm.max_map_count=262144 systemctl restart wazuh-indexer

Please let me know if that worked for you. 

If it doesn't,  it would be really helpful if you could share with me the complete output of the cat /var/log/wazuh-indexer/<name_of_your_cluster>.log command to further analyze the issue.   

Thanks!

[Roye R]

Here's the log file.

[Roye R]

No other thoughts on this issue?