Wazuh-dashboard service won't start, keeps failing

[Russell Anderson]

I had an issue trying to log into my wazuh web interface. 
Told me my username and password were wrong.
Rebooted the server.
updated the server. since it told me there were updates.

now i can't access the dashboard., I tried to restart the service, but it would fail.

Any assistance would be greatly appreciated.

× wazuh-dashboard.service - wazuh-dashboard
     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Wed 2024-12-11 21:19:54 UTC; 2min 43s ago
   Duration: 9.050s
    Process: 3592 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (code=exited, status=1/FAILURE)
   Main PID: 3592 (code=exited, status=1/FAILURE)
        CPU: 10.532s

Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["info","savedobjects-service"],"pid":3592,"message":"Waiting until all OpenSearch nodes are compatible with OpenSearch Dashboards before starting saved objects migrations..."}
Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["info","savedobjects-service"],"pid":3592,"message":"Starting saved objects migrations"}
Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["error","opensearch","data"],"pid":3592,"message":"[circuit_breaking_exception]: [parent] Data too large, data for [<http_request>] would be [1050092680/1001.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1050092680/1001.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]"}
Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["warning","savedobjects-service"],"pid":3592,"message":"Unable to connect to OpenSearch. Error: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1050092680/1001.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1050092680/1001.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]"}
Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["fatal","root"],"pid":3592,"message":"ResponseError: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [<http_request>] would be [1050092680/1001.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1050092680/1001.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:529:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1400:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21) {\n  meta: {\n    body: { error: [Object], status: 429 },\n    statusCode: 429,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '817'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'opensearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}
Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["info","plugins-system"],"pid":3592,"message":"Stopping all plugins."}
Dec 11 21:19:54 wazuh opensearch-dashboards[3592]:  FATAL  {"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1050092680/1001.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1050092680/1001.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]","bytes_wanted":1050092680,"bytes_limit":1020054732,"durability":"TRANSIENT"}],"type":"circuit_breaking_exception","reason":"[parent] Data too large, data for [<http_request>] would be [1050092680/1001.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1050092680/1001.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]","bytes_wanted":1050092680,"bytes_limit":1020054732,"durability":"TRANSIENT"},"status":429}
Dec 11 21:19:54 wazuh systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE
Dec 11 21:19:54 wazuh systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.
Dec 11 21:19:54 wazuh systemd[1]: wazuh-dashboard.service: Consumed 10.532s CPU time, 185.8M memory peak, 0B memory swap peak.

[Bony V John]

Hi Russell,

Please follow the troubleshooting steps below to resolve the issue:

1. Check Disk Space

Run the following command to verify the available disk space on the Wazuh Indexer and Dashboard servers:

df -h

2. Check Memory Usage

Check the memory consumption of the Indexer and Wazuh Dashboard:

free -h

3. Check Wazuh Dashboard Logs

Run the following command to check for any error entries in the Wazuh Dashboard logs:

journalctl -u wazuh-dashboard | grep -iE "error|warn|crit|fatal" >> output.log

After running this command, an output.log file will be created in the directory where the command is executed.

4. Check Wazuh Indexer Status

Verify the status of the Wazuh Indexer service:

systemctl status wazuh-indexer

5. Optimize Wazuh Indexer Performance

In the Wazuh Indexer, you can configure the memory heap to optimize performance. For details, please refer to the Wazuh official documentation.

6. Share Wazuh Indexer Logs

Run the following command to gather error logs from the Wazuh Indexer for further analysis:

cat /var/log/wazuh-indexer/wazuh-cluster.log | grep -iE "error|warn|crit|fatal" >> indexer.log

After running this command, an indexer.log file will be created in the directory where the command is executed.

7. Password reset

If you forgot or you need to change the Wazuh service passwords, you can follow the Wazuh Password Management document.

Let me know if these steps help or if you need further assistance.

Regards,

[Russell W. Anderson]

Hello Bony V John

Here are the outputs of the commands.

Russell W. Anderson IT Systems Administrator rus...@whitfieldwelding.com	O: 519.737.1814 x.250 C: 226.345.9815 F: 519.737.1813
5425 Roscon Industrial Dr. Oldcastle ON N0R 1L0	2070 Fasan Dr. Oldcastle ON N0R 1L0	2045 Fasan Dr. Oldcastle ON N0R 1L0
The information contained in this e-mail, including all of its attachments, is confidential and proprietary information of Whitfield Welding Inc. and intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify the sender by telephone or e-mail, and destroy this communication and all copies. Thank you.

---

From: 'Bony V John' via Wazuh | Mailing List <wa...@googlegroups.com>
Sent: Thursday, December 12, 2024 1:30 AM
To: Wazuh | Mailing List <wa...@googlegroups.com>
Subject: Re: Wazuh-dashboard service won't start, keeps failing

 

--
You received this message because you are subscribed to a topic in the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/wazuh/oijQmIuaXJA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to wazuh+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/wazuh/a5868c57-876c-4432-8926-dce8388c791en%40googlegroups.com.

[Bony V John]

Hi,

Dec 11 21:19:53 wazuh opensearch-dashboards[3592]: {"type":"log","@timestamp":"2024-12-11T21:19:53Z","tags":["error","opensearch","data"],"pid":3592,"message":"[circuit_breaking_exception]: [parent] Data too large, data for [<http_request>] would be [1050092680/1001.4mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1050092680/1001.4mb], new bytes reserved: [0/0b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=0/0b]"

The message you're seeing indicates that a process or task was terminated because it consumed more memory than allowed, specifically the heap memory. To mitigate this issue, you can manually adjust the heap size settings in the jvm.options file. Setting both the minimum (-Xms) and maximum (-Xmx) heap sizes to the same value is recommended to prevent performance degradation due to dynamic resizing. For example, for a system with 8 GB of RAM, you might set both values to 4 GB (-Xms4g -Xmx4g) It is also advisable to enable memory locking to prevent the
You can change the total heap space via /etc/wazuh-indexer/jvm.options file. Here is an example of an increase from ~ 2GB to 4GB:
# -Xms1931m
# -Xmx1931m
-Xms4g
-Xmx4g
Restart the Wazuh-Indexer using the command: systemctl restart wazuh-indexerJVM from swapping memory to disk, can further exacerbate resource consumption issues. This can be done by adding bootstrap.memory_lock: true to the /etc/wazuh-indexer/opensearch.yml configuration file https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html#memory-lockingI hope this helps you, please let me know if you have any doubts related to this.

Regards,

[Bony V John]

Hi,

I hope the above details were helpful. Please let me know if you have any further questions.  

Regards,