Wazuh dashboard server is not ready yet

[Nour El-Din Ahmed]

Hello Wazuh community,
I’m running an all‑in‑one Wazuh 4.11 deployment (Manager, OpenSearch Indexer, and Dashboard on a single node) on an HP Workstation Z840 with:
Dual Intel® Xeon E5‑2680 v4 processors
14 cores / 28 threads each → 28 cores & 56 threads total
35 MB L3 cache each → 70 MB total
Ample RAM (configured at 128 GB)
Fast SSD storage for both /var/ossec and /var/lib/wazuh-indexer
I have 27 standard agents and 1 serverless agent reporting in. During our business hours, when these agents are actively sending data, the Dashboard hangs—API calls consistently time out, saved‑object migrations fail with “all shards failed,” and I see errors like:
vbnetCopyEditERROR: Timeout executing API request  
[search_phase_execution_exception]: all shards failed on .kibana index  
cluster-manager not discovered or elected yet  
(1404): Authentication error. Wrong key or corrupt payload. Message received from agent ‘007’  

Yet, after hours, when agents go offline, a full restart of all services (Indexer → Manager → Dashboard) immediately restores functionality—even though agents reconnect right away.
What I’ve already verified:
Hardware: Dual 28‑core Xeons, 128 GB RAM, SSDs—CPU, memory, and disk are never saturated under load.
Disk usage: / is only 44 % full (98 GB total), indexer data only ~1.6 GB.
Disk I/O: iostat and iotop show no sustained high %util or long await.
OpenSearch health: Cluster briefly goes yellow/red under peak load.
My questions:
Given this beefy hardware, are there configuration best practices (heap sizing, shard counts, refresh intervals) you’d recommend for an all‑in‑one on a high‑core, high‑memory server? Or best practices for when it’s time to split services onto separate nodes, despite the relatively small agent count?
Why does the Dashboard produce those specific errors (timeouts on /agents calls, all shards failed, master‑election warnings, corrupt payload/authentication errors) under load—and what component or configuration misstep typically triggers each of those messages?
Any advice—log paths to watch, specific settings to tweak, or monitoring hooks—would be greatly appreciated. Thanks in advance for your insights!

[hasitha.u...@wazuh.com]

Hi Nour, Given your hardware (28 cores, 56 threads, 128 GB RAM, SSDs), the all-in-one setup should handle 28 agents comfortably, but the default Wazuh/OpenSearch configurations may not be optimized for your high-core, high-memory environment. Here are tailored best practices: Check the heap size and tune accordingly. Update the heap size in the /etc/wazuh-indexer/jvm.options file. 
-Xms24g
-Xmx24g Here are some key points to keep in mind to configure heap memory:

• Use no more than 50% of your available RAM.
• Don’t set the heap size over 32 GB.

After making these changes, restart the Wazuh indexer for them to take effect:
systemctl restart wazuh-indexer
Ref: https://documentation.wazuh.com/current/user-manual/wazuh-indexer/wazuh-indexer-tuning.html

If the indexer up and running please share the cluster health.
curl -XGET -k -u admin:pass "https://<WAZUH_INDEXER_IP>::9200/_cluster/health"
Replace pass with the admin password.

The "Wazuh dashboard server is not ready yet" error normally indicates that the Wazuh dashboard cannot communicate with the indexer.
We need to make sure that your Wazuh-indexer services are up and running. Restart the indexer service and check the service again
systemctl restart wazuh-indexer
systemctl status wazuh-indexer

Check if you have the right indexer IP/address and Dashboard certs in the Dashboard configuration file.
/etc/wazuh-dashboard/opensearch_dashboards.yml

Check that the Wazuh Indexer IP is updated in the following section
opensearch.hosts: https://<Wazuh-IndexerIP>:9200

If you are not sure about your right indexer address, run this command to find the indexer IP
head /etc/wazuh-indexer/opensearch.yml

Run this command to check certificate names. Ensure the paths and filenames match in the configuration
ls -lrt /etc/wazuh-dashboard/certs/

Make sure you have updated the file permissions for the certificates
chmod 500 /etc/wazuh-dashboard/certs
chmod 400 /etc/wazuh-dashboard/certs/*
chown -R wazuh-dashboard:wazuh-dashboard /etc/wazuh-dashboard/certs

Can you run these commands to verify if your component can communicate using those certificates?
curl -u kibanaserver:<kibanaserver_password> --cacert /etc/wazuh-dashboard/certs/root-ca.pem --cert /etc/wazuh-dashboard/certs/wazuh-dashboard.pem --key /etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem -X GET "https://127.0.0.1:9200/_cluster/health?pretty"

Can you search your server's file directory if you have this file?
wazuh-install-files.tar

If you find this file, then run this command to get the Kibana server user’s password
sudo tar -O -xvf wazuh-install-files.tar wazuh-install-files/wazuh-passwords.txt

If you cannot find the file and you do not remember the kibanaserver user password.
Run this command to change the password:
/usr/share/wazuh-indexer/plugins/opensearch-security/tools/wazuh-passwords-tool.sh -u kibanaserver -p '<new_password>'

Make sure the password is between 8 and 64 characters, and includes upper/lowercase letters, numbers, and a symbol (.*+?-).
As you are using AIO, it automatically updates the passwords in the Wazuh dashboard node.
Update the kibanaserver password in the keystore:
echo <KIBANASERVER_PASSWORD> | /usr/share/wazuh-dashboard/bin/opensearch-dashboards-keystore --allow-root add -f --stdin opensearch.password

Restart the indexer service
sudo systemctl restart wazuh-indexer
Restart the dashboard service
sudo systemctl restart wazuh-dashboard

Run this command to verify if the issue is with the dashboard user authentication.
curl -XGET -k -u kibanaserver:<password> "https://<Indexer_IP>:9200/_cluster/health"

The message "Wazuh dashboard server is not ready yet" can also appear when you have just started or restarted the Wazuh dashboard.
Wait for a few minutes and try to access the dashboard UI again.

Wazuh Manager:

• /var/ossec/logs/ossec.log: General Manager errors, including authentication issues.
• /var/ossec/logs/api.log: API logs
• /var/ossec/logs/alerts/alerts.json: Verify alerts are being generated.

Wazuh Indexer:

• /var/log/wazuh-indexer/<cluster-name>.log: Cluster error|warnings, shard failures, etc..

Wazuh dashboard:

• journalctl -u wazuh-dashboard | grep -i -E "error|warn"

For further troubleshooting, you can refer to this.
https://documentation.wazuh.com/current/user-manual/wazuh-dashboard/troubleshooting.html

Let me know the update on this.

Regards,
Hasitha Upekshitha

[Nour El-Din Ahmed]

systemctl status wazuh-dashboard, right after restarting but after a few minutes it goes away on its own:
p":"2025-05-19T07:00:24Z","tags":["error","opensearch","data"],"pid":45744,"message":"[search_phase_execution_exception]: all shards failed"}
p":"2025-05-19T07:00:27Z","tags":["error","opensearch","data"],"pid":45744,"message":"[search_phase_execution_exception]: all shards failed"}
p":"2025-05-19T07:00:29Z","tags":["error","opensearch","data"],"pid":45744,"message":"[search_phase_execution_exception]: all shards failed"}

curl -XGET -k -u admin:pass "https://<WAZUH_INDEXER_IP>::9200/_cluster/health"

{
  "cluster_name" : "wazuh-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 47,
  "active_shards" : 47,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

After waiting for a few minutes after restarting the Dashboard:
Error: Error getting the authorization token

AxiosError: timeout of 20000ms exceeded

/etc/wazuh-dashboard/opensearch_dashboards.yml
server.host: 0.0.0.0
opensearch.hosts: https://127.0.0.1:9200

 head /etc/wazuh-indexer/opensearch.yml
network.host: "127.0.0.1"
node.name: "node-1"
cluster.initial_master_nodes:
- "node-1"
cluster.name: "wazuh-cluster"

node.max_local_storage_nodes: "3"
path.data: /var/lib/wazuh-indexer
path.logs: /var/log/wazuh-indexer


ls -lrt /etc/wazuh-dashboard/certs/
total 12
-r-------- 1 wazuh-dashboard wazuh-dashboard 1204 May 15 13:31 root-ca.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1289 May 15 13:31 wazuh-dashboard.pem
-r-------- 1 wazuh-dashboard wazuh-dashboard 1704 May 15 13:31 wazuh-dashboard-key.pem

speedtest
[2025-05-19 10:07:56.993] [error] Configuration - Timeout was reached (TimeoutException)
[2025-05-19 10:07:56.993] [error] Configuration - Cannot retrieve configuration document (0)
[2025-05-19 10:07:57.002] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[2025-05-19 10:07:57.002] [error] ConfigurationError - Could not retrieve or read configuration (Configuration)
[error] Configuration - Could not retrieve or read configuration (ConfigurationError)

 curl -u kibanaserver:NNN9zwoE4pbH4d?VwCLqwP3tcUca2jVr --cacert /etc/wazuh-dashboard/certs/root-ca.pem --cert /etc/wazuh-dashboard/certs/wazuh-dashboard.pem --key /etc/wazuh-dashboard/certs/wazuh-dashboard-key.pem -X GET "https://127.0.0.1:9200/_cluster/health?pretty"
{
  "cluster_name" : "wazuh-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 1,
  "number_of_data_nodes" : 1,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 47,
  "active_shards" : 47,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0

[Nour El-Din Ahmed]

I changed the heap size to  -Xms24g and restarted the indexer.
The error I'm getting now:
INFO: Current API id [default]
INFO: Checking current API id [default]...
INFO: Current API id [default] has some problem: timeout of 20000ms exceeded
INFO: Getting API hosts...
INFO: API hosts found: 1
INFO: Checking API host id [default]...
INFO: Could not connect to API id [default]: 3099 - ERROR3099 - Timeout executing API request
INFO: Removed [navigate] cookie
ERROR: No API available to connect

[Nour El-Din Ahmed]

Speed test:
Idle Latency:    12.09 ms   (jitter: 0.71ms, low: 11.57ms, high: 13.31ms)
    Download:     1.81 Mbps (data used: 1.5 MB)
                510.46 ms   (jitter: 77.98ms, low: 139.01ms, high: 1255.53ms)
      Upload:     1.85 Mbps (data used: 2.2 MB)
               1395.49 ms   (jitter: 101.06ms, low: 79.17ms, high: 2536.22ms)
 Packet Loss:     2.3%

[Nour El-Din Ahmed]

Does the dashboard get affected by internet speed?
Could it be the reason for the APIs errors that appear when logging in?
timeout of 20000ms exceeded

[Eswar Eswar]

Hi 

Im gtting  Wazuh dashboard server is not ready yet error in my wazuh currently im using 4.7.5 version.

kindly help to resolve 

Please let me for for the required details. 

Regards,

Eswar  

[hasitha.u...@wazuh.com]

Hi  Nour,

This issue typically occurs when the Wazuh Manager experiences high load or lacks sufficient resources to process API requests from the Wazuh Dashboard.

To help us investigate further, could you please provide the following details?

1. The total number of agents in your environment.
2. The number of Wazuh Managers you have.
3. Information about any custom integrations you’ve implemented.

Additionally, try calling the API again from the Wazuh Dashboard and run the following commands on the Wazuh Manager. Share the outputs with us:

htop 

free -h 

cat /var/ossec/logs/ossec.log | grep -iE "error|warn|crit|fatal"

If the issue is related to high load, consider adding more Wazuh nodes to your environment. This can help distribute the load across multiple managers, improving performance and reliability.

According to your network details:
The download and upload delays are very high — over 1 second just to send or receive data. This usually means the network is slow or unstable, possibly due to congestion or a weak internet connection.  
Some data packets are getting lost on the way. Even a small amount of loss can cause delays because the system has to re-send data, and this can slow down communication between agents and the Wazuh manager.  

Make sure to have stable network connectivity across your Wazuh deployment as well as with the agents.

Let me know if you have any questions or need further assistance.

Regards,
Hasitha Upekshitha

[Nour El-Din Ahmed]

27 Agents and 1 Firewall.
htop:
(Attached Below)

free -h:
               total        used        free      shared  buff/cache   available
Mem:            62Gi        28Gi        27Gi       4.1Mi       7.2Gi        34Gi
Swap:           63Gi          0B        63Gi

cat /var/ossec/logs/ossec.log | grep -iE "error|warn|crit|fatal"

2025/05/25 11:19:09 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:11 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:11 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:11 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:11 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:17 wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database.
2025/05/25 11:19:17 wazuh-analysisd: WARNING: Mitre Technique ID 'T1531' not found in database.
2025/05/25 11:19:17 wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database.
2025/05/25 11:19:17 wazuh-analysisd: WARNING: Mitre Technique ID 'T1531' not found in database.
2025/05/25 11:19:19 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:19 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:19 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:19 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:19 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:19 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:21 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:21 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:21 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:21 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:27 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:27 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:29 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:29 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:29 wazuh-analysisd: WARNING: Mitre Technique ID 'T1110.001' not found in database.
2025/05/25 11:19:29 wazuh-analysisd: WARNING: Mitre Technique ID 'T1021.004' not found in database.
2025/05/25 11:19:56 wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database.
2025/05/25 11:19:56 wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database.

[hasitha.u...@wazuh.com]

Hi  Nour,

2025/05/25 11:19:56 wazuh-analysisd: WARNING: Mitre Technique ID 'T1078' not found in database.

I’ve seen a similar issue reported by another user, which turned out to be related to the host machine where the deployment was done. The user resolved it by testing the setup on a local machine instead: https://groups.google.com/g/wazuh/c/vwYih_7oIUk

Try restarting the Wazuh manager and dashboard.
systemctl restart wazuh-manager
systemctl restart wazuh-dashboard

Additionally, please share the API and full OSSEC log file to check further.
cat /var/ossec/logs/ossec.log
cat /var/ossec/logs/api.log

Additionally, try perform the steps below to modify the API connection timeout and Wazuh dashboard session timeout
API connection timeout
Step 1: Log into the Wazuh dashboard as an admin user.
Step 2: Navigate to the App setting under Dashboard management in the Wazuh dashboard. See the image below.
Step 3: Under the General section, find Request timeout. Change the value based on your need. See the image below.
Step 4: Save the changes.


Also, make sure to provide stable network connectivity for the server.

Let me know if you need further assistance on this issue.

Regards,
Hasitha Upekshitha

[Nour El-Din Ahmed]

Thanks again for guiding me through these performance checks. Below is the information and logs you asked for, along with a few follow‑up questions.

---

1. Environment

• Deployment type: All‑in‑one (manager, indexer, dashboard)

• Agents: 27 managed endpoints

• Firewall: 1 device forwarding logs via syslog

• JVM heap (Wazuh Indexer): -Xms24g -Xmx24g on 28 cores / 56 threads, 128 GB RAM

  2. Logs  

• cat /var/ossec/logs/ossec.log
  2025/05/27 15:21:06 wazuh-remoted: ERROR: TCP peer [45]: No route to host (113)
  2025/05/27 15:21:18 wazuh-remoted: ERROR: TCP peer [38]: No route to host (113)
  2025/05/27 15:21:19 wazuh-remoted: ERROR: TCP peer [29]: No route to host (113)
  2025/05/27 15:25:08 wazuh-modulesd:syscollector: INFO: Starting evaluation.
  2025/05/27 15:25:34 wazuh-modulesd:syscollector: INFO: Evaluation finished.
  2025/05/27 15:54:27 wazuh-db: INFO: Created Global database backup "backup/db/global.db-backup-2025-05-27-15:54:27.gz"
  2025/05/27 15:54:27 wazuh-db: INFO: Deleted Global database backup: "backup/db/global.db-backup-2025-05-24-15:54:26.gz"
  2025/05/27 19:26:53 wazuh-modulesd:syscollector: INFO: Starting evaluation.
  2025/05/27 19:27:26 wazuh-modulesd:syscollector: INFO: Evaluation finished.
  2025/05/27 19:27:39 wazuh-modulesd:vulnerability-scanner: INFO: Initiating update feed process.
  2025/05/27 19:27:42 wazuh-modulesd:vulnerability-scanner: INFO: Triggered a re-scan after content update.
  2025/05/27 19:27:42 wazuh-modulesd:vulnerability-scanner: INFO: Feed update process completed.
  2025/05/27 20:27:26 wazuh-modulesd:syscollector: INFO: Starting evaluation.
  2025/05/27 20:27:57 wazuh-modulesd:syscollector: INFO: Evaluation finished.
  2025/05/27 20:41:36 wazuh-analysisd: WARNING: Mitre Technique ID 'T1562.001' not found in database.
  2025/05/27 20:41:36 wazuh-analysisd: WARNING: Mitre Technique ID 'T1562.001' not found in database.
  2025/05/27 21:27:58 wazuh-modulesd:syscollector: INFO: Starting evaluation.
  2025/05/27 21:28:25 wazuh-modulesd:syscollector: INFO: Evaluation finished.
  
• cat /var/ossec/logs/api.log -> Attached below
  

  3. Follow‑up Questions  

• For an all‑in‑one Wazuh setup with 27 managed agents plus one firewall forwarding logs, what minimum total network bandwidth (in Mbps) would you recommend to prevent agent‑manager packet drops or timeouts during regular Syscollector scans, vulnerability‑feed updates, and event‑spike bursts? Additionally, how significantly does overall internet speed impact Wazuh’s performance under those conditions?
• For the Wazuh API’s request_timeout, what value would you set for a 28‑endpoint all‑in‑one under moderate load?  
• In the Dashboard App Settings → General → Request timeout (default 20 000 ms), what would you recommend as a safe upper limit?  
• With 28 cores, 128 GB RAM, SSDs, and a 24 GB JVM heap, are there any extra best‑practice settings you’d advise to smooth out peaks?  

  Thank you for taking the time to review these logs and for any guidance you can provide. I really appreciate your expertise and help!  

[hasitha.u...@wazuh.com]

Hi Nour

The error message wazuh-remoted: ERROR: TCP peer [29]: No route to host (113) from Wazuh indicates that the Wazuh server (specifically the wazuh-remoted component) is unable to establish a TCP connection to a specified host, likely an agent or another network device, due to a "No route to host" error (error code 113). This typically points to a network connectivity issue.

So, at this point, I believe you are not seeing the Dashboard not ready yet error. If that is the case, the indexer and dashboard can communicate properly. The API timeout is related to not being able to connect with the manager API within a given time.
Let's check with an API curl request with api credentials if we can successfully query the API.
curl -k -X GET "https://127.0.0.1:55000/agents" -H "Authorization: Bearer $(curl -u wazuh-wui:<PASSWORD> -k -X GET 'https://127.0.0.1:55000/security/user/authenticate?raw=true')"

The default Wazuh API user credentials are located in the file /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml.
Run this command to have the credentials in output
tail -n 15 /usr/share/wazuh-dashboard/data/wazuh/config/wazuh.yml

And Wazuh doesn't need any internet connection. It needs a proper network connection between the components to work properly. Therefore make sure to provide stable network connectivity for the server.

Let me know the update on this.

Regards,
Hasitha Upekshitha