Unable to start Dashboard service

[Suman Banerjee]

Yesterday I have updated Wazuh all in one installation from 4.3 to 4.5.

My installation is on CentOS7 64bit.

During updating no error message shown.

After finishing those wazuh central components, unable to start web consol.

Then I found, Wazuh Dashboard not started.

After executing systemctl start wazuh-dashboard, it is starting and showing Actibe but within few seconds it fails.

Error details:

[root@wazuh ~]# systemctl status wazuh-dashboard -l
● wazuh-dashboard.service - wazuh-dashboard
   Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Thu 2023-10-26 10:51:39 IST; 6min ago
  Process: 19483 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards -c /etc/wazuh-dashboard/opensearch_dashboards.yml (code=exited, status=1/FAILURE)
 Main PID: 19483 (code=exited, status=1/FAILURE)

Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: {"type":"log","@timestamp":"2023-10-26T05:21:39Z","tags":["info","savedobjects-service"],"pid":19483,"message":"Detected mapping change in \"properties.visualization-visbuilder\""}
Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: {"type":"log","@timestamp":"2023-10-26T05:21:39Z","tags":["info","savedobjects-service"],"pid":19483,"message":"Creating index .kibana_2."}
Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: {"type":"log","@timestamp":"2023-10-26T05:21:39Z","tags":["error","opensearch","data"],"pid":19483,"message":"[validation_exception]: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}
Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: {"type":"log","@timestamp":"2023-10-26T05:21:39Z","tags":["warning","savedobjects-service"],"pid":19483,"message":"Unable to connect to OpenSearch. Error: validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}
Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: {"type":"log","@timestamp":"2023-10-26T05:21:39Z","tags":["fatal","root"],"pid":19483,"message":"ResponseError: validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (events.js:412:35)\n    at IncomingMessage.emit (domain.js:475:12)\n    at endReadableNT (internal/streams/readable.js:1333:12)\n    at processTicksAndRejections (internal/process/task_queues.js:82:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '379'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'opensearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}
Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: {"type":"log","@timestamp":"2023-10-26T05:21:39Z","tags":["info","plugins-system"],"pid":19483,"message":"Stopping all plugins."}
Oct 26 10:51:39 wazuh opensearch-dashboards[19483]: FATAL  {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"},"status":400}
Oct 26 10:51:39 wazuh systemd[1]: wazuh-dashboard.service: main process exited, code=exited, status=1/FAILURE
Oct 26 10:51:39 wazuh systemd[1]: Unit wazuh-dashboard.service entered failed state.
Oct 26 10:51:39 wazuh systemd[1]: wazuh-dashboard.service failed.
[root@wazuh ~]#

Please help.

[Harshal Paliwal]

Hi Team,Thanks for using the Wazuh.
I can see the following error in the logs cluster currently has [1000]/[1000] maximum shards open you can resolve that by following the process. 

type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception"...

There are two possible solutions:

• Increase the shards limit.
• Reduce the number of shards.

Increase the shards limit:
This option will quickly solve the solution but it is not advisable for the long run as it will bring more problems in the future. However, this guide will explain how to do it in case it is needed.
The following setting is the one responsible for this limit: cluster.routing.allocation.total_shards_per_node
It is possible to change the setting using the WI API. You can either use the Dev tools option within the management section in the Wazuh Dashboard:

PUT _cluster/settings { "persistent" : { "cluster.routing.allocation.total_shards_per_node" : 1200 } }

or curl the API directly from a terminal:

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d' { "persistent" : { "cluster.routing.allocation.total_shards_per_node" : 1200 } } '

Reduce the number of shards:
Reaching the limit of shards means no retention policies are applied to the environment. This could lead to storing the data forever and cause failure in the system.
It is necessary to delete old indices to reduce the number of shards. It is necessary to check what the indices stored in the environment, the following API call can help:

GET _cat/indices

Then, it is necessary to delete indices that are not needed or older indices. Bear in mind that this cannot be retrieved unless there are backups of the data either using snapshots or Wazuh alerts backups.
The API call to delete indices is:
DELETE <index_name>
We always recommend This option.Hope this information helps you. Please feel free to reach out to us for any information/issues.