Wazuh-Dashboard Failure after upgrade to v4.9

[Wazuh User 0361]

Hello need help..

wazuh-dashboard failure after upgrade to v4.9

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["error","opensearch","data"],"pid":56585,"message":"[validation_exception]: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["warning","savedobjects-service"],"pid":56585,"message":"Unable to connect to OpenSearch. Error: validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["fatal","root"],"pid":56585,"message":"ResponseError: validation_exception: [validation_exception] Reason: Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;\n    at onBody (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:374:23)\n    at IncomingMessage.onEnd (/usr/share/wazuh-dashboard/node_modules/@opensearch-project/opensearch/lib/Transport.js:293:11)\n    at IncomingMessage.emit (node:events:529:35)\n    at IncomingMessage.emit (node:domain:489:12)\n    at endReadableNT (node:internal/streams/readable:1400:12)\n    at processTicksAndRejections (node:internal/process/task_queues:82:21) {\n  meta: {\n    body: { error: [Object], status: 400 },\n    statusCode: 400,\n    headers: {\n      'content-type': 'application/json; charset=UTF-8',\n      'content-length': '379'\n    },\n    meta: {\n      context: null,\n      request: [Object],\n      name: 'opensearch-js',\n      connection: [Object],\n      attempts: 0,\n      aborted: false\n    }\n  }\n}"}

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["info","plugins-system"],"pid":56585,"message":"Stopping all plugins."}

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]:  FATAL  {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}],"type":"validation_exception","reason":"Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"},"status":400}

Sep 24 16:43:45 wazuh-server systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE

Sep 24 16:43:45 wazuh-server systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.

Sep 24 16:43:45 wazuh-server systemd[1]: wazuh-dashboard.service: Consumed 11.767s CPU time.

# systemctl status wazuh-dashboard

× wazuh-dashboard.service - wazuh-dashboard

     Loaded: loaded (/etc/systemd/system/wazuh-dashboard.service; enabled; vendor preset: enabled)

     Active: failed (Result: exit-code) since Tue 2024-09-24 16:43:45 +08; 12min ago

    Process: 56585 ExecStart=/usr/share/wazuh-dashboard/bin/opensearch-dashboards (code=exited, status=1/FAILURE)

   Main PID: 56585 (code=exited, status=1/FAILURE)

        CPU: 11.767s

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["info","savedobjects-service"],"pid":565>

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["info","savedobjects-service"],"pid":565>

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["error","opensearch","data"],"pid":56585>

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["warning","savedobjects-service"],"pid":>

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["fatal","root"],"pid":56585,"message":"R>

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]: {"type":"log","@timestamp":"2024-09-24T08:43:45Z","tags":["info","plugins-system"],"pid":56585,"me>

Sep 24 16:43:45 wazuh-server opensearch-dashboards[56585]:  FATAL  {"error":{"root_cause":[{"type":"validation_exception","reason":"Validation Failed: 1: thi>

Sep 24 16:43:45 wazuh-server systemd[1]: wazuh-dashboard.service: Main process exited, code=exited, status=1/FAILURE

Sep 24 16:43:45 wazuh-server systemd[1]: wazuh-dashboard.service: Failed with result 'exit-code'.

Sep 24 16:43:45 wazuh-server systemd[1]: wazuh-dashboard.service: Consumed 11.767s CPU time.

# cat /etc/wazuh-dashboard/opensearch_dashboards.yml

server.host: 0.0.0.0

server.port: 443

opensearch.hosts: https://localhost:9200

opensearch.ssl.verificationMode: certificate

#opensearch.username:

#opensearch.password:

opensearch.requestHeadersAllowlist: ["securitytenant","Authorization"]

opensearch_security.multitenancy.enabled: false

opensearch_security.readonly_mode.roles: ["kibana_read_only"]

server.ssl.enabled: true

server.ssl.key: "/etc/wazuh-dashboard/certs/dashboard-key.pem"

server.ssl.certificate: "/etc/wazuh-dashboard/certs/dashboard.pem"

opensearch.ssl.certificateAuthorities: ["/etc/wazuh-dashboard/certs/root-ca.pem"]

uiSettings.overrides.defaultRoute: /app/wz-home

All status indexer, manager, and filebeat are OK

# systemctl status wazuh-indexer

● wazuh-indexer.service - wazuh-indexer

     Loaded: loaded (/lib/systemd/system/wazuh-indexer.service; enabled; vendor preset: enabled)

     Active: active (running) since Tue 2024-09-24 16:33:02 +08; 25min ago

       Docs: https://documentation.wazuh.com

   Main PID: 47287 (java)

      Tasks: 138 (limit: 14191)

     Memory: 5.1G

        CPU: 5min 16.820s

     CGroup: /system.slice/wazuh-indexer.service

             └─47287 /usr/share/wazuh-indexer/jdk/bin/java -Xshare:auto -Dopensearch.networkaddress.cache.ttl=60 -Dopensearch.networkaddress.cache.negative.t>

# systemctl status wazuh-manager

● wazuh-manager.service - Wazuh manager

     Loaded: loaded (/lib/systemd/system/wazuh-manager.service; enabled; vendor preset: enabled)

     Active: active (running) since Tue 2024-09-24 16:40:29 +08; 17min ago

      Tasks: 252 (limit: 14191)

     Memory: 283.9M

        CPU: 56.731s

     CGroup: /system.slice/wazuh-manager.service

             ├─53580 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py

             ├─53581 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py

             ├─53584 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py

             ├─53587 /var/ossec/framework/python/bin/python3 /var/ossec/api/scripts/wazuh_apid.py

             ├─53628 /var/ossec/bin/wazuh-authd

             ├─53650 /var/ossec/bin/wazuh-db

             ├─53669 /var/ossec/bin/wazuh-execd

             ├─53694 /var/ossec/bin/wazuh-analysisd

             ├─53725 /var/ossec/bin/wazuh-syscheckd

             ├─53739 /var/ossec/bin/wazuh-remoted

             ├─53900 /var/ossec/bin/wazuh-logcollector

             ├─53925 /var/ossec/bin/wazuh-monitord

             └─53956 /var/ossec/bin/wazuh-modulesd

# filebeat test output

elasticsearch: https://127.0.0.1:9200...

  parse url... OK

  connection...

    parse host... OK

    dns lookup... OK

    addresses: 127.0.0.1

    dial up... OK

  TLS...

    security: server's certificate chain verification is enabled

    handshake... OK

    TLS version: TLSv1.2

    dial up... OK

  talk to server... OK

  version: 7.10.2

[Stuti Gupta]

Hi 

Validation Failed: 1: this action would add [2] total shards, but this cluster currently has [1000]/[1000] maximum shards open;"}

Your wazuh-interface is not coming up because the shards is reached their limit. The default shard limit is 1000, and increasing it is not recommended because each shard consumes significant resources (CPU, memory, and disk I/O), even if it holds minimal data. Too many small shards can lead to slower query performance, higher overhead, cluster instability, and resource exhaustion. Additionally, the current increased shard count is already nearing capacity. Instead of increasing the shard limit, we recommend the following solutions:

Solution 1: Manually Delete Indices
You should review the stored indices using the following API call:
GET _cat/indices
From there, you can delete unnecessary or old indices. Note that deleted indices cannot be retrieved unless backed up through snapshots or Wazuh alert backups. The API call to delete indices is:
DELETE <index_name>
Or via the CLI:
curl -k -u admin:admin -XDELETE https://<WAZUH_INDEXER_IP>:9200/wazuh-alerts-4.x-YYYY.MM.DD
You can also use wildcards (*) to delete multiple indices in one go.

Solution 2: Index Management Policies
You can automate index deletion by setting up Index Lifecycle Management (ILM) policies, as explained in this post:(https://wazuh.com/blog/wazuh-index-management). Additionally, you can set up snapshots to automatically back up Wazuh indices to local or cloud storage for restoration when needed. More details on this can be found in the (https://wazuh.com/blog/index-backup-management) article.

Solution 3: Add an Indexer Node
Adding another indexer node will increase the capacity and resilience of your Wazuh monitoring infrastructure. For more information on how to do this, refer to the official guide: (https://documentation.wazuh.com/current/user-manual/upscaling/adding-indexer-node.html).

Hope this helps 

[Wazuh User 0361]

Hello,
Anyone can help me please..

[Stuti Gupta]

HI 

Please let me know if you have any issues related to the above query or the solution provided. You can also delete the unassigned shards using the command:
curl -k -XGET -u user:pass "https://<elasticsearxch>:9200/_cat/shards" | grep UNASSIGNED | awk '{print $1}' | xargs -i curl -k -XDELETE -u user:pass "https://<indexer_ip>:9200/{}"  
 
To know the cluster health that includes shards details you can follow the command:
curl -XGET -k -u user:pass "https://localhost:9200/_cluster/health"


Hope to hear from you soon

[Wazuh User 0361]

Hello, thank you for your help..

when i check with curl to my server there is error, but if i curl to external web link google.com it is ok

# curl -XGET -k -u user:password "https://localhost:9200/_cluster/health"

curl: (35) Send failure: Broken pipe

--
You received this message because you are subscribed to the Google Groups "Wazuh | Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/c61c6219-854a-4185-a93d-6a3e8eded841n%40googlegroups.com.

[Wazuh User 0361]

Hello this is my cluster health

# curl -XGET -k -u user:password "https://localhost:9200/_cluster/health"                              

 {"cluster_name":"wazuh-cluster","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":1000,"active_shards":1000,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}

[Wazuh User 0361]

# curl -k -u user:password "https://127.0.0.1:9200/_cat/allocation?v"

shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node

  1000        2.5gb   124.6gb        2tb      2.1tb            5 127.0.0.1 127.0.0.1 node-1

Can i add more shard more than 1000 ?

[Stuti Gupta]

Hi 

As explained above the Sharda are full and increasing it is not recommended because each shard consumes significant resources (CPU, memory, and disk I/O), even if it holds minimal data.  , we recommend the following solutions:

Solution 1: Manually Delete Indices
You should review the stored indices using the following API call:
GET _cat/indices
From there, you can delete unnecessary or old indices. Note that deleted indices cannot be retrieved unless backed up through snapshots or Wazuh alert backups. The API call to delete indices is:
DELETE <index_name>
Or via the CLI:
curl -k -u admin:admin -XDELETE https://<WAZUH_INDEXER_IP>:9200/wazuh-alerts-4.x-YYYY.MM.DD
You can also use wildcards (*) to delete multiple indices in one go.

Solution 2: Index Management Policies
You can automate index deletion by setting up Index Lifecycle Management (ILM) policies, as explained in this post:(https://wazuh.com/blog/wazuh-index-management). Additionally, you can set up snapshots to automatically back up Wazuh indices to local or cloud storage for restoration when needed. More details on this can be found in the (https://wazuh.com/blog/index-backup-management) article.

Solution 3: Add an Indexer Node
Adding another indexer node will increase the capacity and resilience of your Wazuh monitoring infrastructure. For more information on how to do this, refer to the official guide: (https://documentation.wazuh.com/current/user-manual/upscaling/adding-indexer-node.html).

Hope this helps