3 of 37 Shards Failed

93 views
Skip to first unread message

saidurgayaswanth

unread,
Apr 22, 2025, 5:20:05 AM4/22/25
to Wazuh | Mailing List
 Hi Team , 
  
                  I am facing this issue after creating ISM for my indexes. Additionally, I’ve created dashboards for these indexes . it was displaying that the 3 of 37 shards have been failed .
I have tried multiple ways but unable to resolve the issue after several attempts. So, could you please assist me in solving this problem? I have attached the relevant screenshots for your reference.  

{ "took": 253, "timed_out": false, "_shards": { "total": 37, "successful": 34, "skipped": 0, "failed": 3, "failures": [ { "shard": 0, "index": "wazuh-archives-4.x-2025.04.22", "node": "WAoZTZTCSHiC0lQoMmuYmw", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [manager.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } ] }, "hits": { "total": 0, "max_score": null, "hits": [] }, "aggregations": { "buckets": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [] } } }
_______________________________________________________________________________________
{ "sort": [], "size": 100, "from": 0, "aggs": { "buckets": { "terms": { "field": "manager.name", "size": 5, "order": { "_count": "desc" } } } }, "stored_fields": [ "*" ], "script_fields": {}, "docvalue_fields": [ { "field": "@timestamp", "format": "date_time" }, { "field": "data.aws.createdAt", "format": "date_time" }, { "field": "data.aws.end", "format": "date_time" }, { "field": "data.aws.resource.instanceDetails.launchTime", "format": "date_time" }, { "field": "data.aws.responseElements.CreateVpcEndpointResponse.vpcEndpoint.creationTimestamp", "format": "date_time" }, { "field": "data.aws.service.eventFirstSeen", "format": "date_time" }, { "field": "data.aws.service.eventLastSeen", "format": "date_time" }, { "field": "data.aws.start", "format": "date_time" }, { "field": "data.aws.updatedAt", "format": "date_time" }, { "field": "data.ms-graph.activityDateTime", "format": "date_time" }, { "field": "data.ms-graph.complianceGracePeriodExpirationDateTime", "format": "date_time" }, { "field": "data.ms-graph.createdDateTime", "format": "date_time" }, { "field": "data.ms-graph.deviceActionResults.lastUpdatedDateTime", "format": "date_time" }, { "field": "data.ms-graph.deviceActionResults.startDateTime", "format": "date_time" }, { "field": "data.ms-graph.deviceHealthAttestationState.issuedDateTime", "format": "date_time" }, { "field": "data.ms-graph.deviceHealthAttestationState.lastUpdateDateTime", "format": "date_time" }, { "field": "data.ms-graph.easActivationDateTime", "format": "date_time" }, { "field": "data.ms-graph.enrolledDateTime", "format": "date_time" }, { "field": "data.ms-graph.exchangeLastSuccessfulSyncDateTime", "format": "date_time" }, { "field": "data.ms-graph.firstActivityDateTime", "format": "date_time" }, { "field": "data.ms-graph.lastActivityDateTime", "format": "date_time" }, { "field": "data.ms-graph.lastSyncDateTime", "format": "date_time" }, { "field": "data.ms-graph.lastUpdateDateTime", "format": "date_time" }, { "field": "data.ms-graph.managementCertificateExpirationDate", "format": "date_time" }, { "field": "data.ms-graph.resolvedDateTime", "format": "date_time" }, { "field": "data.timestamp", "format": "date_time" }, { "field": "data.vulnerability.published", "format": "date_time" }, { "field": "data.vulnerability.updated", "format": "date_time" }, { "field": "syscheck.mtime_after", "format": "date_time" }, { "field": "syscheck.mtime_before", "format": "date_time" }, { "field": "timestamp", "format": "date_time" } ], "_source": { "excludes": [] }, "query": { "bool": { "must": [], "filter": [ { "range": { "timestamp": { "gte": "now-24h", "lte": "now", "format": "epoch_millis" } } }, { "range": { "rule.level": { "gte": 12, "lte": 14 } } }, { "match_phrase": { "manager.name": { "query": "ip-10-0-2-44" } } } ], "should": [], "must_not": [] } } }

Thanks in Advance 
ss2.png
ss1.png

Stuti Gupta

unread,
Apr 22, 2025, 6:43:50 AM4/22/25
to Wazuh | Mailing List
Hi  saidurgayaswanth

Could you please let me know your Wazuh deployment type? Are you using the Wazuh OVA or a distributed deployment?

Can you please share the ILM policy that you applied? 

From the filebeat test output command, it appears that Filebeat is running correctly. However, to investigate further, we need to review the log files. Please run the following command on the Wazuh manager to capture any relevant log entries:
sudo cat /var/log/filebeat/filebeat
This will create a filebeat.log file in the directory where you run the command. Kindly share this log file with us.

Additionally, please check the Wazuh Indexer logs using the command below and share the complete output:
sudo cat /var/log/wazuh-indexer/wazuh-cluster.log

To check the health status, open the dashboard and click on the hamburger menu in the top-left corner. Navigate to Indexer Management > Dev Tools.
In the Dev Tools console, run the following API calls:
GET /_cluster/health

Also share the output of the following command  GET /_cat/shards?v

Please share the full output of these commands so we can assist you further.

saidurgayaswanth

unread,
Apr 22, 2025, 7:48:50 AM4/22/25
to Wazuh | Mailing List
Shards :- 
index                                              shard prirep state       docs   store ip        node
wazuh-statistics-2025.17w                          0     p      STARTED      163 127.4kb 127.0.0.1 node-1
.ql-datasources                                    0     p      STARTED        0    208b 127.0.0.1 node-1
.opendistro-reports-definitions                    0     p      STARTED        0    208b 127.0.0.1 node-1
.opendistro-reports-instances                      0     p      STARTED        1   6.6kb 127.0.0.1 node-1
.opendistro_security                               0     p      STARTED       10 107.2kb 127.0.0.1 node-1
wazuh-statistics-2025.13w                          0     p      STARTED      335 331.3kb 127.0.0.1 node-1
.opensearch-observability                          0     p      STARTED        0    208b 127.0.0.1 node-1
.opensearch-sap-log-types-config                   0     p      STARTED                  127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.04                        0     p      STARTED     2984     5mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.04                        1     p      STARTED     2945   4.7mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.04                        2     p      STARTED     2933   4.7mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.07                        0     p      STARTED     2046   2.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.07                        1     p      STARTED     2047   2.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.07                        2     p      STARTED     1984   2.1mb 127.0.0.1 node-1
wazuh-monitoring-2025.15w                          0     p      STARTED      945 892.5kb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.03                        0     p      STARTED     1162   2.4mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.03                        1     p      STARTED     1169   2.4mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.03                        2     p      STARTED     1129   2.3mb 127.0.0.1 node-1
wazuh-statistics-2025.14w                          0     p      STARTED      432 355.1kb 127.0.0.1 node-1
wazuh-monitoring-2025.17w                          0     p      STARTED      150 188.9kb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.08                        0     p      STARTED     2352   2.7mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.08                        1     p      STARTED     2477   2.7mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.08                        2     p      STARTED     2517   3.1mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.09                        0     p      STARTED     2216   2.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.09                        1     p      STARTED     2284   2.6mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.09                        2     p      STARTED     2262   2.6mb 127.0.0.1 node-1
.opendistro-ism-config                             0     p      STARTED                  127.0.0.1 node-1
.opendistro-ism-config                             0     r      UNASSIGNED                        
wazuh-archives-4.x-2025.03.28                      0     p      STARTED    28024  22.7mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.03.28                      1     p      STARTED    28265  22.8mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.03.28                      2     p      STARTED    27894  23.2mb 127.0.0.1 node-1
wazuh-states-vulnerabilities-ip-10-0-2-44          0     p      STARTED    23800  14.8mb 127.0.0.1 node-1
.tasks                                             0     p      STARTED        2  17.7kb 127.0.0.1 node-1
wazuh-archives-4.x-2025.03.27                      0     p      STARTED    12592  11.8mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.03.27                      1     p      STARTED    12711  12.4mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.03.27                      2     p      STARTED    12670  12.4mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.10                        0     p      STARTED     2367   2.8mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.10                        1     p      STARTED     2268   2.6mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.10                        2     p      STARTED     2368   2.8mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.15                        0     p      STARTED     2081   2.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.15                        1     p      STARTED     2028   2.2mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.15                        2     p      STARTED     2084   3.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.16                        0     p      STARTED     1907     2mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.16                        1     p      STARTED     2005     2mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.16                        2     p      STARTED     1998   2.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.17                        0     p      STARTED     3917   4.1mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.11                        0     p      STARTED     1975   2.1mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.11                        1     p      STARTED     1916   2.9mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.11                        2     p      STARTED     2010   2.1mb 127.0.0.1 node-1
wazuh-monitoring-2025.16w                          0     p      STARTED      515 477.8kb 127.0.0.1 node-1
wazuh-statistics-2025.15w                          0     p      STARTED     1100 758.1kb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.04                      0     p      STARTED    32031  27.2mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.04                      1     p      STARTED    31970  26.5mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.04                      2     p      STARTED    31739  26.8mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.03                      0     p      STARTED    77528  97.3mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.03                      1     p      STARTED    76852  96.5mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.03                      2     p      STARTED    77098  97.1mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.09                      0     p      STARTED    15304  22.2mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.09                      1     p      STARTED    15478  22.4mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.09                      2     p      STARTED    15365  22.4mb 127.0.0.1 node-1
.opendistro-ism-managed-index-history-2025.04.16-1 0     p      STARTED                  127.0.0.1 node-1
.opendistro-ism-managed-index-history-2025.04.16-1 0     r      UNASSIGNED                        
wazuh-archives-4.x-2025.04.08                      0     p      STARTED    17821  23.3mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.08                      1     p      STARTED    17935  24.8mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.08                      2     p      STARTED    17886    24mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.07                      0     p      STARTED    43159  50.1mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.07                      1     p      STARTED    43752    51mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.07                      2     p      STARTED    43643  50.3mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.11                      0     p      STARTED    25543  23.5mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.11                      1     p      STARTED    25502    24mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.11                      2     p      STARTED    25490  23.8mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.10                      0     p      STARTED    15477  21.5mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.10                      1     p      STARTED    15408  21.9mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.10                      2     p      STARTED    15311  21.6mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.22                        0     p      STARTED     1879   4.7mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.22                        1     p      STARTED     1866   4.8mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.04.22                        2     p      STARTED     1903     5mb 127.0.0.1 node-1
wazuh-monitoring-2025.13w                          0     p      STARTED       55 230.4kb 127.0.0.1 node-1
wazuh-statistics-2025.16w                          0     p      STARTED      593 566.1kb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.16                      0     p      STARTED    28921  22.4mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.16                      1     p      STARTED    28953  22.6mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.16                      2     p      STARTED    29053    22mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.15                      0     p      STARTED    56778  66.3mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.15                      1     p      STARTED    56729  66.4mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.15                      2     p      STARTED    56743  66.3mb 127.0.0.1 node-1
.kibana_1                                          0     p      STARTED       21 120.5kb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.17                      0     p      STARTED    73346  62.5mb 127.0.0.1 node-1
.plugins-ml-config                                 0     p      STARTED        1   3.9kb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.22                      0     p      STARTED    69164 108.5mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.22                      1     p      STARTED    69615 105.1mb 127.0.0.1 node-1
wazuh-archives-4.x-2025.04.22                      2     p      STARTED    69484 104.7mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.03.27                        0     p      STARTED      434   1.3mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.03.27                        1     p      STARTED      433   1.1mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.03.27                        2     p      STARTED      383     1mb 127.0.0.1 node-1
.opendistro-job-scheduler-lock                     0     p      STARTED       37 205.7kb 127.0.0.1 node-1
.opendistro-job-scheduler-lock                     0     r      UNASSIGNED                        
wazuh-alerts-4.x-2025.03.28                        0     p      STARTED     1394   2.9mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.03.28                        1     p      STARTED     1401   2.8mb 127.0.0.1 node-1
wazuh-alerts-4.x-2025.03.28                        2     p      STARTED     1368   2.7mb 127.0.0.1 node-1
wazuh-monitoring-2025.14w                          0     p      STARTED      285 269.3kb 127.0.0.1 node-1
.opensearch-notifications-config                   0     p      STARTED                  127.0.0.1 node-1

saidurgayaswanth

unread,
Apr 22, 2025, 7:48:51 AM4/22/25
to Wazuh | Mailing List
Cluster Health : - 

{
  "cluster_name": "wazuh-cluster",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 1,
  "number_of_data_nodes": 1,
  "discovered_master": true,
  "discovered_cluster_manager": true,
  "active_primary_shards": 98,
  "active_shards": 98,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 3,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 97.02970297029702
}
Thanks For looking into this issue !!

saidurgayaswanth

unread,
Apr 22, 2025, 8:07:45 AM4/22/25
to Wazuh | Mailing List
You can use these files for all the info needed 
ISM policy.txt
shards.txt
wazuh cluster.txt

saidurgayaswanth

unread,
Apr 22, 2025, 8:33:37 AM4/22/25
to Wazuh | Mailing List
I Was using standalone instance hosted in AWS 

Stuti Gupta

unread,
Apr 29, 2025, 7:10:12 AM4/29/25
to Wazuh | Mailing List
Please remove the unassigned shards 
Delete the unassigned shards using the following command:
curl -k -XGET -u user:pass "https://<indexe>:9200/_cat/shards" | grep UNASSIGNED | awk '{print $1}' | xargs -i curl -k -XDELETE -u user:pass "https://<indexer_ip>:9200/{}"

Restart the wazuh-indexer and see if you are still getting the error. In that case please share the Wazuh Indexer logs, using the command below 
sudo cat /var/log/wazuh-indexer/wazuh-cluster.log
Reply all
Reply to author
Forward
0 new messages