Courier fetch: 5 of 15 shards failed.

[Utkarsh Bhargava]

Hi folks,

I am facing an error on elastic search

"Courier fetch: 5 of 15 shards failed." ( screenshot attached )

I tried multiple things but nothing worked. Can anyone tell me how to
fix this issue ?

[Javier Castro]

Hi Utkarsh,

that error can happen because of several reasons.

Can you share what were you doing in the app when it appears?

Also, if you open the Development Tools in your browser and go to the Network tab we can see the specific requests the app is sending to elasticsearch, hopefully we will be able to see the cause.

Best regards.

--
You received this message because you are subscribed to the Google Groups "Wazuh mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to wazuh+unsubscribe@googlegroups.com.
To post to this group, send email to wa...@googlegroups.com.
Visit this group at https://groups.google.com/group/wazuh.
To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/dc9b1e13-6a54-dbe6-f153-b50bc23696b5%40null.co.in.
For more options, visit https://groups.google.com/d/optout.

[Simon Tideswell]

I see the same issue too. It did not occur at all until I upgraded to Wazuh 3.4.0 and ELK 6.3.1, before that I never saw it occur.

I only experience the issue when I click on either of the "File Integrity" or "General" links inside the Wazuh application (screenshot).

There is an associated error in the Elasticsearch log ...

--------------------------------------------------------------

[2018-08-20T13:47:31,135][DEBUG][o.e.a.s.TransportSearchAction] [sa1xm9R] [wazuh-alerts-3.x-2018.08.19][0], node[sa1xm9R2Q96rQWMR1Jy0Eg], [P], s[STARTED], a[id=fThou_10QMqHFaew8gUIwA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[wazuh-alerts-3.x-*], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=[], routing='null', preference='1534643861041', requestCache=null, scroll=null, maxConcurrentShardRequests=5, batchedReduceSize=512, preFilterShardSize=32, allowPartialSearchResults=true, source={"size":0,"query":{"bool":{"must":[{"match_all":{"boost":1.0}},{"range":{"@timestamp":{"from":1534650450546,"to":1534736850546,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"manager.name":{"query":"syd-ossec-02","slop":0,"zero_terms_query":"NONE","boost":1.0}}},{"match_phrase":{"rule.groups":{"query":"syscheck","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"_source":{"includes":[],"excludes":[]},"stored_fields":"*","docvalue_fields":["@timestamp","data.vulnerability.published","data.vulnerability.updated","syscheck.mtime_after","syscheck.mtime_before","data.cis.timestamp"],"script_fields":{},"aggregations":{"2":{"terms":{"field":"syscheck.path","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}] lastShard [true]

org.elasticsearch.transport.RemoteTransportException: [sa1xm9R][127.0.0.1:9300][indices:data/read/search[phase/query]]

Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@68f37724 on QueueResizingEsThreadPoolExecutor[name = sa1xm9R/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 785nanos, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@5b4ba897[Running, pool size = 4, active threads = 4, queued tasks = 1343, completed tasks = 296371]]

--------------------------------------------------------------

I've done some troubleshooting, but, somewhat typically, got distracted by something else before I managed to find a suitable resolution :o)

Simon

[Juanjo Jiménez]

Hello Utkarsh and Simon,

Just like Javier told you, the error could happen because of several reasons. The log error indicates that you’ve got 1000 search requests that have queued up waiting to run, and once the limit is reached, Elasticsearch just starts aborting new requests.

We could do several things in order to fix this:

• Check the RAM memory on your Elasticsearch nodes. If you have a large amount of data stored in indices, you should consider having at least a minimum of 8GB of RAM per node to increase performance.
• Try to reduce the time range on the search bar so the app can narrow the data search to a smaller period of time, and load fewer data.
• You can also try to increase the maximum queue capacity opening the Elasticsearch configuration file (located at /etc/elasticsearch/elasticsearch.yml) and pasting the following configuration:

thread_pool:
  search:
    queue_size: 10000

Be careful with the indentations (2 spaces).

After modifying the file, you have to restart Elasticsearch:

systemctl restart elasticsearch

Make sure Elasticsearch is up and running before opening Kibana with the following command:

curl localhost:9200?pretty

And make sure you get a response like the following:

{
  "name" : "bXQ_XEr",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "jt7nCsLrTQuVtsMbnOGIeQ",
  "version" : {
    "number" : "6.3.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "053779d",
    "build_date" : "2018-07-20T05:20:23.451332Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

After that, you can open Kibana and try again to use the app to see if the error persists.

Let us know if this helps.

Regards,
Juanjo

To view this discussion on the web visit https://groups.google.com/d/msgid/wazuh/95cd2325-fb0f-454f-8e60-6f4b28ebd17b%40googlegroups.com.