Wazuh Dashboard cannot show suggestions for new data

[Simone Brazioli]

Hello to all,

I am getting a Wazuh cluster (currently 2 managers + 1 Indexer + 1 Dashboard) ready for production. This week I found out that while using either Wazuh built-in dashboards or our custom Opensearch dashboards, the suggested values in the filter interfaces appear to be out of sync with the indices and the alerts. I never noticed this delay before (but I can't be sure it wasn't acting like that earlier).

As an example:

I add a new agent, thus generating a number of alerts with field 'agent.name': 'new_agent_name', which is a never-seen-before value. Once I go in some of my dashboards and try to filter alerts based on agent.name with the new name, the filter interface does not list the new agent name as a suggestion and I have to override it as a 'custom option' so that the alerts can show up.

This lasted at least for about a day in this particular case,  but in one of the example I am currently analyzing has lasted since monday (i.e. ~3 days). I tried restarting the cluster a couple of times and results have not been consistent.

I went over the logs in all nodes and troubleshooted quite a bit but could not find some smoking guns about this problem or something useful in docs / mailing list.

Do any of you have some suggestions (pun intended) as to where to look for hints of the problem?

Thanks in advance,

Simone

[Simone Brazioli]

Looking for info, I realized that this problem could very much be linked to the alert I get when using the Controls Visualization, instead of the basic top bar filter. It states that 'Terms list might be incomplete (...) Adjust the autocomplete settings in opensearch_dashboard'. I am attaching the screenshot to show where I can see that. The config file mentioned is supposedly `/etc/wazuh-dashboard/opensearch_dashboards.yml`. However, I searched pretty much with all keywords I could think of and nothing in the doc nor the groups seems to hint to some specific configuration I can setup there about the autocomplete functionality. (https://opensearch.org/docs/latest/opensearch/search/autocomplete/#completion-suggester and https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/config/opensearch_dashboards.yml and https://opensearch.org/docs/latest/security-plugin/configuration/yaml/#opensearchyml)

I am going to try and 'brute-force' my way around to get some configuration, but if anyone has a doc page or group thread that could be useful, I am all ears (eyes, actually).

Simone

[Simone Brazioli]

Good day to all,

I think I figured out what the problem I am facing is.

I put the indexer in TRACE log mode, by adding/editing in `/etc/wazuh-indexer/opensearch.yml`

```

logger.level: TRACE

```

Tailing carefully the log files I found the query that the dashboard passes to the indexer to get value suggestions. I then tried and using that through the Dev Tools to get some insight.

If I try and get suggestions for alerts containing 'johndoe' as value for the field 'data.postfix.to', the dashboard sends a request to the indexer, containing this kind of query:

```

GET _search
{
  "size": 0,
  "timeout": "1000ms",
  "terminate_after": 100000,
  "query": {
    "match_all": {
      "boost": 1
    }
  },
  "aggregations": {
    "suggestions": {
      "terms": {
        "field": "data.postfix.to",
        "size": 10,
        "shard_size": 10,
        "min_doc_count": 1,
        "shard_min_doc_count": 0,
        "show_term_doc_count_error": false,
        "execution_hint": "map",
        "order": [
          {
            "_count": "desc"
          },
          {
            "_key": "asc"
          }
        ],
        "include": ".*johndoe.*"
      }
    }
  }
}

```

Assuming `johndoe` is one of the records that cannot be returned, the default parameters yields an empty response.

Fiddling around with the parameters, I figured that changing either the `timeout` or the `terminate_after` keyword could improve the results. The timeout did not prove useful.

However changing the `terminate_after` value from 100,000 to 1,000,000 (10x) resulted in the query returning the value I was looking for.

Assuming that my understanding is correct, that `terminate_after` applies the query to the first 100,000 docs as a default, I would like to ask you if there is a way of editing and increasing the default value.

I know that that would result in waiting a tad longer to get suggestions, but it is a price I would pay to get correct suggestions in my filters, if there are no smart alternatives. I will continue to look through the config to find a way to edit that, but if any of you already knows how to do it (or knows that it is not possible) that would save me a bit of a headache.

Cheers,

Simone

P.S.: Since this problem arises (AFAIK) because of the high number of alerts, I am curious about how generally admins manage this is situation. I have something like a month of alerts in this cluster and that does not seem like a overwhelming amount of data (not for this purpose at least!).

[Manuel Pedro Gomez Castro]

Hello! Thank you for reaching out to us!

It's possible that a high volume of events may be causing your issue, and I'm glad that you found a solution that, for the time being, seems to match your requirement.

I will bring this to the attention of the app team to see if they can implement a feature to modify the 'terminate_after' limit easily.

However, I notice that the search term you shared does not include anything on which indexes to search in.
I believe you could filter a great deal of entries by limiting your search to the 'wazuh-alerts-*' index pattern as you would be excluding all the information regarding agent status or wazuh statistics. https://documentation.wazuh.com/current/user-manual/elasticsearch/elasticsearch.html

Please keep us posted if you find more information on this matter and feel free to ask any other questions!