Monitor performance

[никита какдела]

Hello, dear Wazuh team.  I really need your help, because I've already spent all my energy trying to solve this problem. Below I will attach my monitor, which counts "Successful login attempts from 3 or more hosts". The fact is that when I "Run" the query, I get about 10 thousand events, which is a huge load on the monitor and, as a result, incorrect operation of mustache and, in general, skipping events. I need some advice, how can I fix this? Also, I want to display an example of the IPAddress, target/subjectusername fields in this and in all my monitors. agent.name , rule.name , rule.description. I want to output 1 event to mustache that will contain all these fields. Right now, it turns out that if a backet is triggered and it contains 5-10 events, they are all sent to mustache. How can I fix this correctly? Please help me. Also, please explain what I'm doing wrong, what the problem is, and how to fix it.

Monitor Query:
{
    "size": 0,
    "query": {
        "bool": {
            "filter": [
                {
                    "range": {
                        "@timestamp": {
                            "from": "{{period_end}}||-30m",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                },
                {
                    "term": {
                        "rule.id": {
                            "value": "100014",
                            "boost": 1
                        }
                    }
                }
            ],
            "must_not": [
                {
                    "wildcard": {
                        "data.win.eventdata.targetUserName": {
                            "wildcard": "*$",
                            "boost": 1
                        }
                    }
                },
                {
                    "terms": {
                        "data.win.eventdata.targetUserName": [
                            "Replicaton_Shuttle",
                            "elmasys",
                            "esbservice",
                            "service_1c_1",
                            "InfoSec",
                            "service_illinois"
                        ],
                        "boost": 1
                    }
                }
            ],
            "adjust_pure_negative": true,
            "boost": 1
        }
    },
    "aggregations": {
        "composite_agg": {
            "composite": {
                "size": 20,
                "sources": [
                    {
                        "data.win.eventdata.ipAddress": {
                            "terms": {
                                "field": "data.win.eventdata.ipAddress",
                                "missing_bucket": false,
                                "order": "asc"
                            }
                        }
                    }
                ]
            },
            "aggregations": {
                "unique_hosts": {
                    "cardinality": {
                        "field": "agent.name"
                    }
                }
            }
        }
    }
}

Trigger:
{
    "buckets_path": {
        "uniq": "unique_hosts.value"
    },
    "parent_bucket_path": "composite_agg",
    "script": {
        "source": "params.uniq > 2",
        "lang": "painless"
    },
    "gap_policy": "skip"
}

Mustache (Action):
{
  "chat_id": "123123",
  "text": "WAZUH\n\n- 🚨 Событие:  {{ctx.monitor.name}}\n- 🚨 Приоритет:  {{ctx.trigger.severity}}\n- ⏳ Время начала: {{ctx.periodStart}} UTC\n- ⌛ Время окончания: {{ctx.periodEnd}} UTC {{#ctx.newAlerts.0.sample_documents}}\n---\n- 🙎‍♂️ Адрес источника:  {{_source.data.win.eventdata.ipAddress}}\n- 👨‍💻 Инициатор:  {{_source.data.win.eventdata.targetUserName}}\n- 👁‍🗨 Агент:  {{_source.agent.name}}\n- 🔎 Правило: {{_source.rule.description}}\n- 🚨 Level: {{_source.rule.level}}\n [Открыть в Wazuh](https://wazuh.ovp.ru/app/data-explorer/discover#?_a=(discover:(columns:!(agent.name,data.win.eventdata.targetUserName,data.win.eventdata.ipAddress,rule.id,rule.description),isDirty:!f,sort:!()),metadata:(indexPattern:'wazuh-alerts-*',view:discover))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:rule.id,negate:!f,params:!('100015','100016','100017','100018','100019','100020'),type:phrases,value:'100015,%20100016,%20100017,%20100018,%20100019,%20100020'),query:(bool:(minimum_should_match:1,should:!((match_phrase:(rule.id:'100015')),(match_phrase:(rule.id:'100016')),(match_phrase:(rule.id:'100017')),(match_phrase:(rule.id:'100018')),(match_phrase:(rule.id:'100019')),(match_phrase:(rule.id:'100020')))))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:data.win.eventdata.ipAddress,negate:!f,params:(query:'{{bucket_keys}}'),type:phrase),query:(match_phrase:(data.win.eventdata.ipAddress:'{{bucket_keys}}')))),query:(language:kuery,query:''))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'{{ctx.periodStart}}',to:now)))\n---{{/ctx.newAlerts.0.sample_documents}}"
}

[Isaiah Daboh]

Hello,

Please note that I am taking a look at this. I will revert as soon as possible.

Regards,

[Isaiah Daboh]

Hello,

The current query has huge query load (10k events) because despite having the top-level "size":0, the monitor till processes all matching docs inside the aggregation window. Which means if 10,000 events match the rule.id=100014 in 30m, the monitor must load them, aggregate them, evaluate the bucket condition and build sample_documents per bucket.

The composite aggregation used here is best for pagination which is not needed, here we need to use TERMS aggregation which will help exclude buckets with few events early.

Please try this instead:

{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {

              "from": "{{period_end}}||-15m",

              "to": "{{period_end}}",
              "include_lower": true,
              "include_upper": true,
              "format": "epoch_millis"
            }
          }
        },

        {
          "term": { "rule.id": { "value": "100014" } }
        }
      ],

      "must_not": [
        {
          "wildcard": {
            "data.win.eventdata.targetUserName": { "wildcard": "*$" }
          }
        },

        {
          "terms": {
            "data.win.eventdata.targetUserName": [
              "Replicaton_Shuttle",
              "elmasys",
              "esbservice",
              "service_1c_1",
              "InfoSec",
              "service_illinois"
            ]
          }
        }
      ]
    }
  },

  "aggregations": {
    "ip_buckets": {
      "terms": {
        "field": "data.win.eventdata.ipAddress",
        "size": 20,
        "min_doc_count": 3

      },
      "aggregations": {
        "unique_hosts": {
          "cardinality": {
            "field": "agent.name"
          }

        },
        "sample_event": {
          "top_hits": {
            "size": 1,
            "_source": {
              "includes": [
                "data.win.eventdata.ipAddress",
                "data.win.eventdata.targetUserName",
                "agent.name",
                "rule.name",
                "rule.description",
                "rule.level"
              ]
            }
          }
        }
      }
    }
  }
}

The top_hits aggregation added (sample_event with size: 1) will make sure each bucket contains exactly one representative document with all the fields you need. This should resolve the issue with the Mustache too.  

Regards,