Per Bucket monitor Error.
никита какдела
musbau....@wazuh.com
The error indicates that there's an issue script inside your bucket selector aggregation. The error message shows your script got cut off. It ends with "source":"params.uniq but is missing the actual condition.
In a bucket selector, the script needs to be a complete expression that returns true or false. For example, if you're trying to filter buckets where the unique IP count is greater than 1, your script should look like:
json
"script": {
"source": "params.uniq > 1"
}
It looks like the script source is incomplete or was truncated. Please double-check the full script in your monitor configuration and make sure it's a valid boolean expression.
You can reference the documentation and issues below
https://docs.opensearch.org/latest/observing-your-data/alerting/monitors/
https://github.com/opensearch-project/alerting/issues/915
https://github.com/opensearch-project/alerting/issues/706
никита какдела
никита какдела
Hello! Do somebody know what does mean this error? (Per Bucket Monitor)
musbau....@wazuh.com
Increasing the limit to 1000/5m might help temporarily by giving the monitor more resources, but it may not fix the root cause if it's the script truncation bug.
Since you have many monitors, I would suggest you
1. First check if the scripts in your failing monitors look complete or cut off in the dashboard
2. Try simplifying one script to something very short like params.uniq > 0 as a test
The GitHub issues show this is a known bug where scripts get cut off at 140 characters when saved
https://github.com/opensearch-project/alerting/issues/915
https://github.com/opensearch-project/alerting/issues/706
никита какдела
{
"buckets_path": {
"_count": "_count"
},
"parent_bucket_path": "composite_agg",
"script": {
"lang": "painless"
},
"gap_policy": "skip"
}
{
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"format": "epoch_millis",
"boost": 1
}
}
{
"term": {
"rule.id": {
"value": "100026",
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"aggregations": {
"composite_agg": {
"composite": {
"size": 10,
"sources": [
{
"terms": {
"field": "agent.name",
"missing_bucket": false,
"order": "asc"
}
}
}
]
}
}
}
}
Can you tell me what's wrong and why do i get this error?
никита какдела
musbau....@wazuh.com
Your configuration looks correct. The fact it works sometimes suggests an environmental issue rather than a configuration bug.
Would you be able to check your cluster's resource usage during the failure times? This intermittent "sometimes" failure could likely be as a result of resource constraints rather than script errors.
I would advise that you do the below to get more insights
1. Check cluster health when the error occurs - look for high CPU, memory pressure, or circuit breaker trips
2. Monitor timing - Does the error happen during peak load times?
3. OpenSearch logs - Search for GeneralScriptException in the cluster logs around the failure time
никита какдела
Perhaps these are the reason why actions in monitors sometimes don't work?
musbau....@wazuh.com
Thank you for the information shared as it gives more insights. This screenshot you provided shows that your cluster has a corrupted monitor definition where someone saved a query as 3=[] (which is invalid). This malformed query is being loaded and causing parsing failures.
QueryShardException[Failed to parse query [*3=[*]]; nested: ParseException[Cannot parse '3=[': Encountered "<EOF>"
I would advise to first search for monitors containing this malformed query then either repair or delete those corrupted monitors.
никита какдела
#! Deprecation: [interval] on [date_histogram] is deprecated, use [fixed_interval] or [calendar_interval] in the future.
{
"_id": "xWdjA5sBoeamHjFYf3Hw",
"_version": 31,
"_seq_no": 37920,
"_primary_term": 39,
"monitor": {
"type": "monitor",
"schema_version": 8,
"name": "MS Windows: Успешное подключение одной УЗ с разных IP адресов",
"monitor_type": "bucket_level_monitor",
"enabled": true,
"enabled_time": 1768918477969,
"schedule": {
"period": {
"interval": 1,
"unit": "MINUTES"
}
},
"inputs": [
{
"search": {
"indices": [
"wazuh-alerts-current"
],
"query": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"from": "{{period_end}}||-300s",
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"format": "epoch_millis",
"boost": 1
}
}
},
],
"must_not": [
{
"terms": {
"data.win.eventdata.targetUserName": [
"ANONYMOUS LOGON",
"АНОНИМНЫЙ ВХОД"
],
"boost": 1
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
},
"aggregations": {
"composite_agg": {
"composite": {
"size": 20,
"sources": [
{
"data.win.eventdata.targetUserName": {
"terms": {
"field": "data.win.eventdata.targetUserName",
"missing_bucket": false,
"order": "asc"
}
}
}
]
},
"aggregations": {
"unique_ips": {
"cardinality": {
"field": "data.win.eventdata.ipAddress"
}
}
}
}
}
}
}
}
],
"triggers": [
{
"bucket_level_trigger": {
"id": "xGdjA5sBoeamHjFYf3Hq",
"name": "MS Windows: Успешное подключение одной УЗ с разных рабочих станций",
"severity": "2",
"condition": {
"buckets_path": {
"uniq": "unique_ips.value"
},
"parent_bucket_path": "composite_agg",
"script": {
"source": "params.uniq > 4",
"lang": "painless"
},
"gap_policy": "skip"
},
"actions": [
{
"id": "notification327539",
"name": "Send to Kaiten",
"destination_id": "xF8JnJoBovKpQ5b8ijIc",
"message_template": {
"source": """{
"title": "Целевая УЗ: {{#ctx.newAlerts}}{{bucket_keys}}{{/ctx.newAlerts}}",
"tags": ["{{ctx.monitor.name}}"],
"links": [
{
"url": "{{#ctx.newAlerts}}https://wazuh.ovp.ru/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!f,sort:!()),metadata:(indexPattern:'wazuh-alerts-*',view:discover))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1h,to:now))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:rule.id,negate:!f,params:(query:'100014'),type:phrase),query:(match_phrase:(rule.id:'100014'))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:data.win.eventdata.targetUserName,negate:!f,params:(query:{{bucket_keys}}),type:phrase),query:(match_phrase:(data.win.eventdata.targetUserName:{{bucket_keys}})))),query:(language:kuery,query:'')){{/ctx.newAlerts}}",
"description": "Открыть в Wazuh Discover"
}
],
"description": "\n- 🚨 Событие: {{ctx.monitor.name}}\n- 🚨 Приоритет: {{ctx.trigger.severity}}\n- ⏳ Время начала: {{ctx.periodStart}} UTC\n- ⌛ Время окончания: {{ctx.periodEnd}} UTC {{#ctx.newAlerts}}\n---{{#sample_documents.0}}\n- 🙎♂️ Инициатор: {{_source.data.win.eventdata.targetUserName}}\n- 👁🗨 Агент: {{_source.agent.name}}\n- 🔎 Правило: {{_source.rule.description}} (id: {{_source.rule.id}})\n- 🚨 Level: {{_source.rule.level}}\n [Открыть в Wazuh](https://wazuh.ovp.ru/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!t,sort:!()),metadata:(indexPattern:'wazuh-alerts-*',view:discover))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:rule.id,negate:!f,params:(query:'100014'),type:phrase),query:(match_phrase:(rule.id:'100014'))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:data.win.eventdata.targetUserName,negate:!f,params:(query:{{bucket_keys}}),type:phrase),query:(match_phrase:(data.win.eventdata.targetUserName:{{bucket_keys}})))),query:(language:kuery,query:''))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now)))\n---{{/sample_documents.0}}{{/ctx.newAlerts}}"
}
""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": [
"NEW"
]
}
}
}
},
{
"id": "notification810195",
"name": "Send to Yandex",
"destination_id": "X4L_5pkBS6jN-8SDuQFi",
"message_template": {
"source": """{
"chat_id": "1/0/191a25c4-b3f1-4e10-a6b1-a412c17b48e5",
"text": "WAZUH\n\n- 🚨 Событие: {{ctx.monitor.name}}\n- 🚨 Приоритет: {{ctx.trigger.severity}}\n- ⏳ Время начала: {{ctx.periodStart}} UTC\n- ⌛ Время окончания: {{ctx.periodEnd}} UTC {{#ctx.newAlerts}}\n---{{#sample_documents.0}}\n- 🙎♂️ Инициатор: {{_source.data.win.eventdata.targetUserName}}\n- 👁🗨 Агент: {{_source.agent.name}}\n- 🔎 Правило: {{_source.rule.description}} (id: {{_source.rule.id}})\n- 🚨 Level: {{_source.rule.level}}\n [Открыть в Wazuh](https://wazuh.ovp.ru/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!t,sort:!()),metadata:(indexPattern:'wazuh-alerts-*',view:discover))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:rule.id,negate:!f,params:(query:'100014'),type:phrase),query:(match_phrase:(rule.id:'100014'))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:data.win.eventdata.targetUserName,negate:!f,params:(query:{{bucket_keys}}),type:phrase),query:(match_phrase:(data.win.eventdata.targetUserName:{{bucket_keys}})))),query:(language:kuery,query:''))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now)))\n---{{/sample_documents.0}}{{/ctx.newAlerts}}"
}
""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": [
"NEW"
]
}
}
}
},
{
"id": "notification225546",
"name": "Send to TG",
"destination_id": "tAAOOpoBAqvA3MNHy-lM",
"message_template": {
"source": """{
"chat_id": "-1002403153612",
"text": "WAZUH\n\n- 🚨 Событие: {{ctx.monitor.name}}\n- 🚨 Приоритет: {{ctx.trigger.severity}}\n- ⏳ Время начала: {{ctx.periodStart}} UTC\n- ⌛ Время окончания: {{ctx.periodEnd}} UTC {{#ctx.newAlerts}}\n---{{#sample_documents.0}}\n- 🙎♂️ Инициатор: {{_source.data.win.eventdata.targetUserName}}\n- 👁🗨 Агент: {{_source.agent.name}}\n- 🔎 Правило: {{_source.rule.description}} (id: {{_source.rule.id}})\n- 🚨 Level: {{_source.rule.level}}\n [Открыть в Wazuh](https://wazuh.ovp.ru/app/data-explorer/discover#?_a=(discover:(columns:!(_source),isDirty:!t,sort:!()),metadata:(indexPattern:'wazuh-alerts-*',view:discover))&_q=(filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:rule.id,negate:!f,params:(query:'100014'),type:phrase),query:(match_phrase:(rule.id:'100014'))),('$state':(store:appState),meta:(alias:!n,disabled:!f,index:'wazuh-alerts-*',key:data.win.eventdata.targetUserName,negate:!f,params:(query:{{bucket_keys}}),type:phrase),query:(match_phrase:(data.win.eventdata.targetUserName:{{bucket_keys}})))),query:(language:kuery,query:''))&_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-24h,to:now)))\n---{{/sample_documents.0}}{{/ctx.newAlerts}}"
}
""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": [
"NEW"
]
}
}
}
}
]
}
}
],
"last_update_time": 1768918477969,
"data_sources": {
"query_index": ".opensearch-alerting-queries",
"findings_index": ".opensearch-alerting-finding-history-write",
"findings_index_pattern": "<.opensearch-alerting-finding-history-{now/d}-1>",
"alerts_index": ".opendistro-alerting-alerts",
"alerts_history_index": ".opendistro-alerting-alert-history-write",
"alerts_history_index_pattern": "<.opendistro-alerting-alert-history-{now/d}-1>",
"comments_index": ".opensearch-alerting-comments-history-write",
"comments_index_pattern": "<.opensearch-alerting-comments-history-{now/d}-1>",
"query_index_mappings_by_type": {},
"findings_enabled": false
},
"delete_query_index_in_every_run": false,
"should_create_single_alert_for_findings": false,
"owner": "alerting"
},
"associated_workflows": []
}
никита какдела
[2026-01-21T11:02:22,465][ERROR][o.o.a.BucketLevelMonitorRunner] [node-1] Failed to retrieve sample documents for alert Z2g44JsB-dPPuwmWQzas from trigger f7QlspsBd-k2aqqxq1jt of monitor gbQlspsBd-k2aqqxq1j8 during execution gbQlspsBd-k2aqqxq1j8_2026-01-21T11:02:22.362117358_7d4e78d8-efaf-4592-82df-7aa4e7b92de7.
[2026-01-21T11:02:22,657][ERROR][o.o.a.BucketLevelMonitorRunner] [node-1] Failed to retrieve sample documents for alert Z2g44JsB-dPPuwmWQzas from trigger f7QlspsBd-k2aqqxq1jt of monitor gbQlspsBd-k2aqqxq1j8 during execution gbQlspsBd-k2aqqxq1j8_2026-01-21T11:02:22.362117358_7d4e78d8-efaf-4592-82df-7aa4e7b92de7.
[2026-01-21T11:02:22,828][ERROR][o.o.a.BucketLevelMonitorRunner] [node-1] Failed to retrieve sample documents for alert Z2g44JsB-dPPuwmWQzas from trigger f7QlspsBd-k2aqqxq1jt of monitor gbQlspsBd-k2aqqxq1j8 during execution gbQlspsBd-k2aqqxq1j8_2026-01-21T11:02:22.362117358_7d4e78d8-efaf-4592-82df-7aa4e7b92de7.
I am getting this ERRORS from this monitor. Can u explain on my monitor whats wrong, please?
#! Deprecation: [interval] on [date_histogram] is deprecated, use [fixed_interval] or [calendar_interval] in the future.
{
"_version": 31,
"_seq_no": 37918,
"_primary_term": 39,
"monitor": {
"type": "monitor",
"schema_version": 8,
"monitor_type": "bucket_level_monitor",
"enabled": true,
"schedule": {
"period": {
"interval": 1,
"unit": "MINUTES"
}
},
"inputs": [
{
"search": {
"indices": [
"wazuh-alerts-current"
],
"query": {
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"format": "epoch_millis",
"boost": 1
}
}
"must_not": [
{
"wildcard": {
"data.win.eventdata.targetUserName": {
"wildcard": "*$",
"boost": 1
}
}
},
{
"terms": {
"data.win.eventdata.targetUserName": [
"Replicaton_Shuttle",
"elmasys",
"esbservice",
"service_1c_1"
],
"boost": 1
}
},
{
"term": {
"data.win.eventdata.authenticationPackageName": {
"value": "Kerberos",
"boost": 1
}
}
}
],
"adjust_pure_negative": true,
"boost": 1
}
"aggregations": {
"composite_agg": {
"composite": {
"size": 20,
"sources": [
{
"data.win.eventdata.targetUserName": {
"terms": {
"field": "data.win.eventdata.targetUserName",
"missing_bucket": false,
"order": "asc"
}
}
}
]
},
"unique_hosts": {
"cardinality": {
"field": "agent.name"
}
}
}
}
}
}
}
}
],
"triggers": [
{
"bucket_level_trigger": {
"id": "f7QlspsBd-k2aqqxq1jt",
"name": "MS Windows: входы более чем с 3-х или на более 3-хоста под одной и той же учетной записью",
"severity": "3",
"condition": {
"buckets_path": {
"uniq": "unique_hosts.value"
},
"parent_bucket_path": "composite_agg",
"script": {
"lang": "painless"
},
"gap_policy": "skip"
"actions": [
{
"id": "notification928046",
"name": "Send to Telegram",
"destination_id": "tAAOOpoBAqvA3MNHy-lM",
"message_template": {
"source": """{
"chat_id": "-1002403153612",
}
""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": [
"NEW"
]
}
}
}
},
{
"name": "Send to Yandex",
"destination_id": "X4L_5pkBS6jN-8SDuQFi",
"message_template": {
"source": """{
"chat_id": "1/0/191a25c4-b3f1-4e10-a6b1-a412c17b48e5",
}
""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": [
"NEW"
]
}
}
}
},
{
"name": "Send to Kaiten",
"destination_id": "xF8JnJoBovKpQ5b8ijIc",
"message_template": {
"source": """{
"title": "Целевая УЗ: {{#ctx.newAlerts}}{{bucket_keys}}{{/ctx.newAlerts}}",
"tags": ["{{ctx.monitor.name}}"],
"links": [
{
"description": "Открыть в Wazuh"
}
],
"description": "\n- 🚨 Событие: {{ctx.monitor.name}}\n- 🚨 Приоритет: {{ctx.trigger.severity}}\n- ⏳ Время начала: {{ctx.periodStart}} UTC\n- ⌛ Время окончания: {{ctx.periodEnd}} UTC {{#ctx.newAlerts}}\n---\n- 🙎♂️ Инициатор: {{bucket_keys}}{{/ctx.newAlerts}}"
}""",
"lang": "mustache"
},
"throttle_enabled": false,
"subject_template": {
"source": "Alerting Notification action",
"lang": "mustache"
},
"action_execution_policy": {
"action_execution_scope": {
"per_alert": {
"actionable_alerts": [
"NEW"
]
}
}
}
}
]
}
}
],
никита какдела
Hi,
musbau....@wazuh.com
Please check this out
GET /_plugins/_alerting/monitors/_search
{
"query": {
"wildcard": {
"monitor.inputs.search.query.query": "*3=[]*"
musbau....@wazuh.com
Please check this out.
GET /_plugins/_alerting/monitors/_search
{
"query": {
"wildcard": {
"monitor.inputs.search.query.query": "*3=[]*"
}
}
}
никита какдела
"took": 1,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 0,
"relation": "eq"
},
"max_score": null,
"hits": []
}
}
никита какдела
I don't understand how to figure out what exactly the error is? Which monitor? The errors are not constant.
Hi,
musbau....@wazuh.com
For every time you see new errors is any change being made on your end? I would advise to restore your Wazuh server to a point before this initial challenge occurred as the previous log you shared showed that it was a change that was made to one of the monitors that caused the error. You can also temporarily disable groups of monitors (by tag or pattern) to see if errors stop when certain monitors are off and narrow down to the problematic ones
Also, based on the last log you shared, you should use the below to check if it gives you more insights. The webhook errors mean monitors are triggering but failing to send notifications.
GET /_plugins/_alerting/alerts/_search
{
"query": {
"term": {
"_id": "X4L_5pkBS6jN-8SDuQFi"
никита какдела
"error": "no handler found for uri [/_plugins/_alerting/alerts/_search] and method [GET]"
}
dev tools response
никита какдела
I have one of the logs.
What command in Dev Tools can I use to find this alert using alert_id?
[2026-01-22T12:51:21,848][ERROR][o.o.a.BucketLevelMonitorRunner] [node-1] Failed to retrieve sample documents for alert JufC5ZsB-dPPuwmWZ1ug from trigger xGdjA5sBoeamHjFYf3Hq of monitor xWdjA5sBoeamHjFYf3Hw during execution xWdjA5sBoeamHjFYf3Hw_2026-01-22T12:51:21.574506534_be3e555b-9d4f-4b1a-ae52-57eeabb96502.