Alertmanger "Not Grouped" alerts

45 views
Skip to first unread message

Romenyrr

unread,
Jun 19, 2020, 2:34:21 AM6/19/20
to Prometheus Users
I've come across this issue where I'm grouping by 'alertname' but nothing is being grouped except for one odd group. When I click on the group tab and click on "Enable custom grouping" that seems to sort everything by 'alertname'. 

This grouping issue is creating an issue where I'm just getting 1 big alert in Opsgenie with 74 items in it. Has anyone come across this before? 

2765A9DC-B163-4D2B-A83B-07D34E71A66F.jpeg

Here's the ouput of Status > Config from the Alertmanager UI


route:
  receiver: opsgenie
  routes:
  - receiver: opsgenie
    group_by:
    - alertname
    match_re:
      severity: warning|critical
    group_wait: 10s
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 30m
receivers:
- name: opsgenie
  opsgenie_configs:
  - send_resolved: true
    http_config: {}
    api_key: <secret>
  


Christian Hoffmann

unread,
Jun 19, 2020, 10:57:58 AM6/19/20
to Romenyrr, Prometheus Users
Hi,

On 6/19/20 8:34 AM, Romenyrr wrote:
> I've come across this issue where I'm grouping by 'alertname' but
> nothing is being grouped except for one odd group. When I click on the
> group tab and click on "Enable custom grouping" that seems to sort
> everything by 'alertname'. 
>
> This grouping issue is creating an issue where I'm just getting 1 big
> alert in Opsgenie with 74 items in it. Has anyone come across this before? 
>
> 2765A9DC-B163-4D2B-A83B-07D34E71A66F.jpeg
>
> Here's the ouput of Status > Config from the Alertmanager UI
>
> |
>
> route:
>   receiver: opsgenie
>   routes:
>   - receiver: opsgenie
>     group_by:
>     - alertname
>     match_re:
>       severity: warning|critical
>     group_wait: 10s

Can you confirm that all of your affected alerts have a severity label
of either warning or critical?

All others will probably be handled by the default route where you don't
have a group_by config. Maybe you intended to place it there instead?

If receiver and everything else is identical, I don't think you should
have child routes (routes:) at all. Just place it at the top-level route
instead.

Kind regards,
Christian

Romenyrr

unread,
Jun 20, 2020, 12:46:11 AM6/20/20
to Prometheus Users
You're a life saver! Some alerts were Critical and others were critical, but the regex was looking for lowercase critical. Adjusting all the rules with the correct severities fixed the issue. Appreciate the tip!
Reply all
Reply to author
Forward
0 new messages