alertmanager to Opgenie P1 (critical) incidents only

49 views
Skip to first unread message

Amit Das

unread,
Nov 14, 2021, 2:29:14 PM11/14/21
to Prometheus Users
Hi 
I was looking for a getting alertmanager alerts to opsgenie.
If i have P1 , P2 multiple alerts in the querry i am getting alerts to email as well as opsgenie when i try to get only P1(critical) alerts i am having 422 error or unexpected EOF.

Please suggest me correct query for getting only P1 alerts (critical). 

- name: 'team-ops-genie'
  opsgenie_configs:
  - send_resolved: true
    teams: Operations
    tags: '{{ range .Alerts }}{{ .Labels.Values | join "," }}{{end}}'
    priority: '{{ range .Alerts }}{{ if eq .Labels.severity "critical"}}P1{{else}}P3{{end}}{{end}}'

Thanks,
amit

Brian Candler

unread,
Nov 14, 2021, 3:01:00 PM11/14/21
to Prometheus Users
Selecting *which* alerts get sent to team-ops-genie is in the part of the config which you haven't shown - the "routes" section.  You can filter alerts which match particular criteria, such as particular values or patterns of labels.

As for the 422 result, maybe it's to do with the loop in priority.  If there are two alerts in the group it might expand to "P1P1" or "P1P3" for example.

You'll have to decide how to deal with this.  One option is to look at only .Alerts[0], assuming that all alerts in the group have the same severity.  But I think it's cleaner to make separate receivers:

- name: 'team-ops-genie-critical'
  opsgenie_configs:
  - send_resolved: true
    teams: Operations
    priority: P1
- name: 'team-ops-genie-warning'
  opsgenie_configs:
  - send_resolved: true
    teams: Operations
    priority: P3

and use your routing rules to deliver alerts to the appropriate receiver.  (I have left out your 'tags' expression because I don't think that will work well as written when there are multiple alerts in the group)

Amit Das

unread,
Nov 18, 2021, 2:40:56 AM11/18/21
to Prometheus Users
Thanks Brian it works. Need to adjust my routes also. 
Reply all
Reply to author
Forward
0 new messages