AlertManager & Prometheus Events API v2 will re-trigger events

[Russ Robinson]

  I have alertmanager configured to send "critical" alerts to Pagerduty over Events v2 api.  If the prometheus rule has an alert that lasts longer than 20 minutes or so; the pagerduty alert will be resolved and then re-triggers a new event.

  I have tried disabling grouping (with "group_by [...]").  The pagerduty alert's log just says: "Resolved through the integration API.".

  However, the alert still shows in Alertmanager.  In addition, I have messages going to slack.  The alert message shows up there; but never a resolved message either.

  Any ideas why alertmanager would close/resolve the pagerduty incident and then re-trigger/open one again?

[Brian Candler]

I think the starting point is to look at your alerting expressions, how they change over time in the PromQL GUI (graph view), and the synthetic metric "ALERTS".

If an alert expression drops out for even a single rule evaluation interval, then the alert is immediately resolved; and then it will re-fire on the next cycle (or after the "for:" period if present)

There is a change in prometheus-2.42.0 which may help address this:

• [FEATURE] Add 'keep_firing_for' field to alerting rules. #11827