Is there a way to get resolved alerts via the API?

[Basil Lalli]

Is there any API call that can provide information on resolved alerts?  I assumed `api/v2/status` would display resolved alerts for some time period afterwards but that doesn't seem to be the case.

[Bryan Boreham]

You could query the ALERTS time-series.  

[Basil Lalli]

The ALERTS table doesn't show alerts in a resolved state.  They simply disappear from the list.  I realize this is sorta how it works (alerts just disappear)... but alertmanager sends resolved alerts so clearly it's doing the job of noticing an alert is gone, and I was hoping it exposes this in the API somewhere.

[Brian Candler]

AFAIK, alertmanager simply notices that the alert has disappeared, and sends a notification when that happens. That is, it retains some internal state about previously firing alerts (which it needs to for other reasons too, e.g. repeat_interval) and compares current to previous. But once it's gone, it's gone; it doesn't hang around in a "resolved" state.

[Bryan Boreham]

It's not a table, it's a time-series, so you can ask the question "what was there a few minutes ago, and is not there now?"

For example `ALERTS offset 5m unless ALERTS`

[Chris Siebenmann]

In theory Alertmanager keeps resolved alerts around for a generally
short interval, between when Prometheus tells Alertmanager that the
alert is resolved and when the alert group's group_interval allows
Alertmanager to successfully send a notification about it, but I don't
know if Alertmanager shows such alerts in the API. In any case, even if
they briefly show up in Alertmanager's API, as Brian Candler says it's
not a useful way to see what alerts Alertmanager sent 'resolved'
notifications about because they don't stick around for the long term.

(These lingering resolved alerts sometimes can have interesting
interactions with inhibitions and I believe it's one cause of getting a
'alert resolved' notification when you never saw the corresponding
'alert firing'.)

In general if you want to know the history of past Alertmanager
activities, I think you should set up a webhook that gets included in
every notification group and either logs the data or puts it in a
database (this is what we eventually wound up doing). If you just want
what Prometheus alerts were firing, you can get that from Prometheus
with suitable queries, although the queries get tricky if you want to
know things like how many times an alert triggered and was resolved.

- cks

> --
> You received this message because you are subscribed to the Google Groups
> "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to prometheus-use...@googlegroups.com.
> To view this discussion visit
> https://groups.google.com/d/msgid/prometheus-users/2f4d06b6-7028-4781-86ba-4406ca4d7d97n%40googlegroups.com.