Can I get historic alert in Prometheus?

[Gary Adam]

Can I get historic alert in Prometheus? How?

[Brian Candler]

What do you mean?

1. Create an alert now, on the value of data as it was some time in the past? Sure: use "expr: foo offset 2d > 123", or use a range vector query such as "expr: min_over_time(foo[2d]) < 123"

2. View the values of historic data, that created an alert some time ago? Sure: if the expression is "expr: foo > 123" then just enter "foo" or "foo > 123" into the PromQL browser and you'll see its history.

3. To find out what alerts had been triggered in the past?  Have a look at these auto-generated metrics:

ALERTS

ALERTS_FOR_STATE

Also, if you are scraping alertmanager's own /metrics endpoint, then you'll have various alertmanager_* metrics as well, including

alertmanager_alerts
(which gives the number of alerts firing at a particular time), and various counters for delivery and failure to deliver alerts to different integrations.