How Can I Find Resolved Alert in Prometheus "ALERTS" Query with API

196 views
Skip to first unread message

Muhammed Niyazi Alpay

unread,
May 11, 2022, 4:23:29 AM5/11/22
to Prometheus Users

Hello, I'm using Prometheus API Client and getting alerts for last 24 hours with below code

import json from prometheus_api_client 
import PrometheusConnect from prometheus_api_client.utils 
import parse_datetime from datetime 
import timedelta 

prom = PrometheusConnect(url="***prometheus-url***", disable_ssl=True) 
start_time = parse_datetime("1d") 
end_time = parse_datetime("now") 
chunk_size = timedelta(days=1) 
metric_data = prom.get_metric_range_data( 
 metric_name='ALERTS', 
 start_time=start_time, 
 end_time=end_time, 
 chunk_size=chunk_size
print(json.dumps(metric_data, indent=4))

The alarm list for the last 24 hours comes, but I want to find the resolved alarms in this list. How can I do that?

Bjoern Rabenstein

unread,
May 25, 2022, 12:16:47 PM5/25/22
to Muhammed Niyazi Alpay, Prometheus Users
On 11.05.22 00:46, Muhammed Niyazi Alpay wrote:
>
> The alarm list for the last 24 hours comes, but I want to find the resolved
> alarms in this list. How can I do that?

I'm not quite sure what you mean by "resolved alarms". In Prometheus,
an alert is essentially a vector element returned by a PromQL
expression. And that element is stored in a time series, named
`ALERTS`. Those time series are accessed by your query.

If an alert resolves, it will simply _not_ return a result
anymore. This "nothingness" cannot be stored, so there is no time
series for resolved alerts. You would need to check for `ALERTS`
series that cease to exist to identify resolved alerts.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in
Reply all
Reply to author
Forward
0 new messages