Prometheus still firing when alert not active

[Mạnh Nguyễn Tiến]

Hi Group

I am developing an application like Alert Manager. It receives the alert from Prometheus and handler this alert and returns Void with status code 200 OK for Prometheus Server.

But when alert does not fire on Prometheus Dashboard, Prometheus Server still sends the alert to my application.

Please help me. 

Here is my Alert Manager Config on Prometheus

and logs from Prometheus when my application return

[Simon Pasquier]

I'm not sure what you're trying to do exactly. But when an alert resolves, Prometheus keeps sending it to Alertmanager for 15 minutes.

Note that a scrape interval of 5 minutes isn't recommended, the maximum value in practice is 2 minutes to deal with staleness and failed scrapes.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7509e465-9659-429d-9fa2-16697a36e230%40googlegroups.com.

[Mạnh Nguyễn Tiến]

Thanks for your answer

Today, I also find this description of Prometheus keeps sending an alert to AlertManager for 15m (https://utcc.utoronto.ca/~cks/space/blog/sysadmin/PrometheusAlertsClearingTime).

But the article describes that Prometheus will send the alert with "ResolvedAt" on the body, but I cannot found.

Please help me, Why Prometheus doesn't send the message with "ResolvedAt" time on the body when the alert is not firing,

Vào 21:24:34 UTC+7 Thứ Năm, ngày 19 tháng 3 năm 2020, Simon Pasquier đã viết:

I'm not sure what you're trying to do exactly. But when an alert resolves, Prometheus keeps sending it to Alertmanager for 15 minutes.

Note that a scrape interval of 5 minutes isn't recommended, the maximum value in practice is 2 minutes to deal with staleness and failed scrapes.

On Thu, Mar 19, 2020 at 10:59 AM Mạnh Nguyễn Tiến <tienma...@gmail.com> wrote:

Hi Group

I am developing an application like Alert Manager. It receives the alert from Prometheus and handler this alert and returns Void with status code 200 OK for Prometheus Server.

But when alert does not fire on Prometheus Dashboard, Prometheus Server still sends the alert to my application.

Please help me. 

Here is my Alert Manager Config on Prometheus

and logs from Prometheus when my application return

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Brian Candler]

Notice the "Error sending alert ... context deadline exceeded".  Does this happen for every POST?

Maybe prometheus is retrying an old alert over and over again.  I would resolve that issue first.

It seems your application *is* receiving the message - so maybe it's not completing the HTTP exchange correctly (e.g. returning a status code 2xx and an empty response body)