Prometheus never stop to fire an alert

[pan...@gmail.com]

Hi 

I try to understand how prometheus firing alert and setup a http proxy between prometheus and alertmanager.

I update rules in prometheus to trigger alert and force prometheus  to reload configuration and so far so good.

When alert reached alertmanager, I update the rule and prometheus reload and alert go away.

However, prometheus never stop post alert to alertmanager

POST /api/v1/alerts HTTP/1.1

Host: alertmanagerHost:9093

User-Agent: Go-http-client/1.1

Content-Length: 526

Content-Type: application/json

Accept-Encoding: gzip

Connection: close

[{"labels":{"alertname":"mqstatistic_high_cpu","monitor":"iib-monitor","severity":"warning"},"annotations":{"description":"MQStatistic CPU usage is 2.898%.","summary":"MQStatistic high CPU usage"},"startsAt":"2017-05-16T01:28:07.803Z","endsAt":"2017-05-16T01:29:22.803Z","generatorURL":"http://c1ed805bbf1e:9090/graph?g0.expr=sum%28rate%28container_cpu_usage_seconds_total%7Bcontainer_label_container_group%3D~%22mq-statistic%22%7D%5B1m%5D%29%29+%2F+count%28node_cpu%7Bmode%3D%22system%22%7D%29+%2A+100+%3E+10\u0026g0.tab=0"}]

Is that normal?

Does prometheus firing alert off when there is no more alert ?

I run prometheus in a container and how could I get prometheus to use proper hostname in the URL without setting hostname in container ?

I use prometheus:v1.6.2 and alertmanager:v0.6.2

Thanks for your attention in advance

[Julius Volz]

I'm not 100% sure if this is your question, but Prometheus will continue sending every firing alert on *every evaluation interval*. It will only stop sending it to Alertmanager once the alert stops firing.

So it's normal that it's sending it "all the time".

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c4a6746d-4166-401f-acb9-7cd53e2b2bb3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[pan...@gmail.com]

But I change the threshold to a lower value and reload. I expected Prometheus to stop sending alert. The only way to stop alert is to restart Prometheus

How people test alert without waiting it happens ?

On Tuesday, May 16, 2017 at 10:24:30 PM UTC+10, Julius Volz wrote:

I'm not 100% sure if this is your question, but Prometheus will continue sending every firing alert on *every evaluation interval*. It will only stop sending it to Alertmanager once the alert stops firing.

So it's normal that it's sending it "all the time".

On Tue, May 16, 2017 at 4:00 AM, <pan...@gmail.com> wrote:

Hi 

I try to understand how prometheus firing alert and setup a http proxy between prometheus and alertmanager.

I update rules in prometheus to trigger alert and force prometheus  to reload configuration and so far so good.

When alert reached alertmanager, I update the rule and prometheus reload and alert go away.

However, prometheus never stop post alert to alertmanager

POST /api/v1/alerts HTTP/1.1

Host: alertmanagerHost:9093

User-Agent: Go-http-client/1.1

Content-Length: 526

Content-Type: application/json

Accept-Encoding: gzip

Connection: close

[{"labels":{"alertname":"mqstatistic_high_cpu","monitor":"iib-monitor","severity":"warning"},"annotations":{"description":"MQStatistic CPU usage is 2.898%.","summary":"MQStatistic high CPU usage"},"startsAt":"2017-05-16T01:28:07.803Z","endsAt":"2017-05-16T01:29:22.803Z","generatorURL":"http://c1ed805bbf1e:9090/graph?g0.expr=sum%28rate%28container_cpu_usage_seconds_total%7Bcontainer_label_container_group%3D~%22mq-statistic%22%7D%5B1m%5D%29%29+%2F+count%28node_cpu%7Bmode%3D%22system%22%7D%29+%2A+100+%3E+10\u0026g0.tab=0"}]

Is that normal?

Does prometheus firing alert off when there is no more alert ?

I run prometheus in a container and how could I get prometheus to use proper hostname in the URL without setting hostname in container ?

I use prometheus:v1.6.2 and alertmanager:v0.6.2

Thanks for your attention in advance

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

The server should send resolved messages for firing alerts for a period of time after the condition is nolonger met.  Maybe this is what you're seeing, not firing, but resolving.

This is defined by the `endsAt` field in the alert post json.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/aa037f40-e275-4f91-82c2-1230f84d1c24%40googlegroups.com.

[pan...@gmail.com]

Thanks
It looks like Prometheus sending resolving for additional 15 mins after alert resolved. Is there any tuning parameter to reduce the window ?

[Brian Brazil]

On 17 May 2017 at 07:28, <pan...@gmail.com> wrote:

Thanks
It looks like Prometheus sending resolving for additional 15 mins after alert resolved. Is there any tuning parameter to reduce the window ?

It's hardcoded, and not something you need to change. Why do you think you want to change this?

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/210ee168-f311-403e-87a0-380a14f48508%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Pan Ho]

It sounds to me it is overkilled after alertmanager acknowledged with success respond. Prometheus still sending  notifications for every 15s for another 15 mins. 

Do you know the reason why?