Is there a way to get prometheus to trigger alerts right away?

[Frany Chevry]

Hello guys, 

I'm not sure how to do this with Prometheus. I want prometheus (alert manager) to trigger an alert right away. I don't want it to wait to trigger. I want it to trigger right away after rule has been detected. 

I tried to set it as such:

ALERT DiskUsage

IF [metric]

FOR 1s

LABELS {

}

But that doesn't seem to trigger the alert. As a matter of fact that seem to have slowed things down more. I've also tried to just remove the FOR statement, and that doesn't work either. 

We are scrapping every 5 minutes, and would like this alert to fire right away after the first scrape. What are the options to resolve this issue?

Thank you.

[Ben Kochie]

Scraping every 5 minutes is too slow due to the way stale metrics are handled in 1.x.  Is there a reason you scrape this slowly?

Typically you want to scrape at least every 2 minutes in order to get enough samples for a valid alert expression.

To answer your question, the correct thing to get an immediate alert is to remove the FOR condition.  Any FOR condition will require that the alert fires for two consecutive evaluation intervals.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/136488f3-747f-4a5a-bdbe-bea457458c16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Frany Chevry]

Okay, I'll try removing the FOR condition again and test to see if it works right away. I thought that I tested that, and it didn't trigger right away, but I'll give it another go.

We're scraping every 5 minutes for no reason really. We can change that of course. 

Thank you. 

On Wednesday, October 25, 2017 at 9:38:28 AM UTC-4, Ben Kochie wrote:

Scraping every 5 minutes is too slow due to the way stale metrics are handled in 1.x.  Is there a reason you scrape this slowly?

Typically you want to scrape at least every 2 minutes in order to get enough samples for a valid alert expression.

To answer your question, the correct thing to get an immediate alert is to remove the FOR condition.  Any FOR condition will require that the alert fires for two consecutive evaluation intervals.

On Oct 25, 2017 2:07 PM, "Frany Chevry" <dfet...@gmail.com> wrote:

Hello guys, 

I'm not sure how to do this with Prometheus. I want prometheus (alert manager) to trigger an alert right away. I don't want it to wait to trigger. I want it to trigger right away after rule has been detected. 

I tried to set it as such:

ALERT DiskUsage

IF [metric]

FOR 1s

LABELS {

}

But that doesn't seem to trigger the alert. As a matter of fact that seem to have slowed things down more. I've also tried to just remove the FOR statement, and that doesn't work either. 

We are scrapping every 5 minutes, and would like this alert to fire right away after the first scrape. What are the options to resolve this issue?

Thank you.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

You are likely bumping into the staleness threshold, I suggest you scrape once per minute in order to solve this.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a1595f4c-cc2b-4790-a702-ac2963af7637%40googlegroups.com.

[de...@unnotech.com]

removing/commenting the

FOR

did the trick for me