Alerts staying in PENDING forever when "for" is set

1,128 views
Skip to first unread message

keith...@crunchydata.com

unread,
Dec 20, 2018, 10:00:49 AM12/20/18
to Prometheus Users
I've set Prometheus up from scratch several times now using the following configuration files:


I uncomment the section in the prometheus config to use the alertmanager. Whenever I force an alert to go off, they then stay in PENDING mode forever (well, at least several hours in my testing). Trying to figure out the cause, I tried removing the "for:" configuration option in any of the alerts and suddenly everything is working fine. The alert fires and I see it show up in the alertmanager. So no issues of alerts actually going out or communicating with the alert manager.

No matter how low I set the "for:", it sits there PENDING and never firing. I don't recall ever having this issue before, but right now I can only test 2.5.0 and 2.6.0 and it's happening for both. When I have a chance I'll try and see if I can install an older version and see if it's still happening.

I've turned DEBUG logging on and nothing is showing up in the logs. And I mean literally nothing, unless I remove the "for" clause. So not sure where things are getting hung up.

Appreciate any insight on something I may be overlooking.

Keith

Simon Pasquier

unread,
Dec 20, 2018, 10:20:26 AM12/20/18
to keith...@crunchydata.com, Prometheus Users
The alert definitions are buggy because they store the value returned
by the alert query as a label [1].
It means that every alert evaluation generates a new alert. Remove the
alert_value label and it should work fine.

[1] https://github.com/CrunchyData/pgmonitor/blob/9aa42c4634ecc50f313b5b29d06e668808f861e1/prometheus/crunchy-alert-rules.yml#L13
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To post to this group, send email to promethe...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ce4ba9e7-4ef7-4765-b465-c2a32ec01dbe%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

keith...@crunchydata.com

unread,
Dec 20, 2018, 11:00:25 AM12/20/18
to Prometheus Users
Ugh. That was it. Odd that I think this was working before because we're using that "alert" value in Grafana so we can display prometheus alerts in a table and I'd wanted to be able to provide the current value of the alert. Was using the metric 

ALERTS{alertstate="firing"} 

to build that table. Not sure if there's another way to get that value into the table.

Thanks for the help!

l.mi...@gmail.com

unread,
Dec 20, 2018, 11:48:00 AM12/20/18
to Prometheus Users
You can put the value as annotation if you're sending your alerts to slack, pagerduty or email. For grafana you can probably still use the value in the table, but if I recall this depends on how you configure the table (AFAIR there are multiple modes - raw values, aggregation, etc...).

keith...@crunchydata.com

unread,
Dec 20, 2018, 11:51:27 AM12/20/18
to Prometheus Users
Yes, we are already including the value in the email/pagerduty alerts. I'd found a simple way to get the alert status in an easy to parse method for a grafana table with ALERTS{}. Will probably have to get fancier to actually get the alert value in there now somehow.

Thanks!
Reply all
Reply to author
Forward
0 new messages