Routes and receivers

[steve waldron]

Is it possible to add a label with a dynamic value? for example - the amount of time an alert has been firing?  if so, any examples of PROMQL queries to accomplish this?

I'm trying to determine if routes can be created using a label defined by length of time an alert has been firing.

Thanks ,

Steve 

[Stuart Clark]

The combinations of labels are what defines an alert, so changing a
label value causes the original alert to stop and a new one to start.
Therefore for dynamic values you would want them as annotations rather
than labels.

[steve waldron]

That makes sense which leads to a couple of other questions:

1. Can an alert rule annotation definition contain a promql query to define? 

2.  if 1. is true, can the annotation label be used in alertmanager config route matching?

Thanks,

Steve

[Julius Volz]

If your goal is to send an alert notification only after it has been firing for some time, the usual way of doing that in Prometheus would be to use the "for" duration field in the alerting rule to define how long the alert has to be active before it transitions from "pending" into "firing" state (at which point it gets sent to AM and routed there).

If you do want to route it before that duration already, but to a different subtree / notifier in your AM routes, you could define the alerting rule twice in Prometheus with differentiating labels and different "for" durations.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/7678533f-7140-4bf6-8fa7-37081ab37ae4n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[steve waldron]

I was looking for a way to escalate the alert after it was firing for a period of time.  Looks like defining the alert twice with different for and label will do the trick.

Thanks for you assistance Julius.  

[Julius Volz]

Great. Other than that, we see escalation as the responsibility of a system after Alertmanager in the chain. For example, PagerDuty (one of the built-in notifier mechanisms in AM) allows setting up both a schedule and an escalation policy for notifications.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/408df294-21c2-4fff-b9e3-3281312fe100n%40googlegroups.com.