Print Breaching Threshold in the Description of Alert

[Aditya Nageswar]

Is there a label to print the threshold of the alert in the description.

[Brian Candler]

When you write an alert rule expression like this:

expr: foo > 200

it's just a single promQL query.  Starting with the universe of timeseries available, prometheus filters them down to just those where the metric name is "foo" and the value is > 200.  So the result of the query is something like this:

foo{instance="bar"} 234

foo{instance="baz"} 911

That's what alertmanager receives from the query.  So it can tell you the current value, but the "threshold" doesn't exist as such. When you write more complex expressions, which match against multiple metrics and multiple conditions, you can see there's no single threshold anyway.

What you *can* do is to put your thresholds into a different timeseries.  See this article:

https://www.robustperception.io/using-time-series-as-alert-thresholds

[Christian Hoffmann]

One can then put the threshold into a label or annotation. This way it
becomes part of the alert and is available to the alertmanager
notification templating. This should work with both static thresholds
from the query (which would have to be repeated) and "dynamic"
thresholds based on timeseries according to the above link.

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
https://prometheus.io/docs/prometheus/latest/configuration/template_examples/


Kind regards,
Christian

[Aditya Nageswar]

Hi Christian,

I think you mis-understood the question. 

The breaching threshold used  foralerting is something I want to print in the description or the summary part of the alert

[Christian Hoffmann]

Hi,

On 3/14/20 8:56 PM, Aditya Nageswar wrote:
> I think you mis-understood the question.

Which part do you mean?

> The breaching threshold used  foralerting is something I want to print
> in the description or the summary part of the alert

Yes, you can put this threshold (like a static number, e.g. 200, or a
the value of timeseries such as foo_threshold) into the alert
description or the summary.

Kind regards,
Christian

[Aditya Nageswar]

I want the threshold to be as a variable, I dont want to use it in static way.

FYI, Similar to  {{ $value }} I would like to know if there is something like {{ $threshold }} to print my alert threshold.

[Christian Hoffmann]

Hi,

On 3/14/20 9:13 PM, Aditya Nageswar wrote:
> I want the threshold to be as a variable, I dont want to use it in
> static way.
>
> FYI, Similar to  {{ $value }} I would like to know if there is something
> like {{ $threshold }} to print my alert threshold.

I don't think there is some special $threshold variable or something.
Prometheus does not know what values or metrics you use as thresholds --
it's just an arbitrary PromQL expression to Prometheus.

Therefore, you have to make this explicit. You probably want to have
something like this:

https://prometheus.io/docs/prometheus/latest/configuration/template_examples/#display-one-value

In other words: It won't be a variable such as $threshold, but rather
some {{ query 'foo_metric' }} in your case.

I cannot provide a more specific answer without knowing your actual query.
If you want specific guidance on some alert from your setup, it would be
best if you posted the actual alert definition here.

Kind regards,
Christian