How to use the query expression as part of annotations in alert rule

Rajesh Reddy Nachireddi

unread,

May 17, 2020, 5:21:59 AM5/17/20

to Prometheus Users

Hi,

How to use the query expression as part of annotations in alert rule

- alert: OutOfDiskSpace
    expr: node_filesystem_free_bytes / node_filesystem_size_bytes * 100 < 10
    for: 1m
    labels:
      severity: Critical
    annotations:
      description: "Disk has occupied the <exprsession> percentage

Regards,

Rajesh

Julius Volz

unread,

May 17, 2020, 5:55:22 AM5/17/20

to Rajesh Reddy Nachireddi, Prometheus Users

In this case you can just directly refer to the sample value of the time series returned by your expression (since it's already percentage you want), something like this:

description: "Filesystem at {{$labels.mountpoint}} on {{$labels.instance}} is only {{$value}}% free."

For other situations you can also really run another query from the annotation template, see https://prometheus.io/docs/prometheus/latest/configuration/template_examples/#simple-iteration

Btw. you might want to look at node_filesystem_avail_bytes rather than node_filesystem_free_bytes, as free bytes also includes space reserved for the root user, but you may run into problems with other users' disk usage before that reserved space runs full.

Btw. another thing: depending on the use case, consider looking into predictive disk usage alerts rather than alerts with static limits: https://www.robustperception.io/reduce-noise-from-disk-space-alerts

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAEyhnpLmr%2BCwB1GfBS0-3vLV2Zb3LuFnL7pH5McOfGyWbSwi0Q%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

Rajesh Reddy Nachireddi

unread,

May 17, 2020, 6:07:03 AM5/17/20

to Julius Volz, Prometheus Users

Thanks Julius,

we are using the predictive analytics for disk usage and only issue is with resolved alerts doesn't have the current utilisation.

How to handle this?

and also above git link looks too old w.r.t syntax, could you give us latest one which has YAML format.

Thanks,

Rajesh

Stuart Clark

unread,

May 17, 2020, 6:41:31 AM5/17/20

to promethe...@googlegroups.com, Rajesh Reddy Nachireddi, Julius Volz, Prometheus Users

As has been mentioned on this list recently an alert being resolved is due to the alert stopping being sent. There is no specific "alert resolution" message being sent.

The Alertmanager sends the resolution alert and only has access to the data at the time the alert was still firing, so has no idea about the current value.

The recommend would be to adjust the wording on the alert to emphasise that the value is from when it was active.

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Julius Volz

unread,

May 17, 2020, 6:49:18 AM5/17/20

to Rajesh Reddy Nachireddi, Prometheus Users

On Sun, May 17, 2020 at 12:06 PM Rajesh Reddy Nachireddi <rajesh...@gmail.com> wrote:

Thanks Julius,

we are using the predictive analytics for disk usage and only issue is with resolved alerts doesn't have the current utilisation.

How to handle this?

-> what Stuart said

and also above git link looks too old w.r.t syntax, could you give us latest one which has YAML format.

Hmm, I didn't send any git link, and the links I sent were all up-to-date. Not sure what you mean?

Reply all

Reply to author

Forward