Per-host thresholds in prometheus alert

[Andrew Kulikov]

Hi,

I have, for example, simple alert for low swap:

name: Low swap
expr: swap_used_percent > 80
for: 10m
labels:
    severity: warning
annotations:
    description: Swap usage is {{ $value | printf "%.2f" }}% on {{ $labels.host }}
    summary: Low swap

For some hosts it's normal to use 50% of swap, for other hosts -- 90% of swap. And I would like to define different thresholds for different hosts. Can I define per-host threshold for swap usage write alert expression like this?

expr: swap_used_percent > {{ $swap_threshold }}

Thanks a lot!

[David Leadbeater]

See 

https://www.robustperception.io/using-time-series-as-alert-thresholds

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/13a55334-4548-4e03-b78e-56154fb6b4e1n%40googlegroups.com.

[Stuart Clark]

On 26/02/2021 14:18, David Leadbeater wrote:

See 

https://www.robustperception.io/using-time-series-as-alert-thresholds

On Fri, 26 Feb 2021 at 14:16, Andrew Kulikov <andrew....@gmail.com> wrote:

Hi,

I have, for example, simple alert for low swap:

name: Low swap
expr: swap_used_percent > 80
for: 10m
labels:
    severity: warning
annotations:
    description: Swap usage is {{ $value | printf "%.2f" }}% on {{ $labels.host }}
    summary: Low swap

For some hosts it's normal to use 50% of swap, for other hosts -- 90% of swap. And I would like to define different thresholds for different hosts. Can I define per-host threshold for swap usage write alert expression like this?

expr: swap_used_percent > {{ $swap_threshold }}

Alternatively instead of comparing against a fixed threshold (which needs periodic checks & adjustment) you can use the predict_linear function (https://prometheus.io/docs/prometheus/latest/querying/functions/#predict_linear) to project how long until all swap is used based on recent usage increases.

-- 
Stuart Clark