derive alert severity from other labels

[Roland Mieslinger]

Hi,

we are using the same set of alert rules for both, our production and qa environment, with the severity label set to a value based on what is appropriate for production.
As a consequence, alert severity is too high for most alerts in our qa environment.

The environment is available as a label form every metric, I could of course duplicate all alert rules, filter by environment label, and set the appropriate severity label; very tedious in the long run, but so far the only solution that came to my mind,

Are there better ways to achieve this, what am I missing?

Something like the ternary operator would be helpful in this case, e. g.:
  labels:

    severity: environment=="qa" ? "warn" : "page"

Alternativly some kind of "functional if " could solve this as well:

  labels:

    severity: iff(environment=="qa", "warn", "page")

note: depending on the implementation this could cause performance issues if the expression engine requires
  evaluation of all parameters passed to the function

[Vu Tuan Dat]

you can try: 

severity: '{{ if eq $labels.environment "qa" }} warn {{ else }} page {{ end }}'

[Brian Brazil]

On Tue, 19 May 2020 at 09:25, Roland Mieslinger <rol...@gmail.com> wrote:

Hi,

we are using the same set of alert rules for both, our production and qa environment, with the severity label set to a value based on what is appropriate for production.
As a consequence, alert severity is too high for most alerts in our qa environment. 

The environment is available as a label form every metric, I could of course duplicate all alert rules, filter by environment label, and set the appropriate severity label; very tedious in the long run, but so far the only solution that came to my mind,

The usual way I'd handle this is via routing alerts differently in the alertmanager for dev/qa environments.

Brian

 

Are there better ways to achieve this, what am I missing?

Something like the ternary operator would be helpful in this case, e. g.:
  labels:

    severity: environment=="qa" ? "warn" : "page"

Alternativly some kind of "functional if " could solve this as well:

  labels:

    severity: iff(environment=="qa", "warn", "page")

note: depending on the implementation this could cause performance issues if the expression engine requires
  evaluation of all parameters passed to the function

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/19c8b790-aa37-435c-a055-a1e41fb6033d%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

[Roland Mieslinger]

Am Dienstag, 19. Mai 2020 10:33:20 UTC+2 schrieb Vu Tuan Dat:

you can try: 

severity: '{{ if eq $labels.environment "qa" }} warn {{ else }} page {{ end }}'

Nice hack, I haven't thought about (ab)using the templating engine for that.

[Roland Mieslinger]

Am Dienstag, 19. Mai 2020 10:46:32 UTC+2 schrieb Brian Brazil:

On Tue, 19 May 2020 at 09:25, Roland Mieslinger <rol...@gmail.com> wrote:

Hi,

we are using the same set of alert rules for both, our production and qa environment, with the severity label set to a value based on what is appropriate for production.
As a consequence, alert severity is too high for most alerts in our qa environment. 

The environment is available as a label form every metric, I could of course duplicate all alert rules, filter by environment label, and set the appropriate severity label; very tedious in the long run, but so far the only solution that came to my mind,

The usual way I'd handle this is via routing alerts differently in the alertmanager for dev/qa environments.

But this would leave the severity at the same level, or am I missing a way to change it this way?

 --

Roland

[Brian Brazil]

It would, however the purpose of severity is to be used for routing in the alertmanager. 

--

Brian Brazil

www.robustperception.io

[Christian Hoffmann]

Hi Roland,

alert_relabel_configs might be another option to override the severity
label of alerts with specific labels (e.g. environment="qa").

I would also go for handling this in alert routing though, I agree.

Kind regards,
Christian