Reusable templates in alert rules

[alin.si...@amanaadvisors.com]

Is there any reason why reusable templates (the {{define "myTemplate"}} sort from .lib files) cannot be referenced from alert rules? Would they significantly affect memory usage/processing time or is this simply not deemed particularly useful?

Since Prometheus doesn't support rule templates (where you could e.g. just plug in the job name and alerting threshold instead of copy-pasting the entire rule), I'm trying to at least keep down the verbosity (and improve readability) by having the templating in the description be reasonably concise.

For example, I have an alerting rule that fires when a certain percentage of instances of my job are down. It issues a query to list all affected instances (since they are already aggregated away by the rule.

job:up:sum = sum by (job)(up{environment="production"})
job:up:count = count by (job)(up{environment="production"})

job:up:ratio =
  job:up:sum
    / on (job)
  job:up:count

ALERT TasksMissing
  IF job:up:ratio < 0.9
  FOR 1m
  LABELS { severity="warning" }
  ANNOTATIONS {
    summary = "Tasks missing from {{ $labels.job }}",
    description = "Tasks missing from {{ $labels.job }}: {{ range printf `up{job=\"%s\"}==0` $labels.job | query }} {{ .Labels.instance }} {{ end }}",
}

This is mostly OK as long as I only have one alert. But as soon as I set up a second/third/etc. similar alert (e.g. with a higher severity for a lower fraction of instances running, or for a different job) it becomes very verbose. I would prefer to have a .lib file with a definition

{{ define "enumerateInstancesForJob" }}

  {{ $job := .arg0 }}
  {{ $query := (or .arg1 `up{job=\"%s\"}`) }}

  {{ range $job | printf $query | query }} {{ .Labels.instance }}, {{ end }}

{{ end }}

and reduce the alert definition to

ALERT TasksMissing
  IF job:up:ratio < 0.9
  FOR 1m
  LABELS { severity="warning" }
  ANNOTATIONS {
    summary = "Tasks missing from {{ $labels.job }}",
    description = "Tasks missing from {{ $labels.job }}: {{ template `enumerateInstancesForJob` (args $labels.job `up{job=\"%s\"}==0`) }}",
 }

Thanks,

Alin.

[Brian Brazil]

On 29 June 2017 at 12:47, <alin.si...@amanaadvisors.com> wrote:

Is there any reason why reusable templates (the {{define "myTemplate"}} sort from .lib files) cannot be referenced from alert rules? Would they significantly affect memory usage/processing time or is this simply not deemed particularly useful?

Alert templates are very frequently evaluated and should be used sparingly. Pulling in libraries would likely lead to bloat and performance problems, thus it's not permitted.

 

Since Prometheus doesn't support rule templates (where you could e.g. just plug in the job name and alerting threshold instead of copy-pasting the entire rule), I'm trying to at least keep down the verbosity (and improve readability) by having the templating in the description be reasonably concise.

The recommendation would be to use the templating of a configuration management system to handle this. You could also write one alert that covered all jobs.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6d8a7d39-6841-4dc5-b986-81b548ceb0eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Alin Sinpalean]

Fair enough, thanks for the very quick answer.

Cheers,

Alin.

On Thu, Jun 29, 2017 at 2:03 PM, Brian Brazil <brian....@robustperception.io> wrote:

--

Alin Sinpalean

Amanara Holding AG / Amana Advisors GmbH

Hirschengraben 33

6003 Luzern

Switzerland

alin.si...@amanaadvisors.com

https://www.amanaadvisors.com

This message may contain confidential, proprietary or legally privileged information and is intended only for the use of the addressees named above. No confidentiality or privilege is waived or lost by any mistransmission. If you are not the intended recipient of this message you are hereby notified that you must not use, disseminate, copy in any form or take any action in reliance on it. If you have received this message in error, please delete it and any copies of it and notify the sender immediately.