How to override Alert?

[Alexander Ryabov]

We're using two groups of Alerts 

- common for all teams

- per-team only

In prometheus config it looks like this:

rule_files:

  - /git-sync/common/*.yaml

  - /git-sync/rules/*.yaml

And directory git-sync comes from team repo. In this repo folder /common/ comes from git submodule.

This mostly works fine, but there are corner cases when team want to override some common Alert rule on their level.

Currently it is not possible, consider this case:

# /git-sync/common/prom.yaml
groups:
- name: prom
  rules:
  - alert: InstanceDown
    expr: up == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Instance scrape failed

# /git-sync/rules/prom.yaml
groups:
- name: prom
  rules:
  - alert: InstanceDown
    expr: up{group!="maintenance"} == 0
    for: 5m
    labels:
      severity: critical
    annotations:
      summary: Instance scrape failed

This still loads as 2 different alerts in Prometheus, and first one continues to fire.

Is it possible to override alerts?

Shouldn't alert names be unique?

[Julius Volz]

Alert names don't have to be unique, in fact it can be quite common to use the same alert name multiple times (for different severity levels, for example).

One way to make per-team customizations to generic alerting rules is outlined in https://www.robustperception.io/using-time-series-as-alert-thresholds. But in your case where a team may want to make completely generic rule expression overrides (that aren't expressed as simple threshold values or similar), you would probably have to solve it outside Prometheus somehow with some templating system and render the final rule files accordingly.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d4532c99-835e-4c0c-b145-9e662a965f2f%40googlegroups.com.

[Jonas Kongslund]

I haven't tried using it yet but Labyrinth Labs has made a preprocessor that makes it possible to override alerts.

See https://www.lablabs.io/2020/04/19/how-we-solved-our-need-to-override-prometheus-alerts/

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.