Alert Defination - Escalation of Severity

[Wajih Ur Rehman]

Hi,

I have a requirement to evaulate an alert "Alert-A" If the result of expr > 10 , I need to set severity as major and if if it is > 25 then critical.

I have defined it like this:

       - alert: Alert-A
          expr: Some Condition > 10
          for: 1m
          labels:
            severity: major
        - alert: Alert-A
          expr: Some Condition > 25
          for: 1m
          labels:
            severity: critical

Is it correct way to escalate the severity ? or there are other efficient way as well. Please advise.

Regards,

Wajih

[Brian Candler]

Yes, that's fine.  And if you look at the AlertManager examples you'll see how you can suppress the "major" alert when there's also a "critical" alert at the same time.

See:

https://github.com/prometheus/alertmanager/blob/v0.23.0/doc/examples/simple.yml#L86-L98

https://prometheus.io/docs/alerting/latest/configuration/#inhibit_rule

(The two alerts will be treated as different alerts by alertmanager because they have different label sets, even though the alertname is the same).

[Wajih Ur Rehman]

Thank you Brian.