Resolved even if not all alerts resolved.

[Sebastian Glock]

Hi,

I have config in alertmanager with 3 levels of alert:

1. fire if  >=70% <80%

2. fire if  >=80 < 90%

3. fire if  >= 90%

and I have send_resolved set on true.

Now my problem is as follows:

When 1st alert is over 90% it sends me "RESOLVED", even if VM is over 91% for ex. CPU.

Is is possible to send RESOLVED if all alerts are RESOLVED? I saw it was future on github alertmanager, but maybe someone know how to do it.

Maybe there is a way to use IF or UNLESS in this alert:

  - alert: CPU load is more than 70%!

    expr: 100 - (avg(irate(windows_cpu_time_total{mode="idle"}[5m])) by (instance)) * 100 >= 70 < 80

    for: 3h

    labels:

      severity: "[Average-CPU]"

    annotations:

      summary: "CPU load is more than 70%!"

      description: "{{ humanize $value }}%"

  - alert: CPU load is more than 80%!

    expr: 100 - (avg(irate(windows_cpu_time_total{mode="idle"}[5m])) by (instance)) * 100 >= 80 < 90

    for: 1h

    labels:

      severity: "[High-CPU]"

    annotations:

      summary: "CPU load is more than 80%!"

      description: "{{ humanize $value }}%"

  - alert: CPU load is more than 90%!

    expr: 100 - (avg(irate(windows_cpu_time_total{mode="idle"}[5m])) by (instance)) * 100 >= 90

    for: 15m

    labels:

      severity: "[Disaster-CPU]"

    annotations:

      summary: "CPU load is more than 90%!"

      description: "{{ humanize $value }}%"

Thanks for all your replies!