exporter crashes, multiple up-metric-related alerts fire

[Mario Cornaccini]

hi, we got a custom made exporter, checking for running linux processes..

i got 1 scrape job with 27 targets;
also there is a general alert rule:
expr: up==0

so, when the exporter crashed, we got 27 alerts.

is there a way or best-practice to detect that the exporter crashed, and inhibit the 27 alerts, and just have one alert, exporter crashed ?

(happy to give any more info if needed)

cheers, fil

[Ben Kochie]

This is what the `group_by` feature in the alertmanager routing is for.

https://prometheus.io/docs/alerting/latest/configuration/#route

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b6796581-dcff-4577-a725-97e9e0e23c69n%40googlegroups.com.

[Mario Cornaccini]

of course !
thank you;

totally overlooked that, because at some point we went for group_by: [...], because we didn't like how when 1 alert of a group resolves, that we then get a new notification, that just the remaining firing one is active;
(a matter of taste somehow, i'd say); 'guess we cannot have the cake and eat the cake,so to say ;-)

cheers,fil

[Brian Candler]

Of course, you don't get that problem if you turn off the sending of "resolved" notifications completely.

https://www.robustperception.io/running-into-burning-buildings-because-the-fire-alarm-stopped
https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit

[Conall O'Brien]

On Thu, 25 May 2023 at 08:24, Mario Cornaccini <fris...@gmail.com> wrote:

of course !
thank you;

totally overlooked that, because at some point we went for group_by: [...], because we didn't like how when 1 alert of a group resolves, that we then get a new notification, that just the remaining firing one is active;
(a matter of taste somehow, i'd say); 'guess we cannot have the cake and eat the cake,so to say ;-)

If you need further tuning after setting group_by, I would tweak group_wait value in alertmanager and/or the alert hysteresis set using the for operator of the alerting rule, https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#defining-alerting-rules

 

On Wednesday, May 24, 2023 at 11:51:39 AM UTC+2 Ben Kochie wrote:

This is what the `group_by` feature in the alertmanager routing is for.

https://prometheus.io/docs/alerting/latest/configuration/#route

On Wed, May 24, 2023 at 11:33 AM Mario Cornaccini <fris...@gmail.com> wrote:

hi, we got a custom made exporter, checking for running linux processes..

i got 1 scrape job with 27 targets;
also there is a general alert rule:
expr: up==0

so, when the exporter crashed, we got 27 alerts.

is there a way or best-practice to detect that the exporter crashed, and inhibit the 27 alerts, and just have one alert, exporter crashed ?

(happy to give any more info if needed)

cheers, fil

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b6796581-dcff-4577-a725-97e9e0e23c69n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ec17bd8d-751f-402e-9a0e-ec220884dd06n%40googlegroups.com.

--

Conall O'Brien