Alert if two out of three targets in a job are down

[Judah Morrison]

Hey promethues users!

I am trying to send an alert when two out of three blackbox targets do not return probe_success 1.

I have tried several solutions mostly along the lines of: " count by(job)(probe_success{job="blackbox_nodes"}) >= bool 2 " (sourced from: "https://www.robustperception.io/booleans-logic-and-math")

Could someone please help with this?

Thanks

Judah Morrison

[Julien Pivotto]

On 06 May 08:31, Judah Morrison wrote:
> Hey promethues users!
> I am trying to send an alert when two out of three blackbox targets do not
> return probe_success 1.
> I have tried several solutions mostly along the lines of: " count
> by(job)(probe_success{job="blackbox_nodes"}) >= bool 2 " (sourced

You can try sum(probe_success) without(instance) < 2

> from: "https://www.robustperception.io/booleans-logic-and-math")
>
> Could someone please help with this?
>
> Thanks
> Judah Morrison
>

> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/927a73dd-0201-4556-a6e3-52748754278bn%40googlegroups.com.


--
Julien Pivotto
@roidelapluie

[Judah Morrison]

Thanks for the response!!!

The three blackbox targets are in one promethues job "blackbox_nodes". I also have other promethues blackbox jobs/targets I don't want this alert to apply to. Is there a way only have your expression only apply to one job? I tried: "sum(probe_success) without(instance) < 2" and it didn't seem to work. Is that how it would be done? (I am kind of a beginner)

[Judah Morrison]

Sorry, I mean to say:" I tried: " sum(probe_success{job="blackbox_nodes"}) without(instance) < 2" and it didn't seem to work.

[Danny de Waard]

Maybe something like this works for you:
sum(probe_success{instance=~"http://lsrv0008..*"})<2

Op donderdag 6 mei 2021 om 20:47:29 UTC+2 schreef kingoft...@gmail.com:

[Judah Morrison]

Thanks everyone!!!

That worked for me.