Alert Manager $labels.instance is empty

[govinda...@gmail.com]

Hi All,

I am trying to configure slack integration in Alert Manager. I am using the below to rules to trigger the alert.

```

# Alert for any telegraf instance that is unreachable for >5 minutes

ALERT TelegrafDown

  IF count((up{job="telegraf",region="sps-qa"}) == 1) < 5

  FOR 1m

  ANNOTATIONS {

    summary = "Telegraf Down",

    description = "Telegraf on {{ $labels.instance }} has been down for more than 5 minutes"

  }
```

For some reason $labels.instance is empty. Even when the alerts gets triggerd, I don't see any labels. (see attached). 

Please suggest. Thanks.

Thanks,

Govind

[Ben Kochie]

The count() function you are using is dropping all labels.  You don't need count for this alert.

ALERT TelegrafDown

  IF up{job="telegraf",region="sps-qa"} == 0

  FOR 1m

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a45c44c9-ab42-4c00-bd01-6e440531400e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Kochie]

Sorry, I didn't read your alert comment carefully.

This is what you want:

# Alert for any telegraf instance that is unreachable for >5 minutes

ALERT TelegrafDown

  IF up{job="telegraf",region="sps-qa"} == 0

  FOR 5m

[govindaraj]

Thanks Ben. Let me check on that. I wanted to alert only if certain number of instances are down. How can I achieve that? 

--

Thanks & Regards
Govindaraj Venkatesan

[Ben Kochie]

Alerting on IF count(up == 0) > 5 would work, but it won't tell you which instances because by doing the count, you are aggregating away the data.

You can execute queries inside the template syntax, I'd have to think about what that would look like.  There are some examples in the prometheus-users archive with out to do that.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAPnj__hJ0YOKacqGoJA8GRWK_w14mZiCWfPEwf0-k%2BMaSAauBg%40mail.gmail.com.

[govindaraj]

Thanks a lot Ben.

[Ionut Ilie]

we noticed a pattern, if you don't see the label in /alerts endpoint you can't use it. in your attached picture you can only use "alertname" and "monitor".
try different query functions and see what you can export. :)

i think most of our alerts uses

Thanks a lot Ben.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a45c44c9-ab42-4c00-bd01-6e440531400e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks & Regards
Govindaraj Venkatesan

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Brian Brazil]

On 21 June 2017 at 06:46, Ionut Ilie <ionut....@gmail.com> wrote:

we noticed a pattern, if you don't see the label in /alerts endpoint you can't use it. in your attached picture you can only use "alertname" and "monitor".
try different query functions and see what you can export. :)

i think most of our alerts uses

I think your email got send before you were finished.

The more general point is that alerts contain the labels of the expression in use.

If your expression looks like count(x) it'll have no labels, which is why a bare aggregator lacking a by/without clause is to be avoided.

count without(instance)(x) will preserve all labels other than instance, producing more useful alerts.

Brian

 

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a63bfd1b-3d71-4fae-85ca-0e4b0e7d2a70%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io