Query metric presence/absence based off labels in another metric.

[Jared Bischof]

I have a question about using the labels from one metric to query for the presence/absence of another metric.

For example, I have some nodes that (when they are online) emit this metric:

up{name="node1"} 1

up{name="node2"} 1

At the same time, I am propagating cadvisor metrics from these nodes:

container_last_seen{name="grafana", node="node1"} 1587672314

container_last_seen{name="fluentd", node="node1"} 1587672284

container_last_seen{name="grafana", node="node2"} 1587672251

In this case, I would like to create an alert that triggers because "node2" is up but the container_last_seen metric is absent for grafana on node2. Is this possible to do?

Thanks.

[Jared Bischof]

Meant to say container_last_seen metric is absent for fluentd on node2

[Julien Pivotto]

Can you try something along:

up{job="node"} == 1 unless on (instance) group_right() container_last_seen{name="grafana"}

>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fb59b518-cf29-41c4-a289-b4e9542546e0%40googlegroups.com.


--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Jared Bischof]

It works! Thanks so much! My actual query ended up being:

up{job="cadvisor"} == 1 unless on(node) container_last_seen{name="grafana"}

On Thursday, April 23, 2020 at 3:24:43 PM UTC-5, Julien Pivotto wrote:

On 23 Apr 13:20, Jared Bischof wrote:
> I have a question about using the labels from one metric to query for the
> presence/absence of another metric.
>
> For example, I have some nodes that (when they are online) emit this metric:
>
> up{name="node1"} 1
> up{name="node2"} 1
>
> At the same time, I am propagating cadvisor metrics from these nodes:
>
> container_last_seen{name="grafana", node="node1"} 1587672314
> container_last_seen{name="fluentd", node="node1"} 1587672284
> container_last_seen{name="grafana", node="node2"} 1587672251
>
> In this case, I would like to create an alert that triggers because "node2"
> is up but the container_last_seen metric is absent for grafana on node2. Is
> this possible to do?

Can you try something along:

up{job="node"} == 1 unless on (instance) group_right() container_last_seen{name="grafana"}

>
> Thanks.
>
> --
> You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.