Alerts: check multiple metrics

[s.arl...@gmail.com]

Hi!

Is it possible to write an alert rule that checks 2 or more metrics?

For example:

ALERT alert_name
IF metric1 == 1 & metric2 == 3
FOR 30s
LABELS { severity = "critical" }
ANNOTATIONS {
summary = "blablabla",
description = "blablabla",
}

Thanks!

Regards,
Sergey

[Tobias Schmidt]

Yes, check the logical set operators: https://prometheus.io/docs/querying/operators/#logical/set-binary-operators and read about vector matching.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[s.arl...@gmail.com]

Thank you for the reply!

But I still do not understand it fully :)

The documentation says :

> These logical/set binary operators are only defined between instant vectors:
and (intersection)
or (union)
unless (complement)

and also it says:

> vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.

But I don't need all original elements. Seems what I need is either 'true' or 'false' . Sorry if I'm saying nonsense - I'm new to prometheus. If two or more expressions are true, then send an alert.

For example i have these two expressions:

1. custom_syscollector_number_of_results{ path="/tmp/dir1", instance="inst1:9118" } > 8

2.custom_syscollector_number_of_results{ path="/tmp/testir",instance="inst2:9118"} < 70

I want to send a single alert only when both of these expressions are true.

So what I do:

ALERT alertname
IF custom_syscollector_number_of_results{ path="/tmp/dir1", instance="inst1:9118" } > 8 and custom_syscollector_number_of_results{ path="/tmp/testir",instance="inst2:9118"} < 70
FOR 10s

LABELS { severity = "critical" }
ANNOTATIONS {
summary = "blablabla",
description = "blablabla",
}

But nothing happens.

Can you please provide an example how an alert rule should be written in this case?

Thank you!

Regards,
Sergey

[s.arl...@gmail.com]

So after discussing this matter in prometheus IRC channel and some checking I found out that

Expression 'metric_1 == value1 or metric_2 == value2' is working.

Expression 'metric_1 == value1 and metric_2 == value2' is not.

OR - working, AND - not.

Does anybody have an idea why this can happen?

Thanks!

Regards,
Sergey