Using groups in prometheus alerts

[Olaf K]

Hi there,

I have already asked this question on Stackoverflow, but recently found this forum and feel like the question is more suitable here.

I have the following query as part of my prometheus alerting rules in my rules.yml:

...
- alert: Test1 expr: time() - camel_route_last_exchange_completed_timestamp{route="route1|route2|route3...|routex"}
for: 10s ...

In short: I am looking for the time since the last time stamp for specific routes. However, the query is not as important as the part in brackets / bold.

I would prefer to leave out the specific routes and refer to a predefined group (e.g. route_group_A = "route1|route2|route3...|routex"}) so that the query is readable:

time() - camel_route_last_exchange_completed_timestamp{route="route_group_A"}

Is this possible? If so, how and in which document to define this?
I tried using templating, but got no further.

Thanks in advance!

[Matthias Rampke]

This approach is not directly possible in Prometheus itself. Our recommendation for this case is to use some form of external templating to reduce repetition in such cases.

However, there is a way using recording rules. You can add a rule for each route (again, you may want to template that) with the constant value 1:

- record: route_group_1

  expr: 1

  labels:

    route: route1

- record: route_group_1

  expr: 1

  labels:

    route: route2

Then you can reformulate your alert rule to "join" with it:

time() - camel_route_last_exchange_completed_timestamp * on(route) group_left() route_group_1

this works because the label matching drops all metrics that do not have a matching route label on the left and right hand side of the "* on(route)". It does not change the value because we are multiplying by unity.

Whether you consider this more readable is up to you …

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/5859aa5c-63af-4ce0-9c7b-e6e107075996n%40googlegroups.com.

[Olaf K]

Thanks, this is a good workaround!

Do you know if it is also possible to combine them in one label? I tested another way, which did not work so far:

- record: route_group_1

  expr: 1

  labels:

    route: route1|route2

Thanks again!

[Matthias Rampke]

No, this won't work – the | in a label value is not special, amd there are no "multi valued labels" in the data model – the way to have a collection is to have multiple time series with different label values.

/MR

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f1334f4a-822f-40d7-b7b0-d82ff5e6c508n%40googlegroups.com.

[Olaf K]

Thanks again!