Is there any way to update prometheus expr？

[Allenzh li]

Hi, recently, I want to add a label to all alerting expr, is there any way to acheive that? 

[Allenzh li]

I find a method is func ParseExpr(input string) (expr Expr, err error) , but still need to resolve each type and add a matcher to each type.

Does anyone know an easier way？

[Christian Hoffmann]

Hi,

On 12/2/20 4:51 AM, Allenzh li wrote:
> Hi, recently, I want to add a label to all alerting expr, is there any
> way to acheive that?

What exactly are you trying to do? Technically, you can use alert
relabeling to add a static (or at least deterministic) label to each
outgoing alert.
Depending on what you want to accomplish, there may be better ways though.

Kind regards,
Christian

[Allenzh li]

Hi,

Thanks for your reply. 

Exactly, I develop a API which accept prometheus rule from web.

When user create a new rule, I want to add a fixed matcher(xxId="xxx"), which label name is fixed and label value is various. 

eg.

    cpu_usage / avg_over_time(cpu_usage[5m] offset 24h) > 2   ->   cpu_usage{xxId="xxx"} / avg_over_time(cpu_usage{xxId="xxx"}[5m] offset 24h) > 2

Kind regards,
Allenzh li

[Christian Hoffmann]

Hi,

On 12/3/20 9:48 AM, Allenzh li wrote:
> Exactly, I develop a API which accept prometheus rule from web.
> When user create a new rule, I want to add a fixed matcher(xxId="xxx"),
> which label name is fixed and label value is various.
>
> eg.
>     cpu_usage / avg_over_time(cpu_usage[5m] offset 24h) > 2   ->
>  cpu_usage{xxId="xxx"} / avg_over_time(cpu_usage{xxId="xxx"}[5m] offset
> 24h) > 2

So it seems like your web application will only handle the
alert-specific PromQL expression which ends up in the alert's "expr" field.

It seems like it would be straight forward to add the static label with
the dynamic value directly into the Prometheus alert using the labels:
field?

https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/
(Ctrl+F "severity: page")

Kind regards,
Christian

[lizhihua0925]

Hi,

It is not same.

What I want to do is limit the rule to eval metrics which have the spcified label.

Kind regards,
Allenzh li

[Ben Kochie]

You can use regular expression matching or not-equals matching.

some_metric_name{some_label!=""}

This will match any metric where the label is present.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/tencent_C7312B3EC100B4BCD59505CC%40qq.com.

[lizhihua0925]

Hi,

Thanks for your reply.

The problem is how to add the matcher to a exist rule expr.

Eg.

Now the rule expr is :

cpu_usage / avg_over_time(cpu_usage[5m] offset 24h) > 2

Which is desired：

cpu_usage{xxId="xxx"} / avg_over_time(cpu_usage{xxId="xxx"}[5m] offset 24h) > 2

[Christian Hoffmann]

Hi,

On 12/3/20 10:06 AM, lizhihua0925 wrote:
> What I want to do is limit the rule to eval metrics which have the
> spcified label.

Ah, I understood that you wanted to add labels, but you already said
that you wanted to add matchers.

So, just to confirm that I understand correctly:

If the user enters the expression 'foo > 1' it should automatically be
modified to read 'foo{some_static_label="some_dynamic_value"} > 1', right?

So, basically you are looking for a way to programatically modify PromQL
expressions.

This is possible, prefereable by using Go as that's where the reference
PromQL parser happens to be.

That sounds like what I'm doing in the prometheus-filter-proxy except
that I'm modifying in-flight requests while you want to modify PromQL
expressions in alerts. You could still use the same pattern:

* Parse the expression
* Traverse all relevant selectors
* Add the matcher you want
* Convert to string representation again

For reference:
https://github.com/hoffie/prometheus-filter-proxy/blob/master/filter.go#L57

There are other projects which do similar things, e.g. in the
Kubernetes/OpenShift ecosystem. This may provide further inspiration. :)

Kind regards,
Christian

[lizhihua0925]

Yes.

That’s exactly what I was looking for.

Thank you very mach.

Kind regards,
Allenzh li

[b.ca...@pobox.com]

On Thursday, 3 December 2020 at 09:17:02 UTC lizhih...@gmail.com wrote:

The problem is how to add the matcher to a exist rule expr.

Eg.

Now the rule expr is :

cpu_usage / avg_over_time(cpu_usage[5m] offset 24h) > 2

Which is desired：

cpu_usage{xxId="xxx"} / avg_over_time(cpu_usage{xxId="xxx"}[5m] offset 24h) > 2

prom-label-proxy does this sort of label application.  You may be able to steal code from it to do the query rewriting you want.

What's the underlying reason for wanting to do this?  Are you building some sort of multi-tenant setup?  If so, there may be other solutions.  Cortex and VictoriaMetrics have native multi-tenant support; and I think Thanos has some support too.

[lizhihua0925]

Thanks for your reply.

It is great. I will import the enforce code.

https://github.com/prometheus-community/prom-label-proxy/blob/master/injectproxy/enforce.go

Kind regards,
Allenzh li

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/gKPOMotGzwE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c13d2755-ac41-4b31-ae8d-6c4b94c2a56en%40googlegroups.com.