function last_over_time

[Julien Pivotto]

Dear all,

We had another request for a promql function, last_over_time.

https://github.com/prometheus/prometheus/issues/7179
https://github.com/prometheus/prometheus/pull/6530
https://github.com/prometheus/prometheus/pull/1139

I would like to see more broadly comments about this - it seems that
there are usescases for this (I needed this - I was lucky to be able to
do it with recording rules - but you can not really do it with
subqueries).

I know that we are careful when adding new promql functions, but this
seems to be a valid one.

Maybe we can revisit the conclusion here and add this promql function?


--
(o- Julien Pivotto
//\ Open-Source Consultant
V_/_ Inuits - https://www.inuits.eu

[Brian Brazil]

On Tue, 28 Apr 2020 at 09:33, Julien Pivotto <roidel...@inuits.eu> wrote:

Dear all,

We had another request for a promql function, last_over_time.

https://github.com/prometheus/prometheus/issues/7179
https://github.com/prometheus/prometheus/pull/6530
https://github.com/prometheus/prometheus/pull/1139

I would like to see more broadly comments about this - it seems that
there are usescases for this (I needed this - I was lucky to be able to
do it with recording rules - but you can not really do it with
subqueries).

Thus far we've had no valid concrete use cases. The requests thus far appear to have been a mix of misunderstanding of PromQL semantics, and attempting to use Prometheus as an event logging system.

What's your use case?

Brian

 

I know that we are careful when adding new promql functions, but this
seems to be a valid one.

Maybe we can revisit the conclusion here and add this promql function?


--
 (o-    Julien Pivotto
 //\    Open-Source Consultant
 V_/_   Inuits - https://www.inuits.eu

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen.

--

Brian Brazil

www.robustperception.io

[Julien Pivotto]

On 28 Apr 09:51, Brian Brazil wrote:
> On Tue, 28 Apr 2020 at 09:33, Julien Pivotto <roidel...@inuits.eu> wrote:
>
> > Dear all,
> >
> > We had another request for a promql function, last_over_time.
> >
> > https://github.com/prometheus/prometheus/issues/7179
> > https://github.com/prometheus/prometheus/pull/6530
> > https://github.com/prometheus/prometheus/pull/1139
> >
> > I would like to see more broadly comments about this - it seems that
> > there are usescases for this (I needed this - I was lucky to be able to
> > do it with recording rules - but you can not really do it with
> > subqueries).
> >
>
> Thus far we've had no valid concrete use cases. The requests thus far
> appear to have been a mix of misunderstanding of PromQL semantics, and
> attempting to use Prometheus as an event logging system.
>
> What's your use case?
>
> Brian

One of our partners have a flaky API ; we don't always get the data we
need, and have gaps of up to 1 hour every now and then.

This is gauges (number of credits we still have) so only the last value
we could scrape matters.

Since it is a third party there isn't much we can do about it. We did
not want to hack the exporter neither. We currently have a recording rule
but that has drawbacks, e.g. upon prometheus restart.

[Brian Brazil]

On Tue, 28 Apr 2020 at 10:06, Julien Pivotto <roidel...@inuits.eu> wrote:

On 28 Apr 09:51, Brian Brazil wrote:
> On Tue, 28 Apr 2020 at 09:33, Julien Pivotto <roidel...@inuits.eu> wrote:
>
> > Dear all,
> >
> > We had another request for a promql function, last_over_time.
> >
> > https://github.com/prometheus/prometheus/issues/7179
> > https://github.com/prometheus/prometheus/pull/6530
> > https://github.com/prometheus/prometheus/pull/1139
> >
> > I would like to see more broadly comments about this - it seems that
> > there are usescases for this (I needed this - I was lucky to be able to
> > do it with recording rules - but you can not really do it with
> > subqueries).
> >
>
> Thus far we've had no valid concrete use cases. The requests thus far
> appear to have been a mix of misunderstanding of PromQL semantics, and
> attempting to use Prometheus as an event logging system.
>
> What's your use case?
>
> Brian

One of our partners have a flaky API ; we don't always get the data we
need, and have gaps of up to 1 hour every now and then.

This is gauges (number of credits we still have) so only the last value
we could scrape matters.

Since it is a third party there isn't much we can do about it. We did
not want to hack the exporter neither. We currently have a recording rule
but that has drawbacks, e.g. upon prometheus restart.

For resource limit issues, I'd expect min/max over time to cover that sufficiently. I'd also argue that if an hour has passed with no new data then either it has already alerted, or it otherwise no longer matters.

The real issue here is the lack of reliability, and I'm not sure if trying to paper over it really helps.

Brian

 

>
>
> >
> > I know that we are careful when adding new promql functions, but this
> > seems to be a valid one.
> >
> > Maybe we can revisit the conclusion here and add this promql function?
> >
> >
> > --
> >  (o-    Julien Pivotto
> >  //\    Open-Source Consultant
> >  V_/_   Inuits - https://www.inuits.eu
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Prometheus Developers" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to prometheus-devel...@googlegroups.com.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msgid/prometheus-developers/20200428083353.GA36342%40oxygen
> > .
> >
>
>
> --
> Brian Brazil
> www.robustperception.io

--
 (o-    Julien Pivotto
 //\    Open-Source Consultant
 V_/_   Inuits - https://www.inuits.eu

--

Brian Brazil

www.robustperception.io

[Julien Pivotto]

min_over_time and the alerts keeps firing '1 hour' after we add new resources.

max_over_time and we are '1 hour' late.

last_over_time would really be the best compromise we have there.

[Brian Brazil]

You could silence the alert, or look at a shorter range for the alert. Predict_linear may also help.

I'm reluctant to add features to PromQL to workaround things that only occur when a target is down for an extended period, as the target being down is a bigger problem that should (in principle) be alerting you anyway.

Brian

--

Brian Brazil

www.robustperception.io