Feature request: Setting a custom lookback delta for a query

[Vilius Pranckaitis]

TL;DR: ability to set a custom lookback delta for a query would be useful when time series have drastically different resolutions.

---

Currently, Prometheus allows configuring lookback delta through configuration flag. This works well if the resolutions/scrape intervals of time series are similar, but is inconvenient if they differ a lot. E.g. if we had two time series, one with a datapoint each hour and another one with a datapoint each minute, a lookback of 2h would work for the first one, but would be way too big for the second one.

This impacts looking at downsampled data greater than 5 minutes (the default lookback period) which both Thanos and M3 frequently keep (depending on user configuration). Both support downsampling data to user configured resolutions, thus lookback delta should be selected according to what resolution is being queried when querying downsampled data. Both Thanos and M3 depend on Prometheus and use `promql.Engine` for executing queries, which doesn't allow setting custom lookback for a particular query.

While I understand that this might be more relevant for M3 and Thanos than for Prometheus, just wanted to check what the maintainers of Prometheus think about this. If this seems useful, someone from M3 should be able to submit a PR (we have already looked at the code and it doesn't look like it needs huge changes).

[Julien Pivotto]

We are willing to have downsampling in Prometheus too in the future.

I would merge that pull request.

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/84c03b13-25d7-479c-b678-44cd8625f2c0n%40googlegroups.com.

[Ben Kochie]

Thanos automatically deals with downsample lookback with no changes. This was fixed at least 2 years ago.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAFJ6V0p-_k_Yjy_gUdvHaogxZz%3D7DeNEMM7zz7uBYD%2BYBA%2B6bw%40mail.gmail.com.

[Vilius Pranckaitis]

> Thanos automatically deals with downsample lookback with no changes.

That's true. We've at M3 have done the same. However, the resulting solutions are not very clean and involve trade-offs (Thanos seem to initialize up to three engine with increasing lookbacks; in M3 we pre-initialize a few engines based on configured resolutions and in any other case create them on the fly).

From what we've discussed among M3 maintainers and from TODOs in Thanos codebase, both seem to agree that it would be more convenient if dynamic lookback was supported by Prometheus.

> We are willing to have downsampling in Prometheus too in the future.

>

> I would merge that pull request.

I'll look into submitting a draft PR, just so it would be easier to see what changes would be needed.

[Bjoern Rabenstein]

As a short- to mid-term remedy, I like the idea of a lookback-delta
per query.

Long-term, I would prefer if we could get rid of the lookback delta
altogether. We want richer metadata anyway, and part of it could be
when a series starts and ends and what the configured scrape interval
was. Once we know that, we can create query strategies so that we can
always return the most recent valid sample (if there is any) without
generally killing query performance.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Darshan Chaudhary]

(slightly tangential)

> We want richer metadata anyway,

+1, richer metadata about the data in tsdb will be required for the query optimizer too. IMO without the metadata, the optimizer cannot choose the best query plan (order of evaluating the query).

[Vilius Pranckaitis]

Made a draft PR: https://github.com/prometheus/prometheus/pull/9946