Lookback delta of Prometheus

[Rodolphe Ghio]

Hello everyone,

I recently opened an issue on the Prometheus Github asking why Prometheus doesn't manage sleep data longer than 5 minutes. And it was explained to me that it was modifiable using the flag --query.lookback-delta. See the Github issue.
So I was wondering if --query.lookback-delta has a maximum?
Can there be any problems other than the slowness of query loopback for several hours?
Have you ever had problems with changing this flag?

Thank you in advance for your answers

[Julius Volz]

Hi,

There's no absolute maximum to this flag, but the longer you set it, the longer "stale" (no longer scraped series) will be selected by instant vector selectors, and thus still appear in tables, or show up as flat lines in graphs, or end up as a potential double-count in aggregations, etc.

However, after reading your issue, I don't think that setting the lookback delta is what you are really looking for. If I understand you correctly, you are trying to accumulate history in your script, and then send a whole accumulated time range of data into Prometheus in a single scrape? Prometheus does not support this - any scraped endpoint must always only expose a single sample per time series, usually at the current time. Endpoints don't expose any history, only the current value, and the history is then built by Prometheus over multiple scrapes of that current value.

See https://prometheus.io/docs/instrumenting/exposition_formats/#grouping-and-sorting - "Each line must have a unique combination of a metric name and labels. Otherwise, the ingestion behavior is undefined."

In effect, there is no way to buffer up history externally first and then ingest it into Prometheus. In the Prometheus model, it's always the Prometheus server that builds up the history by itself, scrape by scrape. That does mean that it's designed for use cases where occasional gaps in collected data have to be ok. If that is not the case for you, then a different metrics backend than Prometheus may be more suitable.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/a9e129d0-d126-4ab2-bd7b-77b79c5564d0n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com