A query on rate

[Manish G]

Hi All,

rate function handles restarts if metrics is of counter type, as documentation tells.

But how is it of help in something like kubernetes environment. For eg, we have application running on several pods and metrics scraped by prometheus has pod id as one of the label. So if a pod goes away and replaced by a new one, then we have 2 time series here, both treated separately.

So I am not sure how does rate function is of help here?

I hope I am able to convey my point.

[Julius Volz]

Hi Manish,

Rates coming from two different pods will be separate time series, as you said. If you apply the rate() function to them, then the rate of increase will be calculated separately for each of the series. You can then choose to sum up those multiple rates to get a total rate of increase over all pods (e.g. "sum(rate(my_counter[5m]))").

When pods are coming and going, there are some sources of uncertainty introduced: Between Prometheus' last scrape of a pod and that pod being terminated, the pod may get some additional counter increments. These extra increments will be "lost" to Prometheus forever and there's no way to recover them, because they died with your pod. Similarly, when a new pod first gets scraped, it may already have seen some counter increments before that first scrape. But these effects are usually small unless the rate of pod terminations + creations comes close to the magnitude of your scrape interval. Usually you should scrape often enough to get most counter increments.

Further, when running the rate() function over a series, Prometheus will apply a heuristic to detect whether a series starts or ends within the provided rate window (see https://github.com/prometheus/prometheus/blob/e483cff61ff1b68e8925db1059393c4d36af9df5/promql/functions.go#L107-L110 for more details). If rate() thinks that a series starts *before* the window / continues *after* the window, rate() extrapolates the calculated slope toward the respective window boundary, becauses it assumes that the counter behavior will be roughly similar during that extrapolated time. Otherwise, if rate() thinks that a series starts or ends under the window, it does not do this extrapolation, and thus returns a lower value. This behavior tries to do the right thing on average, but will never be fully precise.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAOFd6ig%2BjkVGOkXse4JuP%3DTFhszOHGOyMw-LQgYgNF8dhiEn8w%40mail.gmail.com.

--

Julius Volz

PromLabs - promlabs.com

[Julius Volz]

I meant, "Rates coming from two different pods" -> "Counters coming from two different pods"