Summaries and Streaming Quantile “Decay”

[Dave Cadwallader]

I understand that the Summary metric type produces a “streaming quantile” which as I understand it is the client-side calculation of that quantile at that point in time. I’ve read some things about such calculations involving the notion of a “decay” which indicates how much time must elapse before an older observation no longer affects a current quantile calculation.

Is there a decay value that’s hardcoded into the official Prometheus client libraries? And for non-official client libraries like the nodeJS library, could decay be handled differently? Any help understanding how this works would be appreciated!

Thanks,
Dave

[Björn Rabenstein]

On 30 March 2018 at 17:55, Dave Cadwallader <dcadwa...@gmail.com> wrote:

Is there a decay value that’s hardcoded into the official Prometheus client libraries?

In Go, the default value is 10m, see [`DevMaxAge`](https://godoc.org/github.com/prometheus/client_golang/prometheus#pkg-constants). You can change it via `MaxAge` in [`SummaryOpts`](https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts).

 

And for non-official client libraries like the nodeJS library, could decay be handled differently?  Any help understanding how this works would be appreciated!

Note that most client libraries don't offer pre-calculated quantiles in summaries at all.

Since the quantiles are entirely calculated on the client side, each library may do it differently. Decay is hard to do. Often, libraries just start over completely every x minutes or so. The Go client library uses five buckets, i.e. it tracks all quantiles five times and swaps in and out a bucket every two minutes (which is all configurable, and also expensive). Shenanigans like that are one of the reasons why histograms are, in general, to be preferred. With histograms, you can select the time range for a quantile calculation at query time. This kind of flexibility of the Prometheus data model also makes it hard to adopt algorithms like q-digest or t-digest.

--

Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg  | HRB 110657B

[Brian Brazil]

On 3 April 2018 at 17:59, 'Björn Rabenstein' via Prometheus Users <promethe...@googlegroups.com> wrote:

On 30 March 2018 at 17:55, Dave Cadwallader <dcadwa...@gmail.com> wrote:

Is there a decay value that’s hardcoded into the official Prometheus client libraries?

In Go, the default value is 10m, see [`DevMaxAge`](https://godoc.org/github.com/prometheus/client_golang/prometheus#pkg-constants). You can change it via `MaxAge` in [`SummaryOpts`](https://godoc.org/github.com/prometheus/client_golang/prometheus#SummaryOpts).

 

And for non-official client libraries like the nodeJS library, could decay be handled differently?  Any help understanding how this works would be appreciated!

Note that most client libraries don't offer pre-calculated quantiles in summaries at all.

Since the quantiles are entirely calculated on the client side, each library may do it differently. Decay is hard to do. Often, libraries just start over completely every x minutes or so. The Go client library uses five buckets, i.e. it tracks all quantiles five times and swaps in and out a bucket every two minutes (which is all configurable, and also expensive).

Consistency is good for users though, so I would encourage following the general approach of the Go library.

Brian

 

Shenanigans like that are one of the reasons why histograms are, in general, to be preferred. With histograms, you can select the time range for a quantile calculation at query time. This kind of flexibility of the Prometheus data model also makes it hard to adopt algorithms like q-digest or t-digest.

--

Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg  | HRB 110657B

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALSNGhnbxFyRuVtfX7a%3DzS7KOhRyUpenx27fFdNK9924H8PH5w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io