Summary shows step function pattern

[ray...@gmail.com]

I'm trying summary and histogram to expose a put request duration. 

Buckets for histogram are

  

   buckets := []float64{0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1}

and objectives for summary are

   var quantiles = map[float64]float64{
       0.5: 0.01,
       0.9: 0.01,
       0.99: 0.001,
   }

with default 10m MaxAge. However, the 90 quantile I got from summary by using label

 

   ...{quantile="0.9"} 

have a lot of step function patterns, but the 90 quantile from histogram by using 

    

   histogram_quantile(0.9, ...)

is very smooth. Is there something I can do to make the quantile smoother in summary's case?

[Brian Candler]

I'm not sure that "making the graph smooth" is a good goal in itself; the variability you're seeing could well be a better representation of your input data.

Imagine that in one scrape your request durations samples were:

- 60ms

- 62ms

- 65ms

- 63ms

- 61ms

and in a later scrape they were:

- 93ms

- 85ms

- 87ms

- 89ms

- 92ms

With a histogram, all these points fall into the "50-100ms" bucket.  Both cases will be treated as identical (5 entries in the 50-100ms bucket), and hence all the variation has been lost.  Any function which derives data from the histogram. including quantile estimation, will give the same results for both scrapes.

With a summary, a streaming calculation takes place on the client side to return the 50th, 90th and 99th percentile values, based on the current value and recent history.  These percentiles adapt dynamically based on the input values received.  It can therefore capture those percentiles more accurately, tracking variations in the input data.

If you have a look at the raw logs of the put request durations which are being fed in, it may become clearer what's going on.