avg_over_time for sparse metrics

[Alexey Solodkiy]

H! I have statsd_exporter witch provide me honest 95 percentile for every api endpoint in window of 15 sec.

I am also trying to visualize it in this type of chart:

Every block have represent time window with will expense if I zoom out time filter.

I decide to show avg between calculated 15s percentiles for longer $__interval.

I come out with PromQL below, but found out that if for api_object in my $__interval window one point is missing then avg_over_time return null for whole interval.

sum_over_time and quantile_over_time works the same, but max_over_time and min_over_time returns good value in this case.

avg by (api_object) (avg_over_time(vendor_api_request{quantile="0.95", env="$env",  http_method="GET"}[$__interval]))

Could I got avg over time for metric with absent of some values (nulls)?

I am aware of histograms and really interested of avg between percentiles in this case.

[Brian Candler]

The fact you mention $__interval suggests that you're asking a question about Grafana, rather than Prometheus.

Maybe $__rate_interval would help you?

https://grafana.com/blog/2020/09/28/new-in-grafana-7.2-__rate_interval-for-prometheus-rate-queries-that-just-work/

As long as you set the sampling rate correctly in the data source definition, then this will put at least 2 data points in the window, and up to 4 when you zoom in.  Hence if you have one missing point, you'll still get at least one data point for avg_over_time().