Showing difference between 2 histograms

212 views

Skip to first unread message

Hozapero

unread,

Feb 11, 2021, 5:56:31 AM2/11/21

to Prometheus Users

We have a histogram metric with lots of "le" bucket. I usually use histogram_quantile to see 50th percentile for this, etc. as such:

histogram_quantile(0.50, sum(rate(amqp_bucket{env="prod"}[$__rate_interval])) by (cluster,le))

We have a few clusters which run the same service with the same metrics. I would like to see a difference comparison (to see how much faster/slower the calls are in an another cluster for the same services).

Is there any way to do difference with histogram_quantiles?
Something like

histogram_quantile(0.50, sum(rate(ampq_bucket{cluster="first"}[15s)) by (cluster,le))
-
histogram_quantile(0.50, sum(rate(ampq_bucket{cluster="second"}[15s)) by (cluster,le))

Thanks.

Aliaksandr Valialkin

unread,

Feb 16, 2021, 5:30:55 PM2/16/21

to Hozapero, Prometheus Users

The difference should work - just remove the cluster from by(...) list:

histogram_quantile(0.50, sum(rate(ampq_bucket{cluster="first"}[15s)) by (le))
-
histogram_quantile(0.50, sum(rate(ampq_bucket{cluster="second"}[15s)) by (le))

See more details about time series matching during binary operator execution in PromQL - https://prometheus.io/docs/prometheus/latest/querying/operators/

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3cd8f5cd-8f71-41f2-9ddf-112df72df115n%40googlegroups.com.