OOM / Crash after using histogram_quantile

[Dave Cadwallader]

Using Prometheus v2.1.0

I'm capturing a histogram of api response times.  

My Goal: To show a Grafana Singlestat panel with the 90th percentile response time for the past 24 hours.

I put this into the Grafana query, starting with a low value and increasing it to see how long query time took:

histogram_quantile(0.90, sum(rate(api_latency_milliseconds_histogram_bucket[5m])) by (le))

Responded quickly.

histogram_quantile(0.90, sum(rate(api_latency_milliseconds_histogram_bucket[1h])) by (le))

Responded after a few sec.

histogram_quantile(0.90, sum(rate(api_latency_milliseconds_histogram_bucket[24h])) by (le))

Crashed Prometheus with OOM error.  Here's what node_exporter reported at the time:

Am I using this function improperly for my intended goal?  I know that recording rules can be used for speeding up graphs, but I know also that I can't aggregate an aggregate, so I don't think a recording rule could help me here.

Any ideas what I might be doing wrong?

Thanks

Dave

[Chris Marchbanks]

Hello Dave,

When you are made the single stat graph in grafana, did you toggle on the "instant" checkbox? I have found that queries like yours with long rate intervals can take a very long time when doing range queries.

Hope this helps,

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8be9c065-83ba-4a6c-b21c-7ef844b4e3a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Chris Marchbanks | Engineer

FreshTracks.io - Intelligent Alerting for Kubernetes and Prometheus