How to solve data resolution problem when using Prometheus irate() function in the Grafana?

[abc...@gmail.com]

My exporter scrapes metrics every 5 seconds. I set the range-vector of the query to 10 seconds, which is 2x of scrape interval, in order to catch 'spikes and dips',

The problem is Grafana automatically controls the resolution(step) of the graph, and I'm losing some data points when I set a long time range.

Trivial solution is using '$__interval' or '$__rate_interval' variable instead of constant range vector.

But if I use variable for range-vector, I will lose many data points when I monitor a long time range graph. Because irate() considers only the last two samples.

Is it okay to use irate() in Grafana? Or is there any better solution?

[Brian Candler]

You should use rate() instead of irate(), and then if you are using grafana 7.2 or later:

* In your data source definition, set the scrape interval to 5 seconds

* Use rate(foo[$__rate_interval]) in your queries

However, it's slightly broken.  What you'll actually get is a correct foo[...] range at large zoom, but at small zoom it won't go any smaller than foo[20s], which means it smears together three time intervals (i.e. it shows the average rate over 15 seconds).

As a workaround, you can set the data source scrape interval to 2.5 seconds if it accepts fractions, or 3 seconds if it doesn't.

More details here: https://github.com/grafana/grafana/issues/32169

This is considered a "working as intended, we won't fix" by Grafana.