max and min values of a Prometheus metric in Grafana change when zooming

[Ariel Abecasis]

Hi,

I'm displaying in Grafana the outgoing data rate of a particular network interface, which is sampled every 2 minutes, and I configured the graph legend to display the maximum value of the metric in the current time range. My problem is that the displayed maximum value changes when I zoom in or out, even though the actual maximum remains the same.

The attached picture shows an example of this: on the left we can see a sharp peak that occurred at a particular time; on the right we see the same graph after I zoomed out, and the peak has disappeared because it was too short-lived (on the left we have a new pixel every minute, on the right we have one every 4 minutes). Now, if I display a graph of the last 24 hours, which will be similar to the one on the right, and I look for the max value in the legend, I won't see the correct value which is 49.27 Gbps because of the resolution of the graph; I'd like to have a way to display that value while also displaying the 24-hour graph.

I thought this was a Grafana issue, but Torkel Ödegaard, the developer of Grafana, insists that Prometheus uses some kind of aggregation when zooming out because otherwise Grafana would be overwhelmed by datapoints, and it is this aggregation that makes the maximum value disappear.

Is there a way to display the true maximum of a metric in any time range, short or long?

[Brian Brazil]

There is no aggregation going on. What happening is that Grafana isn't requesting data for the time period in which the peak happened. You need to increase the resolution that Grafana is requesting data at, or look into max_over_time.

Brian

 

Is there a way to display the true maximum of a metric in any time range, short or long?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8c50cde6-5560-4338-9feb-fc0128258b46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[Ariel Abecasis]

Thanks a lot for your lightning-fast reply. max_over_time is a pretty good workaround, although I'll miss the flexibility of moving and zooming freely around the graph and still getting the relevant max, min, avg and 95 percentile. Torkel's answer suggests that increasing Grafana's data-requesting resolution is not an option, since getting a whole month's worth of datapoints at 2-min resolution would surely overwhelm it.

[Brian Brazil]

On 5 December 2016 at 13:14, Ariel Abecasis <ariel.a...@gmail.com> wrote:

Thanks a lot for your lightning-fast reply. max_over_time is a pretty good workaround, although I'll miss the flexibility of moving and zooming freely around the graph and still getting the relevant max, min, avg and 95 percentile. Torkel's answer suggests that increasing Grafana's data-requesting resolution is not an option, since getting a whole month's worth of datapoints at 2-min resolution would surely overwhelm it.

Prometheus also has a safety check that prohibits requesting more than 11k steps of data.

Brian

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/205a2d95-22b9-482a-8440-93fd180b36fa%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io