Creating a horizontal line based on average of series

[Christopher Dale]

Hi everyone,

I'm trying to create a control chart using data from Prometheus. Eventually I'd like to have control limits and 1/2/3 sigma lines, but as a start I need to figure out how to calculate a single value from a series and construct a horizontal line based on that value. We're using Grafana as our front-end, but I don't think that should matter. To start, I'm trying to create a query that calculates the average of a data series and creates a horizontal straight line from left to right at that calculated average value. I would expect that when the time range of the data I'm looking at changes, the average would then change and the line would move up or down.

For example (pardon the poor quality of the image):

I've tried using avg and avg_over_time but I'm not getting a horizontal line. I'm assuming it's doing a running average over some interval.

Thanks everyone,

Christopher

[Ben Kochie]

This is a feature I've talked with the Grafana team about. I don't remember what the issue number was.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3cdca348-6dea-4b6d-a5f6-490a30bed7f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Christopher Dale]

Hi Ben,

Thanks for the reply! I wonder why it would be necessary to do anything in Grafana though. I think you should be able to do this work in Prometheus alone. That's a pretty naive notion though since I haven't taken the time to look into the code yet. I get this feeling I'm just missing something in the Prometheus documentation or function/operator list that may help achieve the results I'm after. I'm going to have a look at the avg function and see what it's actually doing.

From the documentation, I'd assume avg would return the overall avg of the returned series (a single number), but I'll need to confirm that it's not a running average over some interval (ex. 1m). When I use avg, it definitely looks like a running average.

I thought it might be possible to simply use avg_over_time and passing in a duration that matched the time range being viewed, but that also resulted in a non-linear line.

Thanks for the information, I'll also take a peak at the bug tracker for Grafana and see what I can find. Do you know off hand what wording you used in the issue?

Thanks again Ben!

Christopher

On Thursday, September 14, 2017 at 12:56:10 PM UTC-5, Ben Kochie wrote:

This is a feature I've talked with the Grafana team about. I don't remember what the issue number was.

On Sep 14, 2017 10:45, "Christopher Dale" <somatr...@gmail.com> wrote:

Hi everyone,

I'm trying to create a control chart using data from Prometheus. Eventually I'd like to have control limits and 1/2/3 sigma lines, but as a start I need to figure out how to calculate a single value from a series and construct a horizontal line based on that value. We're using Grafana as our front-end, but I don't think that should matter. To start, I'm trying to create a query that calculates the average of a data series and creates a horizontal straight line from left to right at that calculated average value. I would expect that when the time range of the data I'm looking at changes, the average would then change and the line would move up or down.

For example (pardon the poor quality of the image):

I've tried using avg and avg_over_time but I'm not getting a horizontal line. I'm assuming it's doing a running average over some interval.

Thanks everyone,

Christopher

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.

[Ben Kochie]

You're doing the right thing, with avg_over_time(), or quantile_over_time().  But the trick is how the evaluation is being done, and this is why we need a little bit of help from Grafana.

Say you have a 1 hour window in the graph, and each point on the graph is 1 minute wide.  Say you have a recorded metric that is the 1min rate() like this

instance:http_requests_total:rate5m = rate(http_requests_total[1m])

This means that you'll get a nice graph of the request rate.

Then if you wan the average over the 1 hour time window, you will query something like this:

avg_over_time(instance:http_requests_total:rate5m[1h])

But, the way Prometheus evaluates things, the 1 hour window is evaluated at every step/point of the graph.

With Grafana, there is the concept of a "threshold" line for a graph.  What we want is to add an option to query a single threshold using what Prometheus calls an instant query at the last timestamp of the graph.

This will correctly do what you're looking for.

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e23a685e-bf61-4242-9ef9-24fd47701913%40googlegroups.com.