Aggregate metric value over last 7 days

5,934 views
Skip to first unread message

Nick

unread,
Nov 15, 2016, 4:34:25 PM11/15/16
to Prometheus Developers
Hello,

I have a Gauge that tracks number of searches performed per day. Metric’s value is set once a day (somewhere around 2 am). I would like to determine how many searches were performed over the last 7 days.

To achieve this, I tried to perform a query such as ‘/api/v1/query_range?start=(-7 days)&end=today(after the metric was set)&step=86400&query=(metric_name)’, however it results in values captured at midnight on each day.

[1478563200,"2"],[1478649600,"1"],[1478736000,"0"],[1478822400,"0"],[1478908800,"5"],[1478995200,"0"],[1479081600,"0"],[1479168000,"0"]

As a result, it does not include the value that was set today (say at 2:15 AM) because the last datapoint above is upto midnight (Nov 15 00:00). So, I need to perform an additional (instant) query to retrieve the value that was set at 2:15 AM today.

api/v1/query?time= 1479176160&query=(metric_name)

What would be an optimal way to determine how many searches were performed over the last 7 days? Setting the retention to 1D for specific timeseries is not a supported capability, it seems.

Thanks,
Nick

Brian Brazil

unread,
Nov 15, 2016, 5:21:26 PM11/15/16
to Nick, Prometheus Developers
On 15 November 2016 at 21:34, Nick <nikhil.v...@veeva.com> wrote:
Hello,

I have a Gauge that tracks number of searches performed per day. Metric’s value is set once a day (somewhere around 2 am). I would like to determine how many searches were performed over the last 7 days.

This is not how Prometheus is intended to be used. Usually you have many time searies for each of your instances reporting a counter of searches.
 

To achieve this, I tried to perform a query such as ‘/api/v1/query_range?start=(-7 days)&end=today(after the metric was set)&step=86400&query=(metric_name)’, however it results in values captured at midnight on each day.

[1478563200,"2"],[1478649600,"1"],[1478736000,"0"],[1478822400,"0"],[1478908800,"5"],[1478995200,"0"],[1479081600,"0"],[1479168000,"0"]

As a result, it does not include the value that was set today (say at 2:15 AM) because the last datapoint above is upto midnight (Nov 15 00:00). So, I need to perform an additional (instant) query to retrieve the value that was set at 2:15 AM today.

api/v1/query?time= 1479176160&query=(metric_name)

What would be an optimal way to determine how many searches were performed over the last 7 days? Setting the retention to 1D for specific timeseries is not a supported capability, it seems.

You're looking for sum_over_time.

Brian 

Thanks,
Nick

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f8253230-2d17-4297-8430-8019dd1399d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

Ben Kochie

unread,
Nov 15, 2016, 6:29:12 PM11/15/16
to Brian Brazil, Nick, Prometheus Developers
What Brian said, you probably want to consider more directly instrumenting this service.  We can recommend some nice tools to help with this if you give us a better idea of what your setup is like.

Also like Brian said, sum_over_time(metric_name[7d]) should give you what you're looking for.

On Tue, Nov 15, 2016 at 11:21 PM, Brian Brazil <brian....@robustperception.io> wrote:
On 15 November 2016 at 21:34, Nick <nikhil.v...@veeva.com> wrote:
Hello,

I have a Gauge that tracks number of searches performed per day. Metric’s value is set once a day (somewhere around 2 am). I would like to determine how many searches were performed over the last 7 days.

This is not how Prometheus is intended to be used. Usually you have many time searies for each of your instances reporting a counter of searches.
 

To achieve this, I tried to perform a query such as ‘/api/v1/query_range?start=(-7 days)&end=today(after the metric was set)&step=86400&query=(metric_name)’, however it results in values captured at midnight on each day.

[1478563200,"2"],[1478649600,"1"],[1478736000,"0"],[1478822400,"0"],[1478908800,"5"],[1478995200,"0"],[1479081600,"0"],[1479168000,"0"]

As a result, it does not include the value that was set today (say at 2:15 AM) because the last datapoint above is upto midnight (Nov 15 00:00). So, I need to perform an additional (instant) query to retrieve the value that was set at 2:15 AM today.

api/v1/query?time= 1479176160&query=(metric_name)

What would be an optimal way to determine how many searches were performed over the last 7 days? Setting the retention to 1D for specific timeseries is not a supported capability, it seems.

You're looking for sum_over_time.

Brian 

Thanks,
Nick

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsubscri...@googlegroups.com.



--

--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrzOYj3H8KcGM%3De%2BwG0HhKrvPpo7ia2hwKX43ppFgmBTA%40mail.gmail.com.

Nick

unread,
Nov 15, 2016, 8:28:39 PM11/15/16
to Prometheus Developers, brian....@robustperception.io, nikhil.v...@veeva.com
> We can recommend some nice tools to help with this if you give us a better idea of what your setup is like.

In Graphite, I could set the retention on a per metric basis. For daily count metrics it is set to 1 day. A background job fires at say Nov 15 @ 02:00 that aggregates the number of searches performed from Nov 14 00:00 to Nov 15 00:00 and stores them. Everyday this job executes and stores an aggregated count for the last day. Over time, there is 7 days or 30 days or 60 days worth of datapoints. To determine the total number of searches performed in the last 7 days, one would fetch 7 datapoints and aggregate them. Each datapoint represents number of searches performed for that day.

I am trying to do the same with Prometheus, however since Prom features a Pull model, the metric is scraped on Nov 15 at 02:00 for instance. I could get the last day just by issuing an instant query with time set to 02:01. The issue is with getting 7 datapoints for the last 7 days. Since each datapoint could have been set at marginally a different time (02:01, 02:02 etc.), I have overlaps in the datapoints. For instance, Nov 15 uptil 01:59 value was 2, beginning from 02:00 value was 5. I would like Prometheus to return that 5 in this case. With last 7 days case, it is not clear what 'step' should one be using. If I use step=86400 or 1d, we get the datapoint at midnight (00:00) on each of the last 7 days, however that leaves out the datapoint that was set today at 02:00. If I use step=more than one day, then I do get less than 7 datapoints depending on the step.

> Also like Brian said, sum_over_time(metric_name[7d]) should give you what you're looking for.

How does sum_over_time take into account two different values scraped on a day?

00:00 to 01:59, value was 2
01:59 to 23:59, value was 5

Thanks,
Nick

On Tuesday, November 15, 2016 at 6:29:12 PM UTC-5, Ben Kochie wrote:
> What Brian said, you probably want to consider more directly instrumenting this service.  We can recommend some nice tools to help with this if you give us a better idea of what your setup is like.
>
>
> Also like Brian said, sum_over_time(metric_name[7d]) should give you what you're looking for.
>
>
> On Tue, Nov 15, 2016 at 11:21 PM, Brian Brazil <brian....@robustperception.io> wrote:
>
>
>
> On 15 November 2016 at 21:34, Nick <nikhil.v...@veeva.com> wrote:
> Hello,
>
>
>
> I have a Gauge that tracks number of searches performed per day. Metric’s value is set once a day (somewhere around 2 am). I would like to determine how many searches were performed over the last 7 days.
>
>
>
> This is not how Prometheus is intended to be used. Usually you have many time searies for each of your instances reporting a counter of searches.
>  
>
>
> To achieve this, I tried to perform a query such as ‘/api/v1/query_range?start=(-7 days)&end=today(after the metric was set)&step=86400&query=(metric_name)’, however it results in values captured at midnight on each day.
>
>
>
> [1478563200,"2"],[1478649600,"1"],[1478736000,"0"],[1478822400,"0"],[1478908800,"5"],[1478995200,"0"],[1479081600,"0"],[1479168000,"0"]
>
>
>
> As a result, it does not include the value that was set today (say at 2:15 AM) because the last datapoint above is upto midnight (Nov 15 00:00). So, I need to perform an additional (instant) query to retrieve the value that was set at 2:15 AM today.
>
>
>
> api/v1/query?time= 1479176160&query=(metric_name)
>
>
>
> What would be an optimal way to determine how many searches were performed over the last 7 days? Setting the retention to 1D for specific timeseries is not a supported capability, it seems.
>
>
>
> You're looking for sum_over_time.
>
>
> Brian 
>
>
> Thanks,
>
> Nick
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f8253230-2d17-4297-8430-8019dd1399d9%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>
>
> Brian Brazil
> www.robustperception.io
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.

Ben Kochie

unread,
Nov 16, 2016, 3:17:04 AM11/16/16
to Nick, Prometheus Developers, Brian Brazil, Nick
On Wed, Nov 16, 2016 at 2:28 AM, Nick <nikh...@gmail.com> wrote:
> We can recommend some nice tools to help with this if you give us a better idea of what your setup is like.

In Graphite, I could set the retention on a per metric basis. For daily count metrics it is set to 1 day. A background job fires at say Nov 15 @ 02:00 that aggregates the number of searches performed from Nov 14 00:00 to Nov 15 00:00 and stores them. Everyday this job executes and stores an aggregated count for the last day. Over time, there is 7 days or 30 days or 60 days worth of datapoints. To determine the total number of searches performed in the last 7 days, one would fetch 7 datapoints and aggregate them. Each datapoint represents number of searches performed for that day.

This is really just not how we use Prometheus.  It's not a general use database.

I think the trick here is that we just don't need to store these kinds of aggregations.  With the Prometheus query language, you can simply request these kinds of statistics live with no need to worry.  Prometheus is just that fast compared to Graphite.

Say the metric is "searches_total" and it repesents a total of all searches handled by each search server.  You can get a total for the last seven days simply by requesting

sum(increase(searches_total[7d]))

There's almost no need to pre-calculate this.  For example I have a cluster of 70 servers with a similar metric.  Processing 7 days of data takes 450ms.

If I expand this to report the give me the increase for 200+ servers, it still only takes 1 second to calculate.
 
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.
>
> To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f8253230-2d17-4297-8430-8019dd1399d9%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
> --
>
>
> Brian Brazil
> www.robustperception.io
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
>
--
You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-developers+unsub...@googlegroups.com.
To post to this group, send email to prometheus-developers@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/b044e94c-cbb1-441c-b88a-496894c397c4%40googlegroups.com.

Nick

unread,
Nov 17, 2016, 1:32:46 PM11/17/16
to Prometheus Developers, nikh...@gmail.com, brian....@robustperception.io, nikhil.v...@veeva.com
Thanks, Ben!
> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/f8253230-2d17-4297-8430-8019dd1399d9%40googlegroups.com.
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> >
>
> > Brian Brazil
>
> > www.robustperception.io
>
> >
>
> >
>
> >
>
> >
>
> >
>
> > --
>
> >
>
> > You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> >
>
> > To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> >
>
> > To post to this group, send email to prometheus...@googlegroups.com.
>
> >
>
> > To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-developers/CAHJKeLrzOYj3H8KcGM%3De%2BwG0HhKrvPpo7ia2hwKX43ppFgmBTA%40mail.gmail.com.
>
> >
>
> >
>
> >
>
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "Prometheus Developers" group.
>
> To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-devel...@googlegroups.com.
>
> To post to this group, send email to prometheus...@googlegroups.com.
>
Reply all
Reply to author
Forward
0 new messages