Nested range vectors for counters

[roman...@percona.com]

Hi guys,

Is it possible somehow to do something like this?

min_over_time(rate(mysql_global_status_questions{instance="db1"}[2s])[60s])]

Getting - Error executing query: parse error at char 71: range specification must be preceded by a metric selector, but follows a *promql.Call instead

Basically, what I want is to get min rate with per second resolution for the interval of the last 60s (on the graph for each 60s interval).

It allows to spot a problem with metric using lower resolution/interval while still analysing data with higher resolution.

rate() returns value and it can't be used as a range vector for min func.

That's not a problem for gauge metrics but for counters where we have to apply rate() it seems not possible.

Seems a workaround is to create recording rule:

my_new_metric = rate(mysql_global_status_questions[2s])

I will get my “my_new_metric” reflecting the rate.

Then I can easily use min(my_new_metric[$interval]) to graph it.

Any thoughts on this?

Thanks,

Roman

[p...@percona.com]

Hi,

I would describe one use case why I'm looking for something like this in layman terms. 

For MySQL (same should be true for any other database) we have  counter variable "Questions"   through which we can see how many queries it processes every second.

There are number of issue when stalls may happen and in this case Questions go to zero  or very low value for 1 second or so. 

I'm looking to be able to see these events on the graphs even than looking at 1 hour or 24 hour time window.    If you look at such high range you normally would see the data averaged   - 1 min or 5 min per datapoint making such events invisible.

What I would like to be able to get is for each of the grouping interval say 5 min    min/max values of the   1 second rates  (or even histogram distribution)  so such items can be spotted. 

[Björn Rabenstein]

On 23 November 2016 at 21:20, <roman...@percona.com> wrote:

min_over_time(rate(mysql_global_status_questions{instance="db1"}[2s])[60s])]

You have to use a recording rule for the inner range expression. Then you can apply the outer range expression to the recording rule.

See https://github.com/prometheus/prometheus/issues/1227 to track a real solution to the problem.

--

Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales with Company No. 6343600 | Local Branch Office | AG Charlottenburg  | HRB 110657B

[Peter Zaitsev]

Thank you for prompt response!

It is great to hear there is already a case to get it fixed some time.

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/JOVfqQsVRl8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CALSNGhm%3DQYGq-eS1zLe_tqka3tXswCNBMYxM68dTsXTY%2BJBAZw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Peter Zaitsev, CEO, Percona
Tel: +1 888 401 3401 ext 7360   Skype:  peter_zaitsev

[Julius Volz]

Well, the issue has been filed, but whether it's going to be supported is not clear. Really, the issues brought up in this comment would need to be addressed: https://github.com/prometheus/prometheus/issues/1227#issuecomment-157778594

Also, this might encourage people to do incorrect counter/rate math, like:

rate(sum(mycounter)[1m])

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BRUij04sn42kAMz8jcE%2BvqM4wBsn%2BK1Jb9M7OMKnoHF3Px8rw%40mail.gmail.com.

[Peter Zaitsev]

Hi,

Well I think any developed language allows for a wrong usage :) 

Also I think there might be some special syntax added to indicate such function.    This really looks somethat similar from Subqueries in SQL - you can get result of the query to be a table and when you can do another query on it again

[Julius Volz]

On Thu, Nov 24, 2016 at 8:30 PM, Peter Zaitsev <p...@percona.com> wrote:

Hi,

Well I think any developed language allows for a wrong usage :)

Yeah, right now we are in the lucky position though that it's really hard to take the rate of a sum (incorrect) - it's only possible via the indirection of a recording rule :)

 

Also I think there might be some special syntax added to indicate such function.    This really looks somethat similar from Subqueries in SQL - you can get result of the query to be a table and when you can do another query on it again

Yeah, SQL has the benefit that it doesn't have to deal with time intervals.

But yeah, maybe it'd have to look something like [1m:10s] or something, where the 1m is the time window, and the 10s is the time interval step at which to evaluate the inner expression.

 

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CA%2BRUij3228ZzCq6%3DOzwY_NaA7V0POhVfBYdYZ%2BJvku4fK76MvA%40mail.gmail.com.

[Peter Zaitsev]

Hi Julius,

the Interval can be presented in the SQL language too.  I frequently would do something like 

SELECT  (TS DIV 60)*60  AS INTV , AVG(METRIC)  M FROM METRICS  GROUP BY INTV

This funky  (TS DIV 60)*60  gives us even starts for every minute  

Now I can take this table as the  time series table again providing me   Metric  M every minute and i can do 

SELECT  (INTV*3600)*3600 INTV2, MAX(M), AVG(M), MIN(M) FROM ( SELECT  (TS DIV 60)*60  AS INTV , AVG(METRIC)  M FROM METRICS  GROUP BY INTV)  GROUP BY INTV2

I think Prometheus Data Structure of Vector is somewhat similar to the Table in relational database and I think it would be nice to have some form of  syntax for getting result as another vector and processing it again.

Note if you do not want to make syntax very complicated it was already suggested to do it through recording rules.   Perhaps one can consider supporting the "virtual metrics" which are same as recording rules but actually computed as macro rather than had to be defined in advance :)