A query to find a burst?

[Victor Sudakov]

Colleagues,

If max_over_time(foo[2d]) returns N, how can I find the exact timestamp(s) in the past when foo=N?

In other words, if there have been very short bursts, how do I find the exact time of those bursts with a PromQL query?

--
Victor Sudakov VAS4-RIPE
http://vas.tomsk.ru/
2:5005/49@fidonet

[Brian Candler]

You can send the query "foo[2d]" and then filter the results in the client, to just those points where the value is N.

This is a use case where it would be nice to be able to build a range vector directly out of a simple instant vector expression, i.e. "(foo == N)[2d]".  However that isn't allowed.

A subquery doesn't cut it here, because it resamples the data.  The subquery "(foo == N)[2d:1s]" gives an approximation, but for a given point you'll see multiple points at 1 second intervals (until the time where foo != N)

[David Leadbeater]

On Sun, 2 Jan 2022 at 20:43, Brian Candler <b.ca...@pobox.com> wrote:
[...]

> A subquery doesn't cut it here, because it resamples the data. The subquery "(foo == N)[2d:1s]" gives an approximation, but for a given point you'll see multiple points at 1 second intervals (until the time where foo != N)

If you don't mind the resampling then you can usefully make use of the
experimental '@' modifier[1] to avoid needing to query for N first:

max_over_time(foo[2d] @ end()) == foo

For example, that is easily graphable like so:
https://prometheus.demo.do.prometheus.io/graph?g0.expr=max_over_time(node_filefd_allocated%5B1d%5D%20%40%20end())%20%3D%3D%20node_filefd_allocated&g0.tab=0&g0.stacked=0&g0.range_input=1d&g0.step_input=10

(But note I had to put the step at 10s; the range vector can find
values that your step may otherwise skip over.)

[1]: https://prometheus.io/docs/prometheus/latest/querying/basics/#modifier

[Victor Sudakov]

Brian Candler wrote:
> You can send the query "foo[2d]" and then filter the results in the client,
> to just those points where the value is N.

Indeed, in the Prometheus Web UI I can use ^F in the browser to look
for N. Thank you for the hint. The problem is not to overwhelm the
browser with data.

> This is a use case where it would be nice to be able to build a range
> vector directly out of a simple instant vector expression, i.e. "(foo ==
> N)[2d]". However that isn't allowed.
>
> A subquery doesn't cut it here, because it resamples the data. The
> subquery "(foo == N)[2d:1s]" gives an approximation, but for a given point
> you'll see multiple points at 1 second intervals (until the time where foo
> != N)

Isn't "(foo == N)[2d:]" what I'm looking for? I don't quite grok
subqueries, but the resolution parameter seems to be optional. At
least "(foo == N)[2d:]" seems to show the timestamps I was looking
for.

[Brian Candler]

On Monday, 3 January 2022 at 03:28:37 UTC Victor Sudakov wrote:

Isn't "(foo == N)[2d:]" what I'm looking for? I don't quite grok
subqueries, but the resolution parameter seems to be optional. At
least "(foo == N)[2d:]" seems to show the timestamps I was looking
for.

As it says here: 

"<resolution> is optional. Default is the global evaluation interval."

So if your global evaluation interval is 1m, then that expression is the same as (foo == N)[2d:1m]

What this does is evaluate the expression foo == N at the current time T, at time T-1m, at time T-2m etc.  In the results, this won't give you the *exact* time that the data point occurred: it will give you a timestamp of T-Nm, which will be up to 1 minute after the timestamp of the point itself.  (The value of a timeseries at time T is the value of the most recent data point on or before time T).

Also, individual scrape jobs can use different scrape intervals.  If you have a global eval interval of 1 minute but this particular scrape job uses 15s, then the above expression will return (on average) 1 in every 4 data points.

[Victor Sudakov]

Brian Candler wrote:
>
> > Isn't "(foo == N)[2d:]" what I'm looking for? I don't quite grok
> > subqueries, but the resolution parameter seems to be optional. At
> > least "(foo == N)[2d:]" seems to show the timestamps I was looking
> > for.
> >
>
> As it says here

> <https://prometheus.io/docs/prometheus/latest/querying/basics/#subquery>:

> "<resolution> is optional. Default is the global evaluation interval."
>
> So if your global evaluation interval is 1m, then that expression is the
> same as (foo == N)[2d:1m]

Hello Brian! I don't quite understand why "(foo == N)[2d:1m]" or even
"(foo == N)[2d:]" is allowed while "(foo == N)[2d]" is not?

>
> What this does is evaluate the expression foo == N at the current time T,
> at time T-1m, at time T-2m etc. In the results, this won't give you the
> *exact* time that the data point occurred: it will give you a timestamp of
> T-Nm, which will be up to 1 minute after the timestamp of the point
> itself. (The value of a timeseries at time T is the value of the most
> recent data point on or before time T).

Sounds fine with me if it does not skip/hide peaks but shows the time
nearest to the peak. Does it?

>
> Also, individual scrape jobs can use different scrape intervals. If you
> have a global eval interval of 1 minute but this particular scrape job uses
> 15s, then the above expression will return (on average) 1 in every 4 data
> points.

I have 15s across all my prometheus instances as I've read somewhere
that it is the best practice to have a unified scrape interval
everywhere.

[Brian Candler]

On Tuesday, 4 January 2022 at 08:07:14 UTC Victor Sudakov wrote:

Hello Brian! I don't quite understand why "(foo == N)[2d:1m]" or even
"(foo == N)[2d:]" is allowed while "(foo == N)[2d]" is not?

foo[2d] is a range vector.  It gives you all the individual timestamped data points belonging to all timeseries for metric "foo" within a time period from evaluation time T to T-2d.

However, range vectors can *only* be applied to pure metrics, not to expressions.  "foo == N" is an expression which generates an instant vector at some evaluation time T.

The reason for this limitation becomes clear when you consider expressions which calculate across multiple timeseries, such as

    sum(foo)

or

   foo / bar

Metrics "foo" and "bar" compromise multiple timeseries, identified by different label sets.  However within each timeseries, the data points have their own unique timestamps: the data points in foo{bar="a"} were not necessarily scraped at the same time as foo{bar="b"}.

Therefore, the only possible way to do arithmetic across timeseries is to pick some arbitrary evaluation time T, take the value of those timeseries at that same point T, and give the result timestamped with T.  A subquery lets you repeat that across a time window: it scans across the window at intervals of some step S, repeating the calculation at those times.

What is the value of a timeseries at time T, given that it may not have a data point at exactly T? It's the value of the most recent data point on *or before* time T, looking back no more than the staleness window (by default 5 minutes)

>
> What this does is evaluate the expression foo == N at the current time T,
> at time T-1m, at time T-2m etc. In the results, this won't give you the
> *exact* time that the data point occurred: it will give you a timestamp of
> T-Nm, which will be up to 1 minute after the timestamp of the point
> itself. (The value of a timeseries at time T is the value of the most
> recent data point on or before time T).

Sounds fine with me if it does not skip/hide peaks but shows the time
nearest to the peak. Does it?

It won't be the time "nearest" the peak, but the first sampling time *after* the peak.  That is, if you have a 15 second step in your subquery, and a 15 second sampling interval, then the timestamp could be up to 14.99 seconds after the event.

You can prove this to yourself by comparing the timestamps of

foo[2d]

foo[2d:15s]

Look for the corresponding peaks / data points.

[Brian Candler]

Here's an example (I'm using 60 second scrape interval)

up{instance="localhost:9090"}[5m]

=>
1 @1641284315.945 1 @1641284375.945 1 @1641284435.945 1 @1641284495.945 1 @1641284555.945

up{instance="localhost:9090"}[5m:1m]

=>

1 @1641284340 1 @1641284400 1 @1641284460 1 @1641284520 1 @1641284580

Note also how the subquery sampling points are rounded to the interval, i.e. here they are at exact multiples of 60 seconds, and don't change until you move into the next 1m.

[Victor Sudakov]

Brian Candler wrote:
>
> > Hello Brian! I don't quite understand why "(foo == N)[2d:1m]" or even
> > "(foo == N)[2d:]" is allowed while "(foo == N)[2d]" is not?
>
>
> foo[2d] is a range vector. It gives you all the individual timestamped
> data points belonging to all timeseries for metric "foo" within a time
> period from evaluation time T to T-2d.
>
> However, range vectors can *only* be applied to pure metrics, not to
> expressions. "foo == N" is an expression which generates an instant vector
> at some evaluation time T.
>
> The reason for this limitation becomes clear when you consider expressions
> which calculate across multiple timeseries, such as
> sum(foo)
> or
> foo / bar
>
> Metrics "foo" and "bar" compromise multiple timeseries, identified by
> different label sets. However within each timeseries, the data points have
> their own unique timestamps: the data points in foo{bar="a"} were not
> necessarily scraped at the same time as foo{bar="b"}.

You probably meant "comprise" ?

>
> Therefore, the only possible way to do arithmetic across timeseries is to
> pick some arbitrary evaluation time T, take the value of those timeseries
> at that same point T, and give the result timestamped with T. A subquery
> lets you repeat that across a time window: it scans across the window at
> intervals of some step S, repeating the calculation at those times.
>
> What is the value of a timeseries at time T, given that it may not have a
> data point at exactly T? It's the value of the most recent data point on
> *or before* time T, looking back no more than the staleness window (by
> default 5 minutes)

Thank you, this was very educational albeit a bit difficult to grasp.

> > > What this does is evaluate the expression foo == N at the current time
> > T,
> > > at time T-1m, at time T-2m etc. In the results, this won't give you the
> > > *exact* time that the data point occurred: it will give you a timestamp
> > of
> > > T-Nm, which will be up to 1 minute after the timestamp of the point
> > > itself. (The value of a timeseries at time T is the value of the most
> > > recent data point on or before time T).
> >
> > Sounds fine with me if it does not skip/hide peaks but shows the time
> > nearest to the peak. Does it?
>
>
> It won't be the time "nearest" the peak, but the first sampling time
> *after* the peak. That is, if you have a 15 second step in your subquery,
> and a 15 second sampling interval, then the timestamp could be up to 14.99
> seconds after the event.
>
> You can prove this to yourself by comparing the timestamps of
>
> foo[2d]
> foo[2d:15s]
>
> Look for the corresponding peaks / data points.

I've noticed that if I take a larger resampling interval, like
"foo[2d:1h]", I lose all my peaks. Which is kind of understandable now
but the question "how to better find peaks" kind of remains.

[Brian Candler]

On Tuesday, 4 January 2022 at 09:34:41 UTC Victor Sudakov wrote:

You probably meant "comprise" ?

Yes.  Disconnect between brain and keyboard!

 

I've noticed that if I take a larger resampling interval, like
"foo[2d:1h]", I lose all my peaks. Which is kind of understandable now
but the question "how to better find peaks" kind of remains.

(foo == N)[2d:15s] will find the peaks, with approximate timestamps within 15 seconds of the actual time the data was sampled.

If you want, you can then hit the API with additional queries

foo[15s] @timestamp

to get the raw metrics with exact timestamps (it will return the raw timeseries between timestamp-15s and timestamp).

But in many applications, you don't care about this.  You're only sampling the data every 15 seconds anyway, which means you'll miss the exact time when the state of the thing you're sampling changed; in other words, the timestamp will already have between 0 and 15 seconds of error. So adding another 0-15 seconds of error is probably not a big deal.

[Victor Sudakov]

Brian Candler wrote:
>
> > I've noticed that if I take a larger resampling interval, like
> > "foo[2d:1h]", I lose all my peaks. Which is kind of understandable now
> > but the question "how to better find peaks" kind of remains.
>
>
> (foo == N)[2d:15s] will find the peaks, with approximate timestamps within
> 15 seconds of the actual time the data was sampled.
>
> If you want, you can then hit the API with additional queries
>
> foo[15s] @timestamp
>
> to get the raw metrics with exact timestamps (it will return the raw
> timeseries between timestamp-15s and timestamp).

This "@" modifier seems quite useful. I had not had it enabled before
this conversation with you. Now I'll be using it more often.

Do you happen to know why it is disabled by default?

>
> But in many applications, you don't care about this. You're only sampling
> the data every 15 seconds anyway, which means you'll miss the exact time
> when the state of the thing you're sampling changed; in other words, the
> timestamp will already have between 0 and 15 seconds of error. So adding
> another 0-15 seconds of error is probably not a big deal.

Thank you Brian, you've been able to help me achieve more clarity. The
PromQL and the CloudWatch approaches to queries are difficult to get
used to for a person who started graphing things with MRTG 20+ years
ago.

[Brian Candler]

On Tuesday, 4 January 2022 at 10:51:45 UTC Victor Sudakov wrote:

This "@" modifier seems quite useful. I had not had it enabled before
this conversation with you. Now I'll be using it more often.

Do you happen to know why it is disabled by default?

I'm guessing because it's experimental and might be withdrawn if it's decided not to be worth the hassle of maintaining it going forward.

You don't need it when using the HTTP API anyway: you specify the time you want the instant query to be evaluated at.  The web interface, which is just a front-end onto the HTTP API, also lets you specify the evaluation time.  So I was using "@timestamp" generically to mean "expression evaluated at that time"; it doesn't have to be literal PromQL.

Cheers,

Brian.

[Victor Sudakov]

Brian Candler wrote:
> On Tuesday, 4 January 2022 at 10:51:45 UTC Victor Sudakov wrote:
>
> > This "@" modifier seems quite useful. I had not had it enabled before
> > this conversation with you. Now I'll be using it more often.
> >
> > Do you happen to know why it is disabled by default?
> >
>
> I'm guessing because it's experimental and might be withdrawn if it's
> decided not to be worth the hassle of maintaining it going forward.
>
> You don't need it when using the HTTP API

> <https://prometheus.io/docs/prometheus/latest/querying/api/#instant-queries>

> anyway: you specify the time you want the instant query to be evaluated
> at. The web interface, which is just a front-end onto the HTTP API, also
> lets you specify the evaluation time. So I was using "@timestamp"
> generically to mean "expression evaluated at that time"; it doesn't have to
> be literal PromQL.

So, the "@timestamp" modifier and the "Evaluation time" selector in the Prometheus
Web UI are the same? I see. But the "@timestamp" modifier in PromQL is
more demonstrative IMHO. Also, if PromQL is a query language, it
should be self-sufficient.

Thanks again for clarification.