Recording rules at the end of hour

[Mathieu Tétreault]

Hello,

I am trying to get my head around an issue that I am having.

Let's say I have metrics A, B  and C that are scraped every 15 seconds. At the end of each hours we need to use metrics A, B and C to create  metric D. The D metric needs to be timestamped at the beginning of the hour. Let says the metric D is calculated at the 14h59, it should be timestamped at 14h00.

In other words, is there a way to save a metric at the end of each hour and can it be timestamped as it was scraped at the beginning of the hour?

I was looking toward using the recording rules, but I haven't found anything that would do it.

Is prometheus made to support that?

Cheers,

Mathieu

[Brian Brazil]

It isn't. Eval intervals like scrape intervals shouldn't be over 2m, and even when in the 2m a given scape/evaluation will be performed is arbitrary for the sake of spreading out load. In addition PromQL can't look into the future, and recording rules can't perform evaluations that output to the past.

Prometheus isn't designed with 100% exact reporting in mind, https://www.robustperception.io/monthly-reporting-with-prometheus-and-python is one way you could approach this.

--

Brian Brazil

www.robustperception.io

[Mathieu Tétreault]

Alright, make sense.

Thank you for your answer, it confirms what I was thinking.

Mathieu

[Mathieu Tétreault]

Hi Brian, 

You got me wondering, why scrape intervals shouldn't be over 2minutes? Is there anydown side to that?

We are looking toward replacing a legacy system, there is already a mysql database that is populated every hour, so that would allow us to deploy a first version using the current mysql database, then in the following release we could instrument the different applications and then remove the mysql database.

Cheers,

Mathieu

[Stuart Clark]

On 2020-07-07 15:58, Mathieu Tétreault wrote:
> Hi Brian,
>
> You got me wondering, why scrape intervals shouldn't be over 2minutes?
> Is there anydown side to that?
>
> We are looking toward replacing a legacy system, there is already a
> mysql database that is populated every hour, so that would allow us to
> deploy a first version using the current mysql database, then in the
> following release we could instrument the different applications and
> then remove the mysql database.

A time series will be marked as stale if it hasn't been updated at least
once every 5 minutes.

So the standard maximum of 2 minutes ensures there is at least once
update every 5 minutes even if one scrape fails.

--
Stuart Clark

[Mathieu Tétreault]

Alright,

To be sure that I am understanding correctly. The consequence would be that time series will be marked as stale, that would mean that grafana wouldn't be able to show the data past the 5 minutes delay. Is that correct?

The other thing that I see, perhaps, I could increase the 5minutes period to something a little bit over an hour as I will scrape the metrics every hours.

[Ben Kochie]

Prometheus also writes samples in 120 sample blocks (chunks). These are flushed to disk at a maximum of every 2 hours. So any scrape interval over 1 minute is under-utilizing chunk compression.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6bd0851f14eeb508cd0a9c5801c9e063%40Jahingo.com.