Start recording rule until N mins of data is collected

[Shubham Choudhary]

I am using the Prometheus recording rule to capture the CPU usage over 5 mins and later want to use it to get maximum CPU used for 5 mins in the last 30 days with max_over_time(instance:node_cpu_usage:rate5m[30d]).

- expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

record: instance:node_cpu_usage:rate5m

Suppose a new node is getting scraped by Prometheus and the first few values for "instance:node_cpu_usage:rate5m" will be very high since there is no data for the first 5 mins and the rate for them will be very less. And doing max_over_time on will result in the wrong.

So, is there a way to limit the recording rule to start after 5 mins or so or ask rate function to do the calculation for the data it has?

What happens when I do rate(metricsX[5m]) but only 1 min of data is present?

[Brian Candler]

> What happens when I do rate(metricsX[5m]) but only 1 min of data is present?

rate() takes the first and last point within the given time window, and calculates the rate between them.  So: as long as there are at least two data points in the window(*), you'll get a result.  If not, you'll get no result.

(*) And the second is higher than the first, since rate works with counters.

[Murali Krishna Kanagala]

Every time the data is scrapped Prometheus does that calculation and record the custom metric. 5min here is a sliding window with latest metric's time stamp as the curren time and the previous one (if it exists within last 5min) as the previous metric. If there is no metric in last 5 min you'll not see any result. 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/686f17dc-988c-4c14-a6bd-a648a9f80a56%40googlegroups.com.

[Brian Candler]

On Wednesday, 8 April 2020 00:14:04 UTC+1, Murali Krishna Kanagala wrote:

Every time the data is scrapped Prometheus does that calculation and record the custom metric. 5min here is a sliding window with latest metric's time stamp as the curren time and the previous one (if it exists within last 5min) as the previous metric.

I don't think that's true for recording rules.  Firstly, they are evaluated at the rule evaluation interval, not the scrape interval.  Secondly, the 5 minute window runs from the rule evaluation time backwards, not the "latest metric's time stamp".

[Shubham Choudhary]

Yes, the rate function will take the first & last and calculate the value.

But, it will give wrong values if I do rate on a metrics let's say [1d] and only 1 min of data is present so it will first get the difference b/w first and last metrics and divide by 24 * 60 * 60(sec in 24 hours) which will be very very small nearly zero. And if I do min_over_range it will give me the wrong values.