Two different values for the same day when calculating max_over_time over two different time ranges

[Raghu c]

I am tracking the number of jobs in a queue at specific time intervals using a gauge metric. Prometheus scrapes this every minute.

However, when I attempt to determine the highest number of jobs in the queue on a given day using the max_over_time query, I receive two distinct values for the same day based on different time ranges.

I am using the query max_over_time(job_count_by_service{service="ServiceA", tenant="TenantA"}[1d]). When I run this query for a 1-day time range (from 2023-08-19 00:00:00 to 2023-08-19 23:59:59), the value I get is 38. However, when I run the same query for a 5-day time range (from 2023-08-18 00:00:00 to 2023-08-22 23:59:59), the result for Aug 19th is 35.

2023-08-19 00:00:00 to 2023-08-19 23:59:59  [Range Selected in Grafana]

2023-08-18 00:00:00 to 2023-08-22 23:59:59  [Range Selected in Grafana]

In Grafana I have configured the Min Step as 1d and Type as Range. I'm not sure whether that could affect the values in any way.

I assumed that max_over_time would pick the max value among all the values that fall in the range vector specified time period. For example, if on Day 1 the values are [1,2,7,6,5] and on Day 2 the values are [8,1,2,3,1] then the query would return 7 & 8 respectively for each day.

To elaborate when I select 2023-08-19 00:00:00 to 2023-08-19 23:59:59 as the time range the values from 2023-08-18T00:00:00.000 ... 2023-08-19T00:00:00.000 is considered to pick a maximum. When I select the time range 2023-08-18 00:00:00 to 2023-08-22 23:59:59 won't the same values again be considered for picking the maximum since the range vector is 1d? So how can the value change from 38 to 35?

[Brian Candler]

The expression

    max_over_time(job_count_by_service{service="ServiceA", tenant="TenantA"}[1d])

gives an *instant* vector as its result.  You should evaluate it at a specific instant in time, and it will give you the maximum over the 24 hours up to that instant.

You shouldn't run this over a time range in Grafana.  If you do, then you'll get a set of maximums swept over that range.  For example, if you run it over a 6 hour range, then the first data point will be the maximum from (now-30h) to (now-6h), and the last data point will be the maximum from (now-24h) to (now)

Alternatively, you can run a regular query over a Grafana time range, and get *Grafana* to work out the maximum over that period.