First value of increase(Counter[Time]) doesn't count

[Johan Jourdan]

Hello,

I have different prometheus counter per Objects.
I want to create a Stacked Bar Chart over the time, an “Acitvity per Day Panel”.
A counter is incremented each time an object is used on the back end side.

My data looks like : 
object_usage_count_total{object="tc1"} 2.0 object_usage_count_total{object="tc5"} 3.0

I'm using the following query : 

increase(object_usage_count_total[5m])
(I’m using 5m for test purpose even if my goal is 1d)

I have the following result for the moment : 

Which is pretty nice, it looks like what I want, except that the first usage of an object is never shown.
For this example I used 3 time TC5 and 2 time TC1. And as we can see there’s a total of 2 activity for TC5 and 1 activity for TC1.

I thought of initializing the counter to 0 before, but the problem is that a new object can be created at any moment, and when it’s created, it is automatically used once. The state “used 0 time” doesn’t really exist.

I have 2 questions:

Is it considered as a bug or is the behavior of “increase” like this on purpose ?

Do you have a solution that might work in my case ?

Thanks

[Brian Candler]

It's working as designed.

Firstly, note that increase() is actually the same as rate() multiplied by the time window size.  It can give you non-integer results.  For example, consider that rate(foo[5m]) takes the rate between the first and last data points within that window; if you are scraping at 1 minute intervals then you will have first and last points which are 4 minutes apart, so it calculates a per-second rate over that 4 minute period. If you use increase() instead of rate() then this rate is multiplied by 300 to estimate the amount it would have increased over 5 minutes.

To get an exact difference between the value now and the value 5 minutes ago, then you can use "foo - foo offset 5m".  But if the counter ever resets, then you will get nonsense values (huge negative spikes).  rate() and increase() skip over those counter drops and calculate a rate using the remaining data points.

Also, "foo - foo offset 5m" will give you no results if the time series did not exist 5 minutes ago (or it had existed, but there's a gap of more than 5 minutes to the previous data point: this is the default "lookback delta" window)

You could do something like this:

    (foo - foo offset 5m) or foo

which will give you the current value of foo if there's no previous value to take a difference against. This takes care of the case where the counter doesn't have an initial zero value.  But again: it will give you nonsense results if there's ever a gap in the timeseries, or if the counter resets.

You can guard against negative spikes with a filter:

    ((foo - foo offset 5m) > 0) or foo

This might be good enough for what you need. It will still give you nonsense values if there are *gaps* in the timeseries, e.g. due to scrape failures.  But more seriously, it will fail if the counter didn't start at zero.

In general, you can't assume that counters start at zero. The counter may have been running and incrementing for a long time, before Prometheus starts scraping it; the first value you see could represent days or years of accumulation.  That's why rate() and increase() only give a value if there are two or more adjacent data points. It's the only way to be sure that the value has actually increased over that time period.  And if you don't generate an initial timeseries value of zero, then it can only use the next two values to calculate a rate between those two values.

[Brian Candler]

 ((foo - foo offset 5m) >= 0) or foo

[Johan Jourdan]

Thanks a lot Brian.
Explanations are super clear, the fix is working perfectly,

You saved my day, thanks !

[Ben Kochie]

FYI, we have work in progress to start supporting the OpenMetrics "created timestamp" feature. This will allow Prometheus to know when a counter started at 0 so it can do appropriate rate/increase calculations for newly created series.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f13f9992-f3d8-47d6-9562-a8de595fc0b3n%40googlegroups.com.