rate/increase function gives unexpected spike

[Johny]

There is one data point in series where rate/irate/increase gives an unexpected spike. underlying series has 2 points with a small increase.

time                series               irate=irate(series[1m])
12:15:00                                  300,000  
12:14:58        400,000
12:14:50        399,974   

The series is coming from a recording rule. what could be the reason for this abnormality? is there a bad data point in series not visible in range query but irate picks it up?

[Brian Candler]

Can you provide more direct evidence of your claim - i.e. the exact API calls and responses you are performing to get the values you show?

And what prometheus version?

[Johny]

I just realized the issue is with counter decreasing at times and losing monotonicity. Thanks.

[Brian Candler]

rate and irate should take care of that automatically.  However if the counter goes 300,000 -> 0 -> 300,000 then you'll see a big spike of course.