Looks like increase()/rate() do not account for series start

[hai...@gmail.com]

I have a counter that first scraped with >0 value. However increase() does not pick it up, still saying there was zero increase(). Is that expected? 

I'm on Prometheus 2.0.0.

Here is what metrics look like:

And here is what increase() returns - not accounting for new series that appeared in ~12.10.

[Brian Brazil]

On 21 November 2017 at 13:20, <hai...@gmail.com> wrote:

I have a counter that first scraped with >0 value. However increase() does not pick it up, still saying there was zero increase(). Is that expected? 

I'm on Prometheus 2.0.0.

Yes, that's expected. There was no increase, merely a time series appearing out of thin air. It is recommended to initialise all potential labels for your metrics to 0 to avoid this.

Brian

 

Here is what metrics look like:

And here is what increase() returns - not accounting for new series that appeared in ~12.10.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/575e6688-738a-4eca-bc33-2742862513c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[hai...@gmail.com]

On Wednesday, 22 November 2017 00:26:13 UTC+11, Brian Brazil wrote:

On 21 November 2017 at 13:20, <hai...@gmail.com> wrote:

I have a counter that first scraped with >0 value. However increase() does not pick it up, still saying there was zero increase(). Is that expected? 

I'm on Prometheus 2.0.0.

Yes, that's expected. There was no increase, merely a time series appearing out of thin air. It is recommended to initialise all potential labels for your metrics to 0 to avoid this.

Thanks for clarifying that. Initialization is a problem in my case, since I split them by path, which is dynamic in my application (with small cardinality, but still, dynamic) and therefore I can't initialize them in advance.

Is there any workaround? Any missing_as_zero=true flag?

 

[Brian Brazil]

There's no real workaround, on average though the results will be correct.

--

Brian Brazil

www.robustperception.io

[hai...@gmail.com]

On average, yes. But in my case, again, timeouts are quite rare, and there is good chance that pod will be restarted/redeployed between them, which will create a new metric, - i.e. I may never see them that way.

I guess I'll create another cumulative metrics for timeouts to see that at least something is going on.