A solution for counters automatic initialization to 0

[Bruno Martinez]

Hello Prometheus Community!

There have been many discussions and issues related to the problem of new timeseries and the increase() (example: here, or here) and even the official documentation of Prometheus mentions to "avoid missing metrics".

The workaround consist in manually initializing our metrics to 0. This can become complex with vectors when we don't know in advance the combination of label values, moreover we need to be sure the first scrape from Prometheus include this 0 value.

At Goto.com we implemented a solution to this problem that has been running in production for more than 1 year and we decided to open it to the community.

It's golang library extending the official client that ensure the first collection(s) of counters always return 0 instead of their actual value. This works for counters/histogram/summaries and the corresponding vectors.

This is done in the background without anything to add in your application code.


We hope this will help the community, try it!

feedback are welcome.


https://github.com/goto-opensource/smart-prometheus-client

Have a nice day!
Bruno

[Julius Volz]

Hi Bruno,

Could you outline how returning 0 (instead of 1) for counters for their first scrape helps with the existence issues of a time series? The sample value of a series should have no effect on the series' existence.

And as https://github.com/goto-opensource/smart-prometheus-client#counters-warm-up notes, it's important to point out here again that this is not really compatible with Prometheus' collection model where producers of metrics should not need to know how many Prometheus servers are scraping them, and GET requests to /metrics should be idempotent for that reason.

Regards,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/63982f2c-32f6-4d91-b694-cd08226203e1n%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Julius Volz]

(Ah, I guess this is about "catching" any single initial counter increases for slow-moving counters as well, so that rate outputs are not 0 on those)

[Bruno Martinez]

Hi Julius,

Indeed this is about catching the initial increase of counters.
If, for example, we count the http request of a server, and we got 30 requests just after start-up, before the first scrape by Prometheus.
Exporting the counter first to 0 (before exporting 30)  ensures these 30 requests are included in the result of increase() over a period that includes the start-up time.

Concerning your second point, the library supports defining a 'warm-up' period during which each new counter (new combination of labels) will be exported to 0 (whoever makes the GET request on /metrics). We use this for the case of multiple Prometheus. In our case we have between 2 and 4 instances and we use thanos for query.

So you don't need to know how many Prometheus, but make sure the warm-up phase is big enough to cover the scrape period (in our case all Prometheus have the same scrape period of 30s).

I agree this adds some dependency between the exporter and Prometheus.

Regards
Bruno

[Bartłomiej Płotka]

While searching for some patterns I noticed this thread.

Looked on this counter warm-up solution - quite clever, thanks for sharing!

We are working on different solution, utilizing created timestamps feature. It's being designed here and client implementation is in progress here. Feedback welcome!