accumulating counter metric that are never incremented since restart

106 views
Skip to first unread message

Johny

unread,
May 3, 2023, 3:19:27 PM5/3/23
to Prometheus Users
I need to sum two separate counter metrics capturing request failures  to compute ratio of error requests for an alerting signal. The code initializing and setting these counters sits in separate modules preventing reuse of one counter.

The problem is when one of the counter is never incremented after a restart, service never exports the data point, prometheus will never get the time series and the summation will return nothing. 

Is there a way to "force" publish a counter to 0 always on service reboot during counter initialization to avoid this problem?

 (fail_count1 + fail_count2) / (total_count1 + total_count2)

Brian Candler

unread,
May 3, 2023, 3:33:38 PM5/3/23
to Prometheus Users
Yes, you should be able to create a counter which publishes its initial value of zero. I'm fairly sure I've done this with the Golang client some time in the past. What language and client library are you using?

You'll have to initialise the counters explicitly. If the first time the client library knows about the counter is when you increment it, then clearly it won't be able to export it until then.

Johny

unread,
May 3, 2023, 3:52:57 PM5/3/23
to Prometheus Users
Yes golang. I am using promauto to auto register my metrics with the default registry at the time of initialization.

Do you recall a way to force export it always with a default initial value of 0? 

Brian Candler

unread,
May 3, 2023, 4:51:53 PM5/3/23
to Prometheus Users
Is it a CounterVec you're using?

If so, I think that v.With(labels...) should be sufficient to initialise it.

Brian Candler

unread,
May 3, 2023, 4:55:03 PM5/3/23
to Prometheus Users
Confirmed. Take the "Basic Example" from here:

Remove the .Inc() from the "m.hdFailures.With" line

$ curl -fsS localhost:8080/metrics | grep hd_errors
# HELP hd_errors_total Number of hard-disk errors.
# TYPE hd_errors_total counter
hd_errors_total{device="/dev/sda"} 0

Johny

unread,
May 3, 2023, 6:42:57 PM5/3/23
to Prometheus Users
Thanks Brian. I was able to use <CounterVec>.withLabels(...).Add(0) in the beginning to export the 0 value.

Brian Candler

unread,
May 4, 2023, 2:35:28 AM5/4/23
to Prometheus Users
Great: I think you can drop the .Add(0) and the counters will still be created with a zero value.
Reply all
Reply to author
Forward
0 new messages