Pattern(s) for having a counter in short-lived applications (batch jobs)

[Sean LeBlanc]

I'm trying to figure out if there is a way to have a counter index even between application runs. My use case would be jobs scheduled via Kubernetes CronJob, and having counters used for when certain methods are called. The app would typically run in a few seconds and be done. So I set up pushgateway, and try running the app - I see the counter show up in the pushgateway, but it starts over again on the next run of the application.

Is there some common practice that is done here that I should be following if I want a counter's value to persist between application runs?

[Murali Krishna Kanagala]

Hi Sean,

Prometheus does not keep track of the counters. Metric source or the exporter does it. 

As the metric goes from 0 to N on every run of the cronjob you can create a synthetic metric in Prometheus that tracks the increase. If the labels are persistent you can track the increase and probably can aggregate them. 

On Tue, Apr 14, 2020, 6:31 PM Sean LeBlanc <seanl...@gmail.com> wrote:

I'm trying to figure out if there is a way to have a counter index even between application runs. My use case would be jobs scheduled via Kubernetes CronJob, and having counters used for when certain methods are called. The app would typically run in a few seconds and be done. So I set up pushgateway, and try running the app - I see the counter show up in the pushgateway, but it starts over again on the next run of the application.

Is there some common practice that is done here that I should be following if I want a counter's value to persist between application runs?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/927f938d-aa3a-4670-9a10-17371e911448%40googlegroups.com.

[sayf eddine Hammemi]

You can use a gauge and create a recording rule that pushes the sum of the gauge over time I guess

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAKimyZv_OwQqBK%3DPErSeD%2B25Y-mfSPZ_nGiQtKM_63D%3D1NdZ8g%40mail.gmail.com.

[Brian Candler]

pushgateway doesn't aggregate counters, but statsd_exporter does.  This sounds more like what you need.

[Murali Krishna Kanagala]

Thanks Brian! 

On Wed, Apr 15, 2020, 5:49 AM Brian Candler <b.ca...@pobox.com> wrote:

pushgateway doesn't aggregate counters, but statsd_exporter does.  This sounds more like what you need.

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f710266e-0cb5-439a-a19c-cbd338a24f27%40googlegroups.com.