Question about using Push Gateway for aggregating metrics

[Mana M]

Hello,

I am new the Prometheus and had a question about how to use PushGateway properly.

Basically, we have different kinds batch job that get scheduled by our application. These batch jobs are run as separate process by the scheduler framework on different worker hosts, they do bunch of things, and go away within a minute or so. Seems to be a good use case for using PushGateway. However, I am not able to set things up properly to get correct metrics values.

I want to measure the rate for how many times batch jobs are running per minute. Inside batch job, I am using Counter, which gets incremented at the end of batch job with batch job name as label. I using new Registry (non-default one) for Counter and also use "pushadd_to_gateway" Python Prometheus client API to push metrics to gateway at the end. However -

* It seems that PushGateway always overwrites the metrics every time I push ephemeral Counter value from the short-lived batch job process.

* Providing "grouping keys" does not help as grouping keys seem to get used to figure out which metrics get overwritten based on grouping key values. 

* Using unique job names or unique values for grouping key won't overwrite the earlier metrics on push gateway, but then this would end up creating separate time series every time I push same metric, which would not be good.

Isn't there an option with Push Gateway, where it would increment earlier pushed metrics every time I push metrics from ephemeral process instead of overwriting? How do I get rate of batch jobs, latencies of batch jobs, etc. with Push Gateway?

Thanks in advance.

[Mana M]

Seems like there is also similar question here: https://stackoverflow.com/questions/50923880/prometheus-intrumentation-for-distributed-accumulated-batch-jobs

This looks like fairly simple usecase for Push Gateway. But still somehow not able to find the solution. Has anyone implemented something like this with Push Gateway?

[Benjamin Ritcey]

Searching for 'aggregating push gateway' brings up https://github.com/weaveworks/prom-aggregation-gateway as my first hit:

"Prometheus Aggregation Gateway is a aggregating push gateway for Prometheus. As opposed to the official Prometheus Pushgateway, this service aggregates the sample values it receives."

No experience with it, though others here have mentioned it.

[Björn Rabenstein]

As Ben said...

The first non-goal listed in
https://github.com/prometheus/pushgateway#non-goals is dealing with
the topic.

The reason why we don't have any "native" support for distributed
counters within the Prometheus Github org is mostly that it is hard to
get right. I would try to avoid being in a situation where you need
distributed counting, but I know that's not always an option.

Efforts like https://github.com/weaveworks/prom-aggregation-gateway
will create some experience with that in the Prometheus context.
Perhaps we will eventually have an "official" story how to do it.

[Austin Poole]

The gauge issue and having to restart prom-aggregation-gateway to delete stale metrics was a no go for our use case. That's why I created the Metrics Accumulator https://github.com/bpoole6/metrics-accumulator.

The Metrics Accumulator doesn't add gauge values together, it instead takes the latest value pushed for a gauge (There is some customization here).

I also made it where your metrics are partitioned into what I call metric groups. Using the service discovery feature within Prometheus it can scrape different metrics path on the same host to read the separate group... Or you could just dump it all in metric group and just scrape that.

It also has a TTL for your metrics.