Deploy Pushgateway across multiple pods

[Quan Chau]

Hello all,

I'm currently deploying Pushgateway to collect metrics from a batch job. I have set up Pushgateway across multiple pods and a load balancer to distribute requests.

However, because of this, the data pushed are not consistent between pods (the same metric has value 0 in one pod and value 30 in another pod). I know that this is because the data are only pushed to one pod at a time. This makes it very hard for me to create Grafana graphs using the data.

Is there any workaround for this? Or Pushgateway is not supposed to be deployed across multiple pods at all?

[Brian Candler]

Whatever the problem is, pushgateway generally isn't the solution.

If this is a one-shot batch job, then the value that pushgateway stores will be the *last* value that the job wrote - generally some sort of exit status.  It doesn't care whether the job ran on node A or node B, but you'll just get whatever the last value was.

You haven't said what the metric value of 0 or 30 represents.  If this value is a success/fail status code, then storing the last value is the right thing to do.  Otherwise, it may not be.

If you want a separate metric for the jobs which ran on each node, then add the node name as a label.

If you're trying to combine metrics from different nodes - e.g. a count of the total number of things - then you may be better off with the stats_exporter, which gives you shared counters that each job can increment

[Stuart Clark]

On 20/10/2020 00:09, Quan Chau wrote:
> Is there any workaround for this? Or Pushgateway is not supposed to be
> deployed across multiple pods at all?

Pushgateway has no clustering ability, so you should only run a single
instance.