Central pushgateway: limits/sizing for pushgateway

[DerekLai Devops]

We have been using Prometheus to monitor our AKS clusters for the various departments. Each has their own AKS cluster for ease of management/billing.

Now the users would like to setup pushgateways to help with getting metrics on jobs which is shorted lived and can't be scraped. Management would like to setup just 1 "central" pushgateway. We have a total of about 10 Prometheus instances.

I know this is probably not recommended due to potential single point of failure/bottleneck issues. However, just wanted to find out the limit in case the reverse is true - that we should setup multiple pushgateway due to performance issues. Still trying to get a sense of how many batch jobs and how frequently they run from the users. 

Is there any kind of sizing guide for the pushgateway itself?

Thanks,

Derek

[Bjoern Rabenstein]

On 16.06.22 02:05, DerekLai Devops wrote:
>
> Is there any kind of sizing guide for the pushgateway itself?

There is no hard limit. Obviously, your Pushgateway will need more and
more resources the more metric it has to keep in RAM. Also, checking
the metrics for consistency will get harder and harder (every new
metric has to be checked against all the existing ones). Finally,
Prometheus will scrape all pushed metrics in one go, so you don't want
to let the scrape get too large, either.

As a rule of thumb, I would say, you should not go beyond a few
hundred metrics present on the Pushgateway at the same time.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in