Kubernetes HPA based on multiple Prometheus servers

[Matthias Rampke]

Hello,

we would like to scale Kubernetes deployments based on custom metrics from namespace-local Prometheus servers, in other words, multiple (tens) Prometheus servers per cluster. Has this been done?

As far as I can tell, the custom metrics API specifies a specific endpoint, so that I can only specify one ApiService to one Prometheus adapter, which in turn only accepts one Prometheus URL. Currently, we work around this by federating selected metrics to one Prometheus server per cluster; however this introduces additional delay in the autoscaling control loop.

Is there another way?

I noticed that KEDA allows Prometheus triggers that each specify a Prometheus URL. Is this useful for continuous scaling based on metrics such as the (time-averaged) number of concurrent requests in flight?

Any experiences appreciated!

Best,

MR