Pushing custom metrics to avoid AKS load balancer

[Marcin Burakiewicz]

Hi.

We have got .NET core application deployed on AKS which exposes an endpoint with some custom metrics in Prometheus format. This application is deployed on two hosts which are behind a load balancer. The problem with load balancer is that whenever Prometheus pulls the metrics from the endpoint it gets them randomly from one of these two hosts depending on load balancer choice.

My idea to solve this is to push metrics from each of these hosts to Prometheus Pushgateway. I'm thinking about setting up separate containers in AKS which will act as sidecars - one per host. They will scrape Prometheus metrics and send them to the Pushgateway from each host independently. 

Please advise me how to scrape and push these metrics to the PushGateway. How they can be collected and what tool/library should be used?

Best regards,

Marcin B.

[Ben Kochie]

If the metrics are unique to each instance behind the load balancer, the best practice is to have Prometheus scrape the pods directly.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/020b8aa1-ef0e-438e-882e-db7bb8312eb0n%40googlegroups.com.

[Marcin Burakiewicz]

Exactly.

So how can this be done? 

I searched for answer here https://prometheus.io/docs/instrumenting/pushing/, but I could not find information how to send to PushGateway the metrics which are already exposed via http in prometheus format.

[Stuart Clark]

On 21/12/2020 10:34, Marcin Burakiewicz wrote:

Exactly.

So how can this be done? 

I searched for answer here https://prometheus.io/docs/instrumenting/pushing/, but I could not find information how to send to PushGateway the metrics which are already exposed via http in prometheus format.

As mentioned you should be scraping each instance and not using the Push Gateway.

piątek, 18 grudnia 2020 o 14:19:25 UTC+1 sup...@gmail.com napisał(a):

If the metrics are unique to each instance behind the load balancer, the best practice is to have Prometheus scrape the pods directly.

On Fri, Dec 18, 2020 at 11:16 AM Marcin Burakiewicz <m.bura...@gmail.com> wrote:

Hi.

We have got .NET core application deployed on AKS which exposes an endpoint with some custom metrics in Prometheus format. This application is deployed on two hosts which are behind a load balancer. The problem with load balancer is that whenever Prometheus pulls the metrics from the endpoint it gets them randomly from one of these two hosts depending on load balancer choice.

My idea to solve this is to push metrics from each of these hosts to Prometheus Pushgateway. I'm thinking about setting up separate containers in AKS which will act as sidecars - one per host. They will scrape Prometheus metrics and send them to the Pushgateway from each host independently. 

Please advise me how to scrape and push these metrics to the PushGateway. How they can be collected and what tool/library should be used?

Best regards,

Marcin B.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/020b8aa1-ef0e-438e-882e-db7bb8312eb0n%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0986394f-9b8b-496d-aa52-8f129be26efdn%40googlegroups.com.

[Marcin Burakiewicz]

What do you mean by "scraping"? How the architecture of this solution should look like? 

[Stuart Clark]

Scraping is what Prometheus does to fetch metrics from an application or exporter.

You would use Service Discovery of some type (static or something like Kubernetes or EC2) to allow each application to be directly scraped (so not via the load balancer).

On 21 December 2020 13:29:29 GMT, Marcin Burakiewicz <m.bura...@gmail.com> wrote:

What do you mean by "scraping"? How the architecture of this solution should look like?

poniedziałek, 21 grudnia 2020 o 14:01:26 UTC+1 Stuart Clark napisał(a):

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.