scrape metrics for applications sitting behind a load balancer

[Eswar Rao Bevara]

Hi,

I have a spring boot application running on two vms and sitting behind a load balancer. when I try to scrape using the application url/insights , each time the call goes to one of the two vms. I want to have the metrics scrapped from each vm separately and then would search for a solution on combining them in Grafana. 

Any suggestion on how to get the metrics from both the vms to the prometheus server. Node manager wouldn't work as this would give the metrics of only linux system but not the application.

Thanks

Eswar

[Christian Hoffmann]

Hi,

On 3/17/20 10:02 AM, Eswar Rao Bevara wrote:
> I have a spring boot application running on two vms and sitting behind a
> load balancer. when I try to scrape using the application url/insights ,
> each time the call goes to one of the two vms. I want to have the
> metrics scrapped from each vm separately and then would search for a
> solution on combining them in Grafana. 
>
> Any suggestion on how to get the metrics from both the vms to the
> prometheus server.

I guess your Prometheus server does not have access to the
non-load-balanced services, right? This would of course be the simplest
variant.

Other than that, you would have to trick your loadbalancer somehow.
Either it already has some kind of session affinity which you could use
or you could introduce such a thing (e.g. ?force-backend=1, different
vhost or something).

I don't think there is a standard way for that...?

Kind regards,
Christian

[Jakub Jakubik]

hi,

well there is a hacky trick you can do - scrape very often, and make sure that you add a label that identifies the backend in the backend response.

instead of

node_cpu_seconds{cpu=1} 10

return

node_cpu_seconds{cpu=1,backend=vm1} 10

then ignore or rewrite the instance label as it would be the same for both.

given that you scrape often enough you should gather the data from both the machines eventually 

hth,

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/4f25af97-3df2-7234-b281-10c1b4d98934%40hoffmann-christian.info.

--

Kuba Jakubik SRE Tech Lead Netguru - Building software for world changers

jakub....@netguru.com netguru.com

[Brian Candler]

On Wednesday, 18 March 2020 09:35:45 UTC, Jakub Jakubik wrote:

then ignore or rewrite the instance label as it would be the same for both.

In that case you can actually return an "instance" label as part of the scrape results, and set "honor_labels: true" in the scrape job to trust and retain the instance label returned by the remote host.

However, this seems to be completely the wrong thing to do here.  You should not be scraping two backends through a load balancer - you should be scraping the two backends separately and directly.  If there is no direct network connectivity between the prometheus server and the two backends, then enable it somehow, e.g. via port forwarding - or at worst use PushProx which can act as a meet-in-the-middle.

[sayf eddine Hammemi]

For me, the load balancer itself should be polled, and it as the sole accessor of the applications needs to expose metrics about them with proper labels

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/ce86f522-e4f0-4e81-9356-b3186ace1dac%40googlegroups.com.

[Brian Candler]

If there is no other way, then you can configure your load balancer to forward /backend1 and /backend2 requests to the two individual backends.  But you probably want to include authentication, as otherwise the whole world will be able to scrape these endpoints.