Prometheus for scraping multiple instances in a load balancer.

Rohit Bharati

unread,

Jul 12, 2021, 5:15:08 AM7/12/21

to Prometheus Users

Hi here !

I am running multiple instances on one domain and I have implemented Prometheus Exporter provide me the data processed. I am counting the number of images received for my application.

Scenario :

Everything is working fine when the app has one Instance in the load balancer.

But when I scale it to 2, the image is processed by other instance and Prometheus scrapes the other one. So even if i upload an Image there is only 50% chnace i will get the count in /metrics

What has to be done in this scenario, has anyone any Idea on this
I know Promregator thanos etc... but can anything be done from exporter side so that only Prometheus is enough to give what I need.

Thanks

Bjoern Rabenstein

unread,

Jul 14, 2021, 1:19:16 PM7/14/21

to Rohit Bharati, Prometheus Users

On 12.07.21 02:15, Rohit Bharati wrote:
>
> I am running multiple instances on one domain and I have implemented
> Prometheus Exporter provide me the data processed. I am counting the number
> of images received for my application.
>

> *Scenario : *

> Everything is working fine when the app has one Instance in the load
> balancer.
> But when I scale it to 2, the image is processed by other instance and
> Prometheus scrapes the other one. So even if i upload an Image there is
> only 50% chnace i will get the count in /metrics
>
> What has to be done in this scenario, has anyone any Idea on this
> I know Promregator thanos etc... but can anything be done from exporter
> side so that only Prometheus is enough to give what I need.

Not sure I fully understand your setup, but it's fundamentally
important to never scrape instances of an app through a load
balancer. Always scrape all the instances directly. Then use suitable
PromQL expressions to aggregate whatever you need.

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

Nemanja Delic

unread,

Jul 16, 2021, 9:12:46 AM7/16/21

to Bjoern Rabenstein, Rohit Bharati, Prometheus Users

Trying to get the different metrics of a few java apps that are exposed through LB(number of hits, incoming traffic etc) . Since it's unlikely that we will switch to spring boot 2 - with sb1 there's poor support for prometheus instrumentation. The way I see we have two options:

1. either we use log parsers directly - and get the metrics that we later aggregate what we need, or

2. we pull the LB logs - and do aggregations in that way

What's the downside of the 2nd solution?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/20210714171911.GD3656%40jahnn.

grze...@gmail.com

unread,

Jul 27, 2021, 3:52:09 AM7/27/21

to Prometheus Users

Hi,

if you cannot use service discovery to discover instances over Load Balancer, you could look at:
* pushgateway
* https://github.com/prometheus-community/PushProx

Scraping via LB is not what you want in most cases, you will be monitoring only part of your app and the metrics would not have sense, i.e. you break monocyty in counters etc.