How to avoid Gaps in the Graphs

[Hari Yada]

Hi Experts,

   I have an application and it is scaled to 3 instances. Using Prometheus I would like to scrape the metrics per each instance. for example i would like to know how many number of http requests received per instance.

   so i am exposing a metric like this requests_total {instance_id="xxxxxxx",} 9.0. finally i have specified the application url in prometheus config file.

problem that i am facing:

since the service is load-balanced, when prometheus scrapes it hits one of the instance and fetch the metrics. eg: if prometheus hits instance one , at this point in time it will not have the details of instance2 and 3 and hence there are

gaps in the graphs for this scrape interval.

to avoid the above situation, i need to use a persistence store to store the metrics for all the instances and when prometheus asks that time i need it fetch from the store.

Generally i would like to know what is the recommended way for this kind of situation

Best Regards,

Hari

[Ben Kochie]

Prometheus should not scrape through your load balancer. It should have direct access to all instances.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d3e8bd55-5091-43e1-9302-0ff31b3ec39bo%40googlegroups.com.

[Hari Yada]

How to have direct access to all instances, I have my app deployed on cloud-foundry and hence don't have access to the individual instance routes.. 

[Ben Kochie]

As far as I know, this is a design flaw in cloud foundry that you can't get around. But I don't really know anything about cloud foundry outside of the fact that it's nearly incompatible with Prometheus.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6f1c0708-b0e9-46f1-8a13-e2a9b8668ba0o%40googlegroups.com.

[Mark Poko]

I had this issue about a year back with Cloud Foundry, and ended up writing a custom exporter that scrapes Cloud Foundries metrics directly. I didn't open source it and it's not robust, but that's always an option. It was ~50 LoC with NodeJS.

On Tuesday, June 9, 2020 at 9:14:28 AM UTC-4, Ben Kochie wrote:

As far as I know, this is a design flaw in cloud foundry that you can't get around. But I don't really know anything about cloud foundry outside of the fact that it's nearly incompatible with Prometheus.

On Tue, Jun 9, 2020 at 2:55 PM Hari Yada <hari....@gmail.com> wrote:

How to have direct access to all instances, I have my app deployed on cloud-foundry and hence don't have access to the individual instance routes.. 

On Tuesday, June 9, 2020 at 10:17:18 AM UTC+5:30, Hari Yada wrote:

Hi Experts,

   I have an application and it is scaled to 3 instances. Using Prometheus I would like to scrape the metrics per each instance. for example i would like to know how many number of http requests received per instance.

   so i am exposing a metric like this requests_total {instance_id="xxxxxxx",} 9.0. finally i have specified the application url in prometheus config file.

problem that i am facing:

since the service is load-balanced, when prometheus scrapes it hits one of the instance and fetch the metrics. eg: if prometheus hits instance one , at this point in time it will not have the details of instance2 and 3 and hence there are

gaps in the graphs for this scrape interval.

to avoid the above situation, i need to use a persistence store to store the metrics for all the instances and when prometheus asks that time i need it fetch from the store.

Generally i would like to know what is the recommended way for this kind of situation

Best Regards,

Hari

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

[Brian Brazil]

On Tue, 9 Jun 2020 at 14:14, Ben Kochie <sup...@gmail.com> wrote:

As far as I know, this is a design flaw in cloud foundry that you can't get around. But I don't really know anything about cloud foundry outside of the fact that it's nearly incompatible with Prometheus.

https://github.com/promregator/promregator is generally what to use with it.

Brian

 

On Tue, Jun 9, 2020 at 2:55 PM Hari Yada <hari.y...@gmail.com> wrote:

How to have direct access to all instances, I have my app deployed on cloud-foundry and hence don't have access to the individual instance routes.. 

On Tuesday, June 9, 2020 at 10:17:18 AM UTC+5:30, Hari Yada wrote:

Hi Experts,

   I have an application and it is scaled to 3 instances. Using Prometheus I would like to scrape the metrics per each instance. for example i would like to know how many number of http requests received per instance.

   so i am exposing a metric like this requests_total {instance_id="xxxxxxx",} 9.0. finally i have specified the application url in prometheus config file.

problem that i am facing:

since the service is load-balanced, when prometheus scrapes it hits one of the instance and fetch the metrics. eg: if prometheus hits instance one , at this point in time it will not have the details of instance2 and 3 and hence there are

gaps in the graphs for this scrape interval.

to avoid the above situation, i need to use a persistence store to store the metrics for all the instances and when prometheus asks that time i need it fetch from the store.

Generally i would like to know what is the recommended way for this kind of situation

Best Regards,

Hari

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6f1c0708-b0e9-46f1-8a13-e2a9b8668ba0o%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmrPfAYwYmLCkr_H%3D36Mkr0sebbGm0p6hr2pS%2B%2BiCG086A%40mail.gmail.com.

--

Brian Brazil

www.robustperception.io

[Harkishen Singh]

You should rather have them exposed as either 3 different NodePort or use their instantaneous cluster IP and have these changes in the prometheus config. Also, providing different labels will help you segregate the results easily.