Invalid Push metric issue while pushing metric

Vidur

unread,

Apr 8, 2020, 8:40:21 AM4/8/20

to Prometheus Users

Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application

void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
            gaugeRegiseryMap.put(request,inprogressRequests);
        }else{
            inprogressRequests = gaugeRegiseryMap.get(request);
        }
        try {
            inprogressRequests.labels(labelValues[labelValues.length-4],
                    labelValues[labelValues.length-2],
                    labelValues[labelValues.length-3],
                    labelValues[labelValues.length-1]).set(Double.parseDouble(counter));
            registry.register(inprogressRequests);
        } finally {
            PushGateway pg = new PushGateway("127.0.0.1:9091");
            pg.pushAdd(registry, request);
        }
    }

the following code is working ,
but when i change inprogressRequests.labels() when i pass the array containing label values
i.e.

 void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
            gaugeRegiseryMap.put(request,inprogressRequests);
        }else{
            inprogressRequests = gaugeRegiseryMap.get(request);
        }
        try {
            inprogressRequests..labels(labelValues).set(Double.parseDouble(counter));
            registry.register(inprogressRequests);
        } finally {
            PushGateway pg = new PushGateway("127.0.0.1:9091");
            pg.pushAdd(registry, request);
        }
    }

the same is giving me errror -
Response code from http://127.0.0.1:9091/metrics/job/loop6_2 was 400, response body: pushed metrics are invalid or inconsistent with existing metrics: collected metric .

This is my first time using prometheus , it will be good if someone can share light on this.

Brian Brazil

unread,

Apr 8, 2020, 9:58:01 AM4/8/20

to Vidur, Prometheus Users

On Wed, 8 Apr 2020 at 13:40, Vidur <vidur...@getfareye.com> wrote:

Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application

void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();

You shouldn't be registering metrics to the default registry in a typical method. Did you mean to register it to the custom registry you just instantiated?


            gaugeRegiseryMap.put(request,inprogressRequests);

Nor should you need to track what metrics exist, see https://www.robustperception.io/label-lookups-and-the-child


        }else{
            inprogressRequests = gaugeRegiseryMap.get(request);
        }
        try {
            inprogressRequests.labels(labelValues[labelValues.length-4],
                    labelValues[labelValues.length-2],
                    labelValues[labelValues.length-3],
                    labelValues[labelValues.length-1]).set(Double.parseDouble(counter));
            registry.register(inprogressRequests);
        } finally {

You shouldn't need a try-finally for this, it should never fail given the above code.

I suspect what you're looking for here is a custom collector by subclassing Collector.

Brian


            PushGateway pg = new PushGateway("127.0.0.1:9091");
            pg.pushAdd(registry, request);
        }
    }

the following code is working ,
but when i change inprogressRequests.labels() when i pass the array containing label values
i.e.

 void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
            gaugeRegiseryMap.put(request,inprogressRequests);
        }else{
            inprogressRequests = gaugeRegiseryMap.get(request);
        }
        try {
            inprogressRequests..labels(labelValues).set(Double.parseDouble(counter));
            registry.register(inprogressRequests);
        } finally {
            PushGateway pg = new PushGateway("127.0.0.1:9091");
            pg.pushAdd(registry, request);
        }
    }

the same is giving me errror -
Response code from http://127.0.0.1:9091/metrics/job/loop6_2 was 400, response body: pushed metrics are invalid or inconsistent with existing metrics: collected metric .

This is my first time using prometheus , it will be good if someone can share light on this.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/11440241-e4f6-40a3-9698-e65498d3db3a%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

Vidur

unread,

Apr 10, 2020, 2:10:26 AM4/10/20

to Prometheus Users

On Wednesday, April 8, 2020 at 1:58:01 PM UTC, Brian Brazil wrote:

On Wed, 8 Apr 2020 at 13:40, Vidur <vidur...@getfareye.com> wrote:
Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application
void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
You shouldn't be registering metrics to the default registry in a typical method. Did you mean to register it to the custom registry you just instantiated?

My use case is something which sets gauge value for a same request in fixed interval, so is there no need to register the gauge ?
And Gauge are created real time with different request values , so i cannot make them static.

Did you check this part where I'm trying to set label values like this ? why is this not working but the above one is?

            registry.register(inprogressRequests);
        } finally {
            PushGateway pg = new PushGateway("127.0.0.1:9091");
            pg.pushAdd(registry, request);
        }
    }
the same is giving me errror -
Response code from http://127.0.0.1:9091/metrics/job/loop6_2 was 400, response body: pushed metrics are invalid or inconsistent with existing metrics: collected metric .

This is my first time using prometheus , it will be good if someone can share light on this.
--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/11440241-e4f6-40a3-9698-e65498d3db3a%40googlegroups.com.

--
Brian Brazil
www.robustperception.io

Brian Brazil

unread,

Apr 10, 2020, 3:55:54 AM4/10/20

to Vidur, Prometheus Users

On Fri, 10 Apr 2020 at 07:10, Vidur <vidur...@getfareye.com> wrote:

On Wednesday, April 8, 2020 at 1:58:01 PM UTC, Brian Brazil wrote:
On Wed, 8 Apr 2020 at 13:40, Vidur <vidur...@getfareye.com> wrote:
Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application
void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
You shouldn't be registering metrics to the default registry in a typical method. Did you mean to register it to the custom registry you just instantiated?
My use case is something which sets gauge value for a same request in fixed interval, so is there no need to register the gauge ?

In that case you're doing normal direct instrumentation, so you shouldn't be using the pushgateway.

And Gauge are created real time with different request values , so i cannot make them static.

Metric names in direct instrumentation should never be procedurally generated. Most likely you should be using a label here, not a metric name.

Brian

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6a132978-c86e-4370-9dc5-51495e2c7956%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

Vidur

unread,

Apr 10, 2020, 5:27:28 AM4/10/20

to Prometheus Users

On Friday, April 10, 2020 at 7:55:54 AM UTC, Brian Brazil wrote:

On Fri, 10 Apr 2020 at 07:10, Vidur <vidur...@getfareye.com> wrote:
On Wednesday, April 8, 2020 at 1:58:01 PM UTC, Brian Brazil wrote:
On Wed, 8 Apr 2020 at 13:40, Vidur <vidur...@getfareye.com> wrote:
Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application
void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
You shouldn't be registering metrics to the default registry in a typical method. Did you mean to register it to the custom registry you just instantiated?
My use case is something which sets gauge value for a same request in fixed interval, so is there no need to register the gauge ?
In that case you're doing normal direct instrumentation, so you shouldn't be using the pushgateway.

But all different metrics have different interval which can be changed , thats why I'm using pushgateway .Otherwise i have to store those states somewhere else and create endpoint in my system for prometheus to pull.

And Gauge are created real time with different request values , so i cannot make them static.

Metric names in direct instrumentation should never be procedurally generated. Most likely you should be using a label here, not a metric name.

Are you suggessting me to use labels to distinguish between different type of metric.
could you share more info on direct instrumentation & other types.

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/6a132978-c86e-4370-9dc5-51495e2c7956%40googlegroups.com.

--
Brian Brazil
www.robustperception.io

Brian Brazil

unread,

Apr 10, 2020, 6:19:38 AM4/10/20

to Vidur, Prometheus Users

On Fri, 10 Apr 2020 at 10:27, Vidur <vidur...@getfareye.com> wrote:

On Friday, April 10, 2020 at 7:55:54 AM UTC, Brian Brazil wrote:
On Fri, 10 Apr 2020 at 07:10, Vidur <vidur...@getfareye.com> wrote:
On Wednesday, April 8, 2020 at 1:58:01 PM UTC, Brian Brazil wrote:
On Wed, 8 Apr 2020 at 13:40, Vidur <vidur...@getfareye.com> wrote:
Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application
void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
You shouldn't be registering metrics to the default registry in a typical method. Did you mean to register it to the custom registry you just instantiated?
My use case is something which sets gauge value for a same request in fixed interval, so is there no need to register the gauge ?
In that case you're doing normal direct instrumentation, so you shouldn't be using the pushgateway.
But all different metrics have different interval which can be changed , thats why I'm using pushgateway .Otherwise i have to store those states somewhere else and create endpoint in my system for prometheus to pull.

Metrics don't have an interval, events do. The frequency prometheus samples at is independent of when metrics are updated.

And Gauge are created real time with different request values , so i cannot make them static.

Metric names in direct instrumentation should never be procedurally generated. Most likely you should be using a label here, not a metric name.

Are you suggessting me to use labels to distinguish between different type of metric.
could you share more info on direct instrumentation & other types.

Use a normal static gauge and expose it out over http.

Brian

To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/44269654-2d97-484f-bbb4-dd0096e602cc%40googlegroups.com.

--

Brian Brazil

www.robustperception.io

Vidur

unread,

Apr 13, 2020, 2:15:13 AM4/13/20

to Prometheus Users

On Friday, April 10, 2020 at 10:19:38 AM UTC, Brian Brazil wrote:

On Fri, 10 Apr 2020 at 10:27, Vidur <vidur...@getfareye.com> wrote:
On Friday, April 10, 2020 at 7:55:54 AM UTC, Brian Brazil wrote:
On Fri, 10 Apr 2020 at 07:10, Vidur <vidur...@getfareye.com> wrote:
On Wednesday, April 8, 2020 at 1:58:01 PM UTC, Brian Brazil wrote:
On Wed, 8 Apr 2020 at 13:40, Vidur <vidur...@getfareye.com> wrote:
Hi
I'm using prometheus simple client for collecting metrics in my system via a spring boot application
void insertBatchJob(String request , String[] labelNames, String[] labelValues,String counter) throws Exception {
        CollectorRegistry registry = new CollectorRegistry();
        Gauge inprogressRequests ;
        if( !gaugeRegiseryMap.containsKey(request) ) {
            inprogressRequests = Gauge.build()
                    .name(request).labelNames(labelNames).help(request).register();
You shouldn't be registering metrics to the default registry in a typical method. Did you mean to register it to the custom registry you just instantiated?
My use case is something which sets gauge value for a same request in fixed interval, so is there no need to register the gauge ?
In that case you're doing normal direct instrumentation, so you shouldn't be using the pushgateway.
But all different metrics have different interval which can be changed , thats why I'm using pushgateway .Otherwise i have to store those states somewhere else and create endpoint in my system for prometheus to pull.
Metrics don't have an interval, events do. The frequency prometheus samples at is independent of when metrics are updated.

And Gauge are created real time with different request values , so i cannot make them static.

Metric names in direct instrumentation should never be procedurally generated. Most likely you should be using a label here, not a metric name.

Are you suggessting me to use labels to distinguish between different type of metric.
could you share more info on direct instrumentation & other types.

Use a normal static gauge and expose it out over http.

But there are dynamic labels for every metric type , so i think keeping a normal static gauge wont solve this problem.

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/44269654-2d97-484f-bbb4-dd0096e602cc%40googlegroups.com.

--
Brian Brazil
www.robustperception.io

Stuart Clark

unread,

Apr 13, 2020, 2:52:32 AM4/13/20

to promethe...@googlegroups.com

On 13/04/2020 07:15, Vidur wrote:

And Gauge are created real time with different request values , so i cannot make them static.

Metric names in direct instrumentation should never be procedurally generated. Most likely you should be using a label here, not a metric name.

Are you suggessting me to use labels to distinguish between different type of metric.
could you share more info on direct instrumentation & other types.

Use a normal static gauge and expose it out over http.

But there are dynamic labels for every metric type , so i think keeping a normal static gauge wont solve this problem.

What do you mean by dynamic labels?

While the values of labels can change during the application run (but should be chosen carefully to keep cardinality in check) the actual label names should be fixed for a particular metric.

Stuart Clark

Vidur

unread,

Apr 13, 2020, 2:55:05 AM4/13/20

to Prometheus Users

I mean different labels for different metric only , but can a single gauge instance be used for that.
As the metrics can be created during application run with its own set of labels.

Stuart Clark

Reply all

Reply to author

Forward