Go equivalent of Python Counter... labelnames != ConstLabels

237 views
Skip to first unread message

Devin Bost

unread,
Jan 14, 2020, 12:17:55 AM1/14/20
to Prometheus Users
I am attempting to migrate a Python implementation of the Prometheus Client to an implementation of Go with identical functionality (to add Go language support to an open source framework), but I'm a little confused by the differences in the Python and Go APIs for Prometheus. 

In my Python code, I have many lines like this:

metrics_label_names = ['tenant', 'namespace', 'name', 'instance_id', 'cluster', 'fqfn']
stat_total_processed_successfully = Counter(PULSAR_FUNCTION_METRICS_PREFIX + TOTAL_SUCCESSFULLY_PROCESSED,
                                            'Total number of messages processed successfully.', metrics_label_names)
stat_total_sys_exceptions = Counter(PULSAR_FUNCTION_METRICS_PREFIX+ TOTAL_SYSTEM_EXCEPTIONS, 'Total number of system exceptions.',
                                    metrics_label_names)
stat_total_user_exceptions = Counter(PULSAR_FUNCTION_METRICS_PREFIX + TOTAL_USER_EXCEPTIONS, 'Total number of user exceptions.',
                                     metrics_label_names)

However, when looking at the NewCounter constructor for Go, it expects to be provided with an Ops struct, which lacks the labelnames field:

type Opts struct {
    Namespace string
    Subsystem string
    Name      string
    Help string
    ConstLabels Labels
}

(https://godoc.org/github.com/prometheus/client_golang/prometheus#Opts), when compared to the Python constructor for Counter, which looks like this:

def __init__(self,
    name,
    documentation,
    labelnames=(),
    namespace='',
    subsystem='',
    unit='',
    registry=REGISTRY,
    labelvalues=None,
    ):



Of these, I assume that:
  •  name (in Python) = Name (in Go)
  •  document (in Python) = Help (in Go)
However, labelnames (in Python) != ConstLabels (in Go).

The labelnames field in Python allows me to pass an array, but for Go, ConstLabels is a Labels type, which is a string-> string map. So, I have no idea what the equivalent is. If the labelnames are the keys, then what would be the values? 

Can someone please help me understand the differences and what my migration path looks like?

Thanks,

Devin



Devin Bost

unread,
Jan 14, 2020, 1:04:06 AM1/14/20
to Prometheus Users
I also need to know if there's a Go equivalent of the 

def labels(self, *labelvalues, **labelkwargs):


because there are a lot of usages of that method in the code I need to migrate. 

Thanks

Devin

Brian Brazil

unread,
Jan 14, 2020, 3:36:02 AM1/14/20
to Devin Bost, Prometheus Users

Brian


The labelnames field in Python allows me to pass an array, but for Go, ConstLabels is a Labels type, which is a string-> string map. So, I have no idea what the equivalent is. If the labelnames are the keys, then what would be the values? 

Can someone please help me understand the differences and what my migration path looks like?

Thanks,

Devin



--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/fe228dd4-8f00-438e-9043-133326ae5c08%40googlegroups.com.


--

Devin Bost

unread,
Jan 14, 2020, 11:55:17 AM1/14/20
to Brian Brazil, Prometheus Users
Thanks! 

That answers my first question. 

What about this method? 


def labels(self, *labelvalues, **labelkwargs):


It's not clear to me what it's used for. 
--
Devin G. Bost

Brian Brazil

unread,
Jan 14, 2020, 12:08:43 PM1/14/20
to Devin Bost, Prometheus Users
On Tue, 14 Jan 2020 at 16:55, Devin Bost <devin...@gmail.com> wrote:
Thanks! 

That answers my first question. 

What about this method? 


def labels(self, *labelvalues, **labelkwargs):


It's not clear to me what it's used for. 

That's WithLabelValues in Go.

Brian 

Devin Bost

unread,
Jan 14, 2020, 2:18:21 PM1/14/20
to Prometheus Users
Thanks again!

That brings me almost to completion.

The last ones I have to migrate are:
counter._value.get()
observer._count.get()
observer._sum.get()
counter.set(time)

How do I migrate these? 

Thanks for all the help,

Devin

On Tuesday, January 14, 2020 at 10:08:43 AM UTC-7, Brian Brazil wrote:
On Tue, 14 Jan 2020 at 16:55, Devin Bost <devi...@gmail.com> wrote:
Thanks! 

That answers my first question. 

What about this method? 


def labels(self, *labelvalues, **labelkwargs):


It's not clear to me what it's used for. 

That's WithLabelValues in Go.

Brian 
--
Devin G. Bost

On Tue, Jan 14, 2020, 1:35 AM Brian Brazil <brian...@robustperception.io> wrote:
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.


--

Brian Brazil

unread,
Jan 14, 2020, 4:11:16 PM1/14/20
to Devin Bost, Prometheus Users
On Tue, 14 Jan 2020 at 19:18, Devin Bost <devin...@gmail.com> wrote:
Thanks again!

That brings me almost to completion.

The last ones I have to migrate are:
counter._value.get()
observer._count.get()
observer._sum.get()

These don't exist in Go, and you really shouldn't be poking around the internals of the Python client like that. Why do you want to get the values of counters?
 

counter.set(time)

This should not be possible.

It sounds like you should be writing a customer collector, for which you'll want MustNewConstMetric in Go.

Brian

 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/f96b5710-f1a5-4178-910d-cb003f47fb84%40googlegroups.com.


--

Devin Bost

unread,
Jan 14, 2020, 4:18:06 PM1/14/20
to Prometheus Users
Take a look at our example Python implementation here: https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/python/function_stats.py#L128

Is there a better way for us to get this data back to our end-users? 

Regarding:

-Devin

On Tuesday, January 14, 2020 at 2:11:16 PM UTC-7, Brian Brazil wrote:

Brian Brazil

unread,
Jan 14, 2020, 4:27:18 PM1/14/20
to Devin Bost, Prometheus Users
On Tue, 14 Jan 2020 at 21:18, Devin Bost <devin...@gmail.com> wrote:
Take a look at our example Python implementation here: https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/python/function_stats.py#L128

Is there a better way for us to get this data back to our end-users? 

That looks to be trying to build a different style of instrumentation client on top of a Prometheus client. In addition having all metrics for an application centralised in one file is a smell, just as it would be if you did it for all application logging.

I'd suggest moving the counters to the files where they're actually used, getting rid of the _1min metrics (they're redundant), and using seconds. You shouldn't have to do any math, gets, or resets when doing Prometheus instrumentation.
That's a gauge, not a counter. That translates directly to Go.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3480a316-748c-4351-8a15-1ac88971fd24%40googlegroups.com.


--

Devin Bost

unread,
Jan 14, 2020, 4:39:37 PM1/14/20
to Prometheus Users

That looks to be trying to build a different style of instrumentation client on top of a Prometheus client. . . You shouldn't have to do any math, gets, or resets when doing Prometheus instrumentation.

It is true that we're building a different style of instrumentation client on top of a Prometheus client. However, we are also using the library to keep track of these statistics, and it's redundant to have another library keep track of these stats again. We have multiple ways users can get stats, such as: from Prometheus, REST, etc. Ideally, we use one library to keep track of all these stats and massage the data if needed to support other stats use cases. We are doing the same thing in our Java client, such as here: https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/FunctionStatsManager.java#L352

-Devin

On Tuesday, January 14, 2020 at 2:27:18 PM UTC-7, Brian Brazil wrote:

Brian Brazil

unread,
Jan 14, 2020, 4:51:50 PM1/14/20
to Devin Bost, Prometheus Users
On Tue, 14 Jan 2020 at 21:39, Devin Bost <devin...@gmail.com> wrote:

That looks to be trying to build a different style of instrumentation client on top of a Prometheus client. . . You shouldn't have to do any math, gets, or resets when doing Prometheus instrumentation.

It is true that we're building a different style of instrumentation client on top of a Prometheus client. However, we are also using the library to keep track of these statistics, and it's redundant to have another library keep track of these stats again. We have multiple ways users can get stats, such as: from Prometheus, REST, etc. Ideally, we use one library to keep track of all these stats and massage the data if needed to support other stats use cases. We are doing the same thing in our Java client, such as here: https://github.com/apache/pulsar/blob/master/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/stats/FunctionStatsManager.java#L352

If you want to produce different output work from the registry, rather than complicating your instrumentation. The registry can give you a dump of all the current metric values, which is how exposition works. 

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8315f565-c137-4ad1-9886-5660f41f1bfe%40googlegroups.com.


--

Devin Bost

unread,
Jan 14, 2020, 5:21:52 PM1/14/20
to Brian Brazil, Prometheus Users
Do you have an example? 

--
Devin G. Bost

Brian Brazil

unread,
Jan 14, 2020, 5:40:50 PM1/14/20
to Devin Bost, Prometheus Users
On Tue, 14 Jan 2020 at 22:21, Devin Bost <devin...@gmail.com> wrote:
Do you have an example? 

The graphite bridges in the various client libraries are one example.

Devin Bost

unread,
Jan 14, 2020, 7:02:06 PM1/14/20
to Prometheus Users
Does that require Prometheus to be running and wired up? In our current approach, even if Prometheus isn't running, we can still get the statistics from our service.

-Devin

On Tuesday, January 14, 2020 at 3:40:50 PM UTC-7, Brian Brazil wrote:
On Tue, 14 Jan 2020 at 22:21, Devin Bost <devi...@gmail.com> wrote:
Do you have an example? 

The graphite bridges in the various client libraries are one example.

Brian
 

--
Devin G. Bost

On Tue, Jan 14, 2020, 2:51 PM Brian Brazil <brian...@robustperception.io> wrote:


--

Brian Brazil

unread,
Jan 14, 2020, 7:20:37 PM1/14/20
to Devin Bost, Prometheus Users
On Wed, 15 Jan 2020 at 00:02, Devin Bost <devin...@gmail.com> wrote:
Does that require Prometheus to be running and wired up? In our current approach, even if Prometheus isn't running, we can still get the statistics from our service.

No, it's all internal to the client library.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9e4637b6-bd18-49de-b76d-34106773d898%40googlegroups.com.


--

Devin Bost

unread,
Jan 14, 2020, 8:07:17 PM1/14/20
to Prometheus Users
So, to get the metrics from the Registry, do I call:

mfs, err := reg.Gather()

and then filter the MetricFamily for each case where I previously had a "get" operation? 
If so, how do I filter it from there? By the name of the metric? 


On Tuesday, January 14, 2020 at 5:20:37 PM UTC-7, Brian Brazil wrote:

Devin Bost

unread,
Jan 14, 2020, 8:58:15 PM1/14/20
to Prometheus Users
I have several concerns with the suggested approach. 
First of all, based on the example from the graphite bridge, it appears that I would need to filter the MetricFamily instances like this:

mfs, err := reg.Gather()
vec, err2 := expfmt.ExtractSamples(&expfmt.DecodeOptions{
    Timestamp: now,
}, mfs...)

How am I supposed to know what Timestamp value to filter them on? 
Even if I did, how do I get my required information out of the resulting *model.Vector instances?
It appears that *model.Vector is really a model.Sample type, but model.Sample is comprised of Metric, SampleValue, and Time. Nowhere in there is it clear what parameter I can use to filter down to the information I'm looking for. 

Now, let's say that we figure out how to do that. Wouldn't it be prohibitively expensive to need to perform this kind of filter every single time we need to get the information from each of those getters? 

-Devin 

Brian Brazil

unread,
Jan 15, 2020, 3:49:28 AM1/15/20
to Devin Bost, Prometheus Users
On Wed, 15 Jan 2020 at 01:58, Devin Bost <devin...@gmail.com> wrote:
I have several concerns with the suggested approach. 
First of all, based on the example from the graphite bridge, it appears that I would need to filter the MetricFamily instances like this:

mfs, err := reg.Gather()
vec, err2 := expfmt.ExtractSamples(&expfmt.DecodeOptions{
    Timestamp: now,
}, mfs...)

How am I supposed to know what Timestamp value to filter them on? 

There won't be a timestamp.
 
Even if I did, how do I get my required information out of the resulting *model.Vector instances?
It appears that *model.Vector is really a model.Sample type, but model.Sample is comprised of Metric, SampleValue, and Time. Nowhere in there is it clear what parameter I can use to filter down to the information I'm looking for. 

Now, let's say that we figure out how to do that. Wouldn't it be prohibitively expensive to need to perform this kind of filter every single time we need to get the information from each of those getters? 

This is fine if you're for example taking the entire output and converting it to json, this is how /metrics works internally.

Brian
 
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/0b0a6e74-c1bc-4a74-bf65-3d50363ab1c9%40googlegroups.com.


--

Devin Bost

unread,
Jan 15, 2020, 1:35:56 PM1/15/20
to Brian Brazil, Prometheus Users
Okay, it sounds like it won't be too expensive. 

So, how do I filter down to the values I'm looking for? This is still not clear to me, and the examples haven't been sufficient.

Devin G. Bost

Devin Bost

unread,
Jan 15, 2020, 6:12:16 PM1/15/20
to Prometheus Users
ExtractSamples expects to be provided a DecodeOptions struct, which has Timestamp as its only property. 
What do you mean there won't be a timestamp?
I still don't understand how to filter down the values, and we will need to use something other than Prometheus to handle this logic if Prometheus isn't a viable solution for this task. 

-Devin


On Wednesday, January 15, 2020 at 11:35:56 AM UTC-7, Devin Bost wrote:
Okay, it sounds like it won't be too expensive. 

So, how do I filter down to the values I'm looking for? This is still not clear to me, and the examples haven't been sufficient.

Devin G. Bost


To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
Message has been deleted

Devin Bost

unread,
Jan 16, 2020, 3:45:20 PM1/16/20
to Prometheus Users, Brian Brazil
Is there at least documentation somewhere that describes how to do this? 

This is the only remaining thing I'd need to figure out for this implementation. 

Devin G. Bost


To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/NpkERPC17H4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e7589cfc-ddf4-45f0-b237-719312c911e3%40googlegroups.com.

Devin Bost

unread,
Jan 16, 2020, 9:51:33 PM1/16/20
to Prometheus Users
I figured this out after locating this: https://github.com/prometheus/client_golang/blob/master/prometheus/examples_test.go
and inspecting the data inside some of the tests that I cloned and ran locally. 

Thanks for the help. 

-Devin
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to a topic in the Google Groups "Prometheus Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/prometheus-users/NpkERPC17H4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to prometheus-users+unsubscribe@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages