Dynamically add Labels prometheus go-client

[Gustavo Neves]

Hi Guys,

I hope you are having a good weekend, I would like to have an information about CounterVec, is there a way to dynamically add Labels inside an already registered metric?

For example:

Creating a metric using two labels: status and environment

```go

metricName := "test_purposes"

help := "tests"

counter = promauto.NewCounterVec(prometheus.CounterOpts{

Name: metricName,

Help: help,

}, []string{"status", "environment"})

counter.WithLabelValues("success", "dev")

```

But when the status is error I would like to add one more label: reason, to the same metric and it's not possible

```go

counter = promauto.NewCounterVec(prometheus.CounterOpts{

Name: metricName,

Help: help,

}, []string{"status", "environment", "reason"})

counter.WithLabelValues("success", "dev", "npe")

```

Just to explain why I need it, we're migrating an implementation of datadog to prometheus and in the implementations based on datadog we have a lot of microservices sending metrics using different tags (labels) to the same metric, if I could do the same using the prometheus lib it would be great because we could only change the client without any modifications on metric publishers.

[Brian Candler]

> But when the status is error I would like to add one more label: reason, to the same metric and it's not possible

That's because it wouldn't be the same metric.

In Prometheus, it's the metric name together with the complete set of labels which form the identity of a timeseries.  If you add another label, then you've created a completely different timeseries.

The normal way to handle this is to create separate counters (metrics) for each outcome:

test_purposes{result="success",environment="dev"} 0

test_purposes{result="npe",environment="dev"} 0

test_purposes{result="oops",environment="dev"} 0

Then you increment the appropriate one, depending on the result.

You could also use your scheme:

test_purposes{status="success",environment="dev"} 0

test_purposes{status="error",reason="npe",environment="dev"} 0

test_purposes{status="error",reason="oops",environment="dev"} 0

Again you'd just create three counters as three separate metrics.  Personally I find the first version easier to work with in queries, because it's easier to aggregate when the labels are consistent, but either way will work.

[Gustavo Neves]

Hi Brian, 

Thank you so much for your answer. I appreciate and agree with your comments, for my use case using the second approach would be nice but using the golang official lib https://github.com/prometheus/client_golang it's not possible to create the same metric more than once even using different labels.

Thank you.

[Brian Candler]

For any label you don't want, set the label value to empty string.

[Brian Candler]

Trying it out:

--------

package main

import (
        "flag"
        "log"
        "net/http"

        "github.com/prometheus/client_golang/prometheus"
        "github.com/prometheus/client_golang/prometheus/promhttp"
)

var addr = flag.String("listen-address", ":8080", "The address to listen on for HTTP requests.")

func main() {

        metricName := "test_purposes"
        help := "tests"

        counter := prometheus.NewCounterVec(prometheus.CounterOpts{

                Name: metricName,
                Help: help,
        }, []string{"status", "environment", "reason"})

        prometheus.MustRegister(counter)
        counter.WithLabelValues("success", "dev", "")
        counter.WithLabelValues("error", "dev", "npe")

        flag.Parse()
        http.Handle("/metrics", promhttp.Handler())
        log.Fatal(http.ListenAndServe(*addr, nil))
}

--------

Result:

# HELP test_purposes tests
# TYPE test_purposes counter
test_purposes{environment="dev",reason="",status="success"} 0
test_purposes{environment="dev",reason="npe",status="error"} 0

So, it does actually expose an empty value label - but prometheus should treat that the same as the label not existing.

Personally, I'd much prefer that a given metric comes with a consistent set of labels - and I think the API is trying to encourage that.

--------

        counter.With(prometheus.Labels{"status": "success", "environment": "dev"})
        counter.With(prometheus.Labels{"status": "error", "environment": "dev", "reason": "npe"})

--------

=> panic: inconsistent label cardinality: expected 3 label values but got 2 in prometheus.Labels{"environment":"dev", "status":"success"}