Append new labels to existing list of lables in prometheus metrics

[Hammad Akhtar]

Note: I've also asked the same question on stackoverflow as well [here]. I'd appreciate some pointers to docs.

Question Begins

I've an existing histogram metric which is being published with **job_type** label

```go
var executionTime = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "job_execution_time",
        Help: "Histogram for job execution time",
    },
    []string{"job_type"},
)
```

I'm publishing metric in the following manner

```go
executionTime.WithLabelValues("aws_lambda").Observe(timeInSeconds)
```


Note that this metric is already getting published. However now I want to append a new label **cloud_provider** to the existing metric like below.

```go
var executionTime = prometheus.NewHistogramVec(
    prometheus.HistogramOpts{
        Name: "job_execution_time",
        Help: "Histogram for job execution time",
    },
    []string{"job_type", "cloud_provider"},
)
```

1. Does Prometheus supports such mutation of metrics?
2. If no why is it not supported? If yes are there any implications on metric data stored for previous combinations of labels?

[Stuart Clark]

On 25/02/2021 21:43, Hammad Akhtar wrote:
>
>
> 1. Does Prometheus supports such mutation of metrics?
> 2. If no why is it not supported? If yes are there any implications on
> metric data stored for previous combinations of labels?

Every time series (different combination of labels) is unique and
treated separately, so replacing a metric that used to have 1 label with
one than now has 2 just results in the old time series no longer getting
updates while some totally new time series start having values.

In terms of the implications it is mostly around the effect on any
queries you produce. If your query relies on the new label then it might
not return values before the switch over, while queries that worked on
the old version may now return unexpected values after the change. So
you may need to update your dashboards or alerting rules accordingly.
Very large changes to how the metric is presented could be confusing, so
in those situations it might make sense to change the name to explicitly
split things into a "before" and "after".

--
Stuart Clark