dropping a label

[GI D]

Running Prometheus on K8s v1.20 on AWS/EKS, In my v33 Helm chart I need to drop the ID label in order to reduce the TSDB size. According to this article, this can be done with Metric Relabelings. So in all relevant sections of the values file I have the following:

metricRelabelings:

  - sourceLabels: [id]

     action: "drop"

In the resulting Prometheus config that I can see in the web GUI on port 9090, this gets translated (again in all relevant sections) as follows:

metric_relabel_configs:

  - source_labels: [id]      

       separator: ;

       regex: (.*)

       replacement: $1

       action: drop

However in the TSDB status in the GUI the "id" label is still there, with the same cardinality as before the attempt to drop it.

What am I missing?

[Julius Volz]

To drop a label from a series during the scrape, you need the "labeldrop" action ("drop" drops the entire series), see https://training.promlabs.com/training/relabeling/writing-relabeling-rules/keeping-and-dropping-labels

However, just dropping a label that is required to distinguish series from each other will cause you problems if that results in multiple time series now having the same labelset identity after the relabeling. Maybe your intent is to aggregate over multiple series instead?

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/06058e34-00e4-4034-a572-bb45098f7d3en%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[GI D]

Thanks for answering,

Following what you wrote, I now have the following multiple times in my config:

metric_relabel_configs:

   - separator: ;

     regex: id

     replacement: $1

    action: labeldrop

However, and after restarting the Prometheus stateful set,  in my TSDB status the ID label is still there, with the same cardinality as before, in the table titled: "Top 10 label names with high memory usage"

Please advise.

[Julius Volz]

Hi,

Is the label really a label coming directly from the instrumentation, that is, the target's /metrics page, or is it a target label (in which case you'll want to use relabel_configs, not metric_relabel_configs)? What kind of ID is it?

Regards,

Julius

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/8d45baf5-e25c-474d-8f7a-b30a6035a48cn%40googlegroups.com.

[GI D]

It is my understanding that this ID label is a built-in, "intrinsic" label for Prometheus. It is not a target label.

[Brian Candler]

That's incorrect.  The only "instrinsic" labels for prometheus are the "instance" and "job" labels, which are added at scrape time. (The "instance" label is copied from "__address__", but only if "instance" has not already been set)

So your "id" label is *either* coming from your service discovery mechanism (i.e. added *prior* to scrape, in which case you need to use "relabel_configs" to modify it), or it is coming from the exporter (in which case use "metric_relabel_configs").

It's easy to tell which: perform a manual scrape using "curl", and see if the "id" label is present in the metrics returned.

If you show the configuration of the job in prometheus.yml, it will be clear which SD mechanism you're using and that may explain the issue.

[GI D]

OK, I stand corrected.

Here is the entire config file.

[Brian Candler]

There are 17 scrape jobs in that config.  Can you show an example of a metric that has the problem? Then the "job" label will identify which job it came from, and we'll also see the format of the "id" label you are concerned about.

I don't see any "target_label: id" in there, and I'm pretty sure kubernetes_sd_config doesn't add an "id" label, so it seems likely that it's coming from the exporter as part of the scrape.

[GI D]

The way I test this is by looking at the TSDB status on the web GUI and looking for the ID label in the tables with the labels with the highest cardinality. I keep on finding the ID label there, with its cardinality unchanged from before the drop.  I expect to not find it, or to find it with a much lower cardinality.

Please suggest a more targeted way of testing this.

[Brian Candler]

As Julius said, just dropping the id label may break things - if you have multiple timeseries which are distinguished by the id label alone, then dropping the id will result in multiple clashing metrics.

Maybe you want to combine multiple metrics into a single metric?  Showing examples of the *actual* metrics may be able to show what's going on - where the id is coming from, and what might be an appropriate aggregation to do.