Drop labels defined by service-discovery

[Carsten Buchberger]

We are looking for a way  to drop scrpaed labels dynamically. We are using consul-service discovery for scraping various targets/exporters.

We are trying to define re-usable scrapeconfigs, where most parameters are defined in the consul-service definition. That works pretty well, we are also able to define custom-labels that way.

But we are struggeling when trying to delete labels from the scraped metrics. We have cases where the exporter exposes labels we don't want. But instead of defining separate scrape_configs with hard-codes drop-label actions, we want to re-use our scrape-config.

We tried the following

We are using a technique like the one described here (https://www.robustperception.io/extracting-full-labels-from-consul-tags) to extract all labels that should be removed fom a comma-separated-string. This string comes from our consul service-discovery.

We apply those relabeling in metric_relabel_configs. To remove the labels we set the value to an empty string.

It looks like that

relabel_configs:

    - source_labels: [__meta_consul_service_metadata_labeldrop]
      target_label: 'labeldrop'

metric_relabel_configs:

- source_labels: [labeldrop]

  regex: '(?:[^,]+,){0}([^,]+).*'

  replacement: ''

  target_label: ${1}

- source_labels: [labeldrop]

  regex: '(?:[^,]+,){1}([^,]+).*'

  replacement: ''

  target_label: ${1}

and so on

But unfortunately it does not work, the labels are still there. After investigating a little bit further it seems like using an empty replacement in combination with the regex-capturing-group causes problems.

When e.g. using a static value like 'val' for the replacement all labels from our "list" get that value.

I opened an issue for this (https://github.com/prometheus/prometheus/issues/8186), but was told it is an configuration issue.

Maybe that is not the right way to approach out problem - but i am not able to find a solution (besides having multiple scrape_configs with different droplabel actions)

Any help is appreciated - thank you