Metrics drop based on labels

[Mohan Nagandlla]

Hi Team,

I just want to drop the metrics from federation instnace

by enabling the relabel concept but its not success

the sample metrics is

up{container="prometheus", endpoint="9090", exported_exported_exported_exported_exported_instance="20.xxxxx......}  

up{container="prometheus", endpoint="9090", exported_instance="20.xxxxxxx.................}

I dont want to see this metrics which is having more than 1-time exported_exported_.* i need the metrics which contain only exported_instance label

I have tries this

- job_name: "aws-prometheus"

  metrics_path: '/federate'

  honor_timestamps: true

  scrape_interval: 30s

  scrape_timeout: 10s

  params:

      'match[]':

        - '{job=~".*"}'

  relabel_configs:

    - source_labels: [__name__]

      regex: (exported_exported_.*)

      action: drop

  static_configs:

  - targets: ["k8s-mohan.prometheus.com"]

[Brian Candler]

relabel_configs is *target* relabelling.  This processes the results of the service discovery phase only, *before* the scrape even takes place.  Since you are using static service discovery, it's clear that no labels are coming from here, therefore they must be coming from the scraped data, i.e. the remote system you are federating from.

To process the scraped metrics you need metric relabelling instead (i.e. "metric_relabel_configs").  You can drop metrics you don't want, or alter their labels.  However the expression you've given won't work, because label __name__ contains the metric name (i.e. "up" in your example).  Matching unknown label names against a pattern is not something I've done before, but I think you should be able to do it using "labelmap" action to set a temporary label as a marker (say "__tmp_drop"), then "drop" action to drop the entire metric if that label is set.

  

A more efficient option is to avoid scraping those metrics in the first place, which you could do easily if the label is always exactly the same:

 params:

      'match[]':

        - '{job=~".*",exported_exported_exported_exported_exported_instance=""}'

This works because PromQL treats a non-existent label the same as if it were a label with empty value.

If possible, I would also question where all those exported_exported_exported_...blah labels are coming from , and try to clean this upstream.  I realise this isn't always possible, but it does seem like something is wrong here.