Metric relabel configs in Promtheus not working as expected

[墨生]

recently I'm trying to drop some metrics since prometheus usage is getting way too much than I expected.
```
metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: (istio_requests_total)
    replacement: $1
    action: drop

```
But I tried this fairly straight forward metric_relabel_configs, "istio_requests_total" doesn't drop as expected. I know you may ask why I don't drop this in relabeling.

My very first goal is to drop all the metrics with destination_service = unknown associated with istio like :
```
  - source_labels: [destination_service]
    separator: ;
    regex: ^unknown(.*)
    replacement: $1
    action: drop
```
This doens't work ,so I go back to a simple "metric" drop see if this can work. Anyways let me know if I made anything wrong and thank you all in advance !

[Brian Candler]

On Monday, 16 May 2022 at 08:51:05 UTC+1 yuuk...@gmail.com wrote:

metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: (istio_requests_total)
    replacement: $1
    action: drop

I think that should work, although "replacement" is superfluous here (because it doesn't do anything with "drop" action), and "separator" is also unnecessary (because semicolon is the default, and you're not joining multiple labels anyway).

Did you tell prometheus to reload its config after changing this? Is it under the correct scrape job? Does it produce any errors on reloading?

 

```
But I tried this fairly straight forward metric_relabel_configs, "istio_requests_total" doesn't drop as expected. I know you may ask why I don't drop this in relabeling.

I don't understand that last sentence.  Metric relabelling is the only place you can drop individual metrics, post-scrape.  Target relabelling can only be used to drop entire targets, pre-scrape.

 

My very first goal is to drop all the metrics with destination_service = unknown associated with istio like :
```
  - source_labels: [destination_service]
    separator: ;
    regex: ^unknown(.*)
    replacement: $1
    action: drop

For safety it's best to quote any strings which contain special characters, just in case YAML decoding treats them specially. I don't think there's a problem with what you've written, but I would use:

 - source_labels: [destination_service]
    regex: '^unknown(.*)'
    action: drop