Metrics relabeling issue

[Edoardo Beltramo]

Hello,

I’m facing an issue where I can’t properly save a metric;
Inside a k8s cluster I have a rule inside a servicemonitor.yaml file that drops any metric coming from namespaces that match a specific regex, something like:

- sourceLabels: [namespace]   regex: z{3,6}   action: drop

this works, but now I have to implement a way to save a single metric, specifically kube_namespace_created before dropping all the others, only for those namespace that match the regex.

Can someone give me an heads up? Thanks

[Brian Candler]

The solution should be the same as this recent thread: https://groups.google.com/g/prometheus-users/c/gy510HjV-Sw

1. Set a label like __tmp_keep to any string, for those metrics you want to keep

- sourceLabels: [__name__]   regex: kube_namespace_created  replacement: 'true'  target: __tmp_keep

2. Then only drop those for which __tmp_keep is empty (unset)

- sourceLabels: [namespace,__tmp_keep]   regex: 'z{3,6};'  action: drop

[Brian Candler]

Sorry, "target" probably should be "targetLabel" in your k8s setup (it's "target_label" in regular prometheus)

[Edoardo Beltramo]

Hi Brian,

thanks for the answer,

according to your script I should be able to save the metric I need inside '__tmp_keep' and then being able to use it anyhow I need right?

[Brian Candler]

You can't put a metric "inside" __tmp_keep; rather, it's a label added to the label set of the existing metric.

For example, it would change

    mymetric{foo="bar"} 123.4

to

    mymetric{foo="bar",__tmp_keep="true"} 123.4

Labels which start with double-underscore are silently removed at the end of the relabelling phase, and labels starting with __tmp are guaranteed not to clash with any labels used by prometheus internally.