Prometheus Federated Istio not replacing metric names

[Roee Landesman]

Hi all,

I'm attempting to do a basic metric relabeling where I replace the __name__ by removing a prefix (specifically workload:). This is taken directly from the upstream istio documentation found here. However, I am able to create a new metric without the workload prefix, but am totally unable to either replace nor drop the existing workload metric in order to save space. This is further confirmed when looking at the TSDB status endpoint of my main prometheus.

My ServiceMonitor:

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: istio-federation

namespace: prometheus-system

labels:

app.kubernetes.io/name: istio-prometheus

spec:

namespaceSelector:

matchNames:

- istio-system

selector:

matchLabels:

release: prometheus-istio

endpoints:

- interval: 30s

scrapeTimeout: 30s

params:

'match[]':

- '{__name__=~"workload:(.*)"}'

path: /federate

targetPort: 9090

honorLabels: true

metricRelabelings:

- sourceLabels: ["__name__"]

regex: 'workload:(.*)'

targetLabel: "__name__"

action: replace

Any help or pointers in this case would be super appreciated