Help with metric_relabel_config

[Ryan Ash]

I am using Prometheus to monitor a Kubernetes cluster.  Prometheus has been deployed using the CoreOS Prometheus-Operator and Kube-Prometheus.  

I have used the ServiceMonitor to deploy a metric_relabel_config as follows.  

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

  labels:

    k8s-app: node-exporter

  name: node-exporter

  namespace: monitoring

spec:

  endpoints:

  - bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

    interval: 30s

    port: https

    scheme: https

    tlsConfig:

      insecureSkipVerify: true

    metricRelabelings:

    - sourceLabels: [__meta_kubernetes_pod_node_name]

      targetLabel: hostname

      regex: (\w+).*

      replacement: $1

  jobLabel: k8s-app

  namespaceSelector:

    matchNames:

    - monitoring

  selector:

    matchLabels:

      k8s-app: node-exporter

This is mapping fine into Prometheus as follows:

  metric_relabel_configs:
  - source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (\w+).*
    target_label: hostname
    replacement: $1
    action: replace

Because these are done at ingestion time I didn't expect to see the new label on the <server>/service-discovery or /targets.  is that correct?

What I did expect was the ability to query using the new label.  

avg(node_load1) by (hostname)

The above query returns a single chart for "{}"

Am I missing some aspect of this process?   

[Julius Volz]

Correct.

 

What I did expect was the ability to query using the new label.  

avg(node_load1) by (hostname)

The above query returns a single chart for "{}"

One problem here is that the "__meta_*" labels are only available during target relabeling, as they are removed at the end of the relabeling phase. See https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Crelabel_config%3E:

"Labels starting with __ will be removed from the label set after relabeling is completed."

You could copy that meta label to a persisted (non-"__") target label during target relabeling and then use that instead. But if the label is the same for the entire target anyway, why even use metric relabeling and not normal (target) relabeling to apply that label to all series for that target? Hmm, actually looking at https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md I don't know how to do custom target relabeling with the Prometheus Operator, I only see metric relabeling there. No idea if it's possible... maybe the Operator people have better suggestions.

[i...@whitehat.com.ua]

Hi Ryan,

I'm just curious if you solved this problem? I cannot find any solution except for forking/patching prometheus-operator :(