relabel for scrapping port issue

[Eleanore Jin]

Hi, 

I am running in kubernetes, and some of the pods uses annotation:

      annotations:

        prometheus.io/path: /

        prometheus.io/port: "9250"

        prometheus.io/scrape: "true"

however, for some other pods, we are not using annotation, but simply define:

        ports:

        - containerPort: 8889

          name: metrics

and the config as below, after adding __meta_kubernetes_pod_container_port_name, the ones with just annotation stopped working. Prometheus no longer scrape from those pods, any suggestions?

Thanks!

- job_name: 'kubernetes-pod-annotation'
  kubernetes_sd_configs:
  - role: pod
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__meta_kubernetes_pod_container_port_name] # try to keep the port only if the name is metrics
    action: keep
    regex: metrics
  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
    action: replace
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
    target_label: __address__
  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod_name

[Brian Candler]

  - source_labels: [__meta_kubernetes_pod_container_port_name] # try to keep the port only if the name is metrics
    action: keep
    regex: metrics

will drop any target that doesn't have that label with that value.  If you want to keep targets that *don't* have that label, then allow empty string as a match:

  - source_labels: [__meta_kubernetes_pod_container_port_name] # try to keep the port only if the name is metrics
    action: keep

    regex: 'metrics|'

[Eleanore Jin]

Thanks a lot! That works!