Multi container scraping

[Tiago Barbosa]

Hello everyone!

I´m working an environment where Prometheus version 1.8 is scraping k8s using a POD role. Job description in PS.

However my aim is go deeper over k8s hierarchy. I would like to set a custom Prometheus configuration per container. I would like to have this kind of input on a monitored k8s pod yml(or equivalent):

annotations:
 prometheus.io/container1: {"path":"/container1/v2/metrics","port":"8080","protocol":"http"}
 prometheus.io/container2: {"path":"/container2/metrics","port":"8081","protocol":"http"}
 prometheus.io/container2: {"path":"/metrics","port":"8082","protocol":"http"}

  

My goal is to have a Prometheus job scraping configured annotated containers as listed above: on such given path, port and protocol. Naturally, I have pods with 0..n monitored containers.

I did some experiments on relabeling config but I couldn't find anything that works for the above scenario.

Does anybody know how to perform this configuration?

PS: this is my job: 

- job_name: 'kubernetes-pods'
      scheme: https
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Allow the override of scheme with k8s pod annotation 'prometheus.io/scheme'
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: kubernetes_container_name
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: kubernetes_pod_node_name
      - replacement: @region@
        target_label: monitoring_region
      - replacement: @cluster_label@
        target_label: monitoring_cluster

[Bartłomiej Płotka]

Hello,

Please see the following topic for the answer and potential solutions: https://groups.google.com/forum/#!topic/prometheus-users/ihMUWtX477Q