Invalid Scrape Target in 'kubernetes-pods' for Pods with Multiple Containers

[illya.c...@gmail.com]

Hello, as advised I am asking the same question here.

When I run multi-container pod with specific prometheus port annotation and port definition in one container only, Prometheus creates targets entries in kubernetes-pods for all pods/containers, adding default port 80 for a container w/out port definition and which are not part of the "scrape" annotation.

More details here: https://github.com/prometheus/prometheus/issues/3530

[Brian Brazil]

Hmm, do you know which scrape_config is producing it? The next version of Prometheus should contain a feature to make debugging this sort of thing easier.

 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/46f362d5-c65a-4ef8-bf25-3dd9ad67f06f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Brian Brazil

www.robustperception.io

[illya.c...@gmail.com]

From the image ^ you can see that it attempts to scrape both containers in my pod, while only one has a pod (declared and annotated). 

[Simon Pasquier]

The documentation states that "If a container has no specified ports, a port-free target per container is created for manually adding a port via relabeling" [1].

IIUC to discard this additional target, you need to add something along those lines to the relabel_configs

     # Drop port-free targets

      - source_labels: [__meta_kubernetes_pod_container_port_number]            

        action: drop                                                            

        regex: ""    

I haven't tested though.

The other option is to use the endpoint role but maybe it doesn't work for you.

HTH

Simon

[1] https://prometheus.io/docs/prometheus/latest/configuration/configuration/#pod

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-users+unsubscribe@googlegroups.com.
To post to this group, send email to prometheus-users@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/29492b79-77c7-4abc-8fe6-8366c845526c%40googlegroups.com.

[illya.c...@gmail.com]

Simon, Thank you for the tip!

This is what I ended up using:

```

      - source_labels: [__meta_kubernetes_pod_container_port_number]            

        action: drop                                                            

        regex: ^$ 

```