kubernetes pod scraping with multiple ports

[Maxim Filatov]

Hello!

Here is an example about how to annotate pods to be prometheus-scrapable: https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L154-L161

It works like a charm in most cases, but there is kube-dns addon:

# curl 192.168.48.11:10055/metrics

# HELP go_gc_duration_seconds A summary of the GC invocation durations.

# TYPE go_gc_duration_seconds summary

go_gc_duration_seconds{quantile="0"} 0.000508958

go_gc_duration_seconds{quantile="0.25"} 0.001117607

…

# HELP skydns_skydns_dns_request_duration_seconds Histogram of the time (in seconds) each request took to resolve.

# TYPE skydns_skydns_dns_request_duration_seconds histogram

skydns_skydns_dns_request_duration_seconds_bucket{system="auth",le="0.001"} 897

skydns_skydns_dns_request_duration_seconds_bucket{system="auth",le="0.003"} 898

skydns_skydns_dns_request_duration_seconds_bucket{system="auth",le="0.005"} 898

…

# curl 192.168.48.11:10054/metrics

# HELP dnsmasq_cache_errors Number of errors that have occurred getting metrics

# TYPE dnsmasq_cache_errors counter

dnsmasq_cache_errors 0

…

So, we have one pod, but two ports for scraping. 

Is there any way to achieve this?

Thanks in advance.

--

Best regards,

Maxim Filatov

[asan...@gmail.com]

I'm also interested in knowing whats the best way to handle this.

Thanks.

[vteja...@gmail.com]

Its a nice, question, waiting to know the answer

[Björn Rabenstein]

On 10 December 2016 at 22:46, Maxim Filatov <pipo...@gmail.com> wrote:
>
> So, we have one pod, but two ports for scraping.
> Is there any way to achieve this?

This is definitely an FAQ (and we should document the answer more prominently).

https://github.com/prometheus/prometheus/issues/3756 is the most
recent incarnation I noticed, but there have been more discussions on
GitHub, IRC, and the mailing lists in the past.

(Side note: Seeing so many users repeatedly bump their head into the
more-than-one-port-to-scrape-on-a-K8s-pod wall, I'm tempted to use
Brian's usual concern about increased support effort to make us create
a simpler solution for the use case than the currently endorsed one.)

--
Björn Rabenstein, Engineer
http://soundcloud.com/brabenstein

SoundCloud Ltd. | Rheinsberger Str. 76/77, 10115 Berlin, Germany
Managing Director: Alexander Ljung | Incorporated in England & Wales
with Company No. 6343600 | Local Branch Office | AG Charlottenburg |
HRB 110657B

[Brian Brazil]

On 31 January 2018 at 13:29, 'Björn Rabenstein' via Prometheus Users <prometheus-users@googlegroups.com> wrote:

On 10 December 2016 at 22:46, Maxim Filatov <pipo...@gmail.com> wrote:
>
> So, we have one pod, but two ports for scraping.
> Is there any way to achieve this?

This is definitely an FAQ (and we should document the answer more prominently).

https://github.com/prometheus/prometheus/issues/3756 is the most
recent incarnation I noticed, but there have been more discussions on
GitHub, IRC, and the mailing lists in the past.

(Side note: Seeing so many users repeatedly bump their head into the
more-than-one-port-to-scrape-on-a-K8s-pod wall, I'm tempted to use
Brian's usual concern about increased support effort to make us create
a simpler solution for the use case than the currently endorsed one.)

One simpler option would be to encode the information in the port name, that is closer to per-port annotations than trying to do it up in the pod.

Beyond that, I suspect that all solutions on our end are likely to be more complex overall.

I've never been a major fan of the approach the example k8 scrape configs take, all the annotations the user must create are a bit much overhead for the user for my liking.

--

Brian Brazil

www.robustperception.io