Splitting kubernetes service ip with regex

[bahhoo]

Hi there,

I had opened an github issue about this, but it was brought to my attention that it would be a better idea to ask such a question here.


I have pods with jmx-exporter. jmx is exposed in one particular port and there are other ports exposed for different purposes. I would like prometheus to scrape only the jmx-port and not the other ones.

I do use the annotation "prometheus.io/port" in my pods, but to be sure that only that port is scraped I want to do this on prometheus side, as well  in case some pods/services don't have that annotation.

To do so I am trying to replace the ports in the addresses that I get from the service endpoints.

My current config looks like this:

- job_name: 'kubernetes-test'
      kubernetes_sd_configs:
      - role: endpoints

      relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement:  

With the current config I get endpoints in prometheus like

http://10.131.0.2:1111 http://10.131.0.2:2222 http://10.131.0.2:3333 http://10.131.0.2:4444

I would like to split up the pod IP and append the port number I would like to scrape. e.g. anly 10.131.0.2:2222

I don't seem to be able to make use of the regexes...

I assume what I need is something like this:

    - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: functioning regex 
        replacement:  ${1}:2222

where ${1} is the service IP and 2222 is the port I want.

Can anybody help me with that?

Thanks.

[Matthias Rampke]

Hey,

you can see how it's done in the example configuration here:

https://github.com/prometheus/prometheus/blob/4b868113bbcaeea0df2325dd591a8576d6e19fad/documentation/examples/prometheus-kubernetes.yml#L159-L163

The idea is that you can regex from multiple source labels by specifying more than one in the source_labels array. They will be concatenated with the separator (defaults to a semicolon). So with

source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]

you get something like 10.11.12.13:2222;3333 and the example regex rearranges that to 10.11.12.13:3333.

/MR

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To post to this group, send email to promethe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/19851a51-e137-49e3-8bab-8aa33cf04ff8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[bahhoo]

Hey Matthias,

when I don't have the port annotation, with the example confih  I get the targets

 

http://10.129.3.56:2222/metrics unknown instance="10.129.3.56:2222" kubernetes_name="" kubernetes_namespace="" Never

http://10.129.3.56:8080/metrics unknown instance="10.129.3.56:8080" kubernetes_name="" kubernetes_namespace="" Never

http://10.129.3.56:8778/metrics unknown instance="10.129.3.56:8778" kubernetes_name="" kubernetes_namespace="" Never

http://10.129.3.56:8888/metrics 

What I want to have is that I get only the address from the above list 10.129.3.56 and add the port "2222" manually.