Does automatic service discovery work for TLS based targets on k8s?

[vteja...@gmail.com]

Could you help me with below questions please?

Using Prometheus k8s scrape configuration, Prometheus can identify all scrape targets that are non-TLS based.

I wonder if Prometheus to support automatic service discovery while scraping TLS based Prometheus clients.

As far as I understand, 

1. Prometheus doesn't support mutual TLS at scraping

2. Each service should add its Prometheus TLS job configuration in the Prometheus.yaml. Prometheus Operator can be used to automate this process instead of manual Prometheus configuration edit.

Am I correct with my understanding?

Thanks,

Teja

[Brian Candler]

That's incorrect on both counts:

1. Prometheus supports scraping with TLS, including TLS client certificate authentication (assuming that's what you mean by "mutual TLS")

2. Any service discovery mechanism can be used to scrape either HTTP or HTTPS targets.  You're not limited to using "static" or "file" service discovery.

Scraping over TLS is enabled by setting "scheme: https" in the scrape job, and if required adding a tls_config to set things like the client key and certificate.

If for some reason you want a mix of http and https in the same scrape job then you can instead set __scheme__ in your relabel_config.

I'm talking about standalone Prometheus here.  As for k8s and Prometheus Operator, you'll need to work out for yourself how to modify adn deploy the prometheus configuration.

[vteja...@gmail.com]

I am confused when you say " Any service discovery mechanism can be used to scrape either HTTP or HTTPS targets"

1. Can Prometheus scrape TLS and non-TLS targets at the same time using a single k8s scrape job? If yes, could you point me an example configuration?

2. When I looked at the TLS configuration: 

# CA certificate to validate API server certificate with.

[ ca_file: <filename> ] 

 # Certificate and key files for client cert authentication to the server. 

[ cert_file: <filename> ]

[ key_file: <filename> ] 

 # ServerName extension to indicate the name of the server. # https://tools.ietf.org/html/rfc4366#section-3.1

 [ server_name: <string> ]

 # Disable validation of the server certificate.

 [ insecure_skip_verify: <boolean> ]  

Lets say we have service A, B, C and D deployed on k8s. Only A and B exposes metrics over TLS. C and D expose over non-TLS.

Could you point me how mTLS configuration and k8s scrape configuration look like? 

How can Prometheus identify cert and key files if we don't explicitly mention them?

Thanks,

Teja

[Brian Candler]

On Friday, 18 September 2020 11:27:06 UTC+1, vteja...@gmail.com wrote:

I am confused when you say " Any service discovery mechanism can be used to scrape either HTTP or HTTPS targets"

1. Can Prometheus scrape TLS and non-TLS targets at the same time using a single k8s scrape job? If yes, could you point me an example configuration?

What do you mean by a "k8s scrape job"?  Do you mean a prometheus scrape job which uses a kubernetes_sd_config to identify targets to scrape?

 

Lets say we have service A, B, C and D deployed on k8s. Only A and B exposes metrics over TLS. C and D expose over non-TLS.

Could you point me how mTLS configuration and k8s scrape configuration look like? 

You need some way to distinguish the hosts which are exposing metrics over TLS, and those which are not.  For example, perhaps there is a kubernetes label which can tell you whether it's HTTP or HTTPS.  Then you can add a step in your metric relabelling to set __scheme__ accordingly.

I suggest you:

- show your current scrape config

- explain how you can distinguish between HTTP and HTTPS endpoints

- then it should be possible to amend your config to set __scheme__

However this seems odd to me, and you might want to explain *why* some endpoints are TLS and some are not.  Are the TLS and non-TLS endpoints listening on different ports?  If so, you'll need to relabel __address__ as well.