Prometheus HTTPs setup questions

[H T]

Hi,

I am trying to setup HTTPS endpoint as below and have few questions -

1.) An app/client gave their https endpoint as https://test.example.com:4443/metrics   and I am trying to connect to it via the Prometheus server hosted on a different VM. 

Should I generate a new CA.crt and a TLS cert and key file and call their endpoint from the Prometheus server?  While the Prometheus document says to use client certs and the client ca certs( test.example.com in my case) I am not quite sure how this helps as I am the one connecting from the Prometheus server.

So, should I ask the client to provide all 3 certs or should I generate new certs and use them? I tried with both client and new certs and it works- but not sure what's the best practise or correct approach here

- job_name: ‘metrics_orderer_4443’

scrape_interval: 10s

scheme: https

tls_config:

ca_file: "ca.crt"

cert_file: "server.crt"

key_file: "server.key"

static_configs:

- targets: ['test.example.com:4443']

labels:

instance: '** orderer **'

2.) Can we enable https endpoints via service discovery? If yes, can i use consul for the same ?

--

[Brian Candler]

First, get your access working with curl.

(1) Does the remote server have a certificate which is signed by (a) a public CA, (b) a private CA, or (c) self-signed? In cases (b) and (c) you'll find that curl rejects the connection.

If so, find out the CA which signed the cert, and you should be able to pass this as the "--cacert" option to curl.  Once that works, it becomes the "ca_file" option to prometheus.

(2) Does the server require you as the client to provide a certificate to authenticate?  If so, the server operator will tell you what you need to provide.  If it's a private CA, they'll most likely provide you with the private key and the certificate

For curl pass these as the --key and --cert options.  When this is working, these become key_file and cert_file options to prometheus.

If as you said it works with random certs that you've generated, this most likely means that the server isn't doing any certificate authentication at all.  In which case, you can drop the key_file and cert_file options altogether from prometheus.

For your second question: yes, any form of service discovery will work.  You will either set the scheme "https" at the job level as you showed, or set the label __scheme__ in your rewriting rules.

[H T]

Thanks, Brian for your reply.

In my case, remote server is using a self-signed certificate and based on your comments I tried the below approach which worked and other which didn't.

Working Scenario  -> (Both remote server and Prometheus using same CA certs but separate private and crt files)

1.) Self-signed CA certs were generated first(crt and key files)

2.) Remote server (Node.js app) used above CA to generated its crt and key files.

3.) Prometheus server running on a separate box used above CA to generated its crt and key files.

Not working Scenario  -> (Remote server and Prometheus using their own CA certs) -  I think this is expected behavior.

Should we always use the same CA to generate Remote server and Prometheus certs for HTTPS communications? Also, should both remote servers and Prometheus client use the same key and cert files? In my case, different files generated(key and crt) for the remote server and Prometheus from the same CA worked.

Lastly, our remote server is running on a VM and we have other apps running inside a docker container and pods. As of now, I am we are using DNS names like (test.server.com) and it works. But I would like to know if we can generate certs for IP address(127.0.0.1), localhost, and use them instead of DNS name. Our goal is that the remote server will expose /metrics api at  default IP like 127.0.0.1 and generate certs which Prometheus uses to connect- but we are not sure how can we generate certs for default IP or loccalhost

Thanks,

Hemanth

[Brian Candler]

On Thursday, 7 May 2020 01:27:14 UTC+1, H T wrote:

In my case, remote server is using a self-signed certificate and based on your comments I tried the below approach which worked and other which didn't.

Working Scenario  -> (Both remote server and Prometheus using same CA certs but separate private and crt files)

1.) Self-signed CA certs were generated first(crt and key files)

2.) Remote server (Node.js app) used above CA to generated its crt and key files.

3.) Prometheus server running on a separate box used above CA to generated its crt and key files.

For clarity, this isn't what I'd describe as the server using a "self-signed cert".  All CA root certificates are self-signed, by definition.

This is what I'd describe as a private CA, which is issuing certs for both the remote server and the prometheus server.

 

Not working Scenario  -> (Remote server and Prometheus using their own CA certs) -  I think this is expected behavior.

Do you mean, two different private CAs?

This should work fine.  At each end, the "ca_cert" setting that you give is the *other* CA, i.e. the one you need to use to verify the peer's certificate.

e.g. at the prometheus end:

tls_config:

key_file: "server.key" # my key

cert_file: "server.crt" # my certificate, signed by CA 1

ca_file: "ca.crt" # CA 2's root certificate, used to verify exporter's cert

 

At the exporter side it's the other way round.

Should we always use the same CA to generate Remote server and Prometheus certs for HTTPS communications? Also, should both remote servers and Prometheus client use the same key and cert files?

Don't use the same key and cert for both ends.  This would mean that the node's certificate could be used to pretend to be prometheus.

Actually, that's a good reason for using two different CAs.  At the moment, prometheus verifies the identity of the exporter's certificate matches the hostname in the HTTPS scrape URL (you can override the identity using the "server_name:" setting); but node_exporter doesn't verify the identity of prometheus' certificate.  So any node could use its own cert to scrape any other node.

You can workaround this by signing prometheus' certificate by a different CA, and then node_exporter only trusts *that* CA.  However if you are writing your own custom exporter, you can add a check to verify the client certificate's identity is the one which belongs to prometheus.

 

Lastly, our remote server is running on a VM and we have other apps running inside a docker container and pods. As of now, I am we are using DNS names like (test.server.com) and it works. But I would like to know if we can generate certs for IP address(127.0.0.1), localhost, and use them instead of DNS name. Our goal is that the remote server will expose /metrics api at  default IP like 127.0.0.1 and generate certs which Prometheus uses to connect- but we are not sure how can we generate certs for default IP or loccalhost

Well, you technically you *can* issue certs for an IP address - there is a SAN IP address extension.  It's unconventional and I wouldn't recommend it, and I wouldn't be surprised if not all software supports it.  Better to use /etc/hosts if you need to force the IP address.

Also, using SSL for communication to 127.0.0.1 or localhost is pretty pointless.

[H T]

Thanks, Brian for the info. I was able to set up mutual TLS using the information you shared.

 1.) If a client has https enabled without client auth can I extract the metrics using Prometheus? I tried below settings but didn't work. Not sure if I am missing few settings.

- job_name: ‘test_4443’

scrape_interval: 10s

scheme: https

tls_config:

cert_file: "localhost.crt"

key_file: "localhost.key"

static_configs:

- targets: ['127.0.0.1:4443']

labels:

instance: '** test **'

2.) For node_exporter apart from using a proxy for HTTPS is there an alternate way to setup HTTPS?

3.) I am trying to understand 'server_name' setting. If possible can you please share any link or info that can help in understanding more about its usage?

Thanks

[Brian Candler]

On Saturday, 9 May 2020 02:40:03 UTC+1, H T wrote:

Thanks, Brian for the info. I was able to set up mutual TLS using the information you shared.

 1.) If a client has https enabled without client auth can I extract the metrics using Prometheus?

Yes.  Prometheus will validate the certificate of the exporter (what you call the "client", but from the point of view of scrape it's a HTTPS server).  The exporter will accept scrapes from anyone, and happily export the data.

 

I tried below settings but didn't work. Not sure if I am missing few settings.

- job_name: ‘test_4443’

scrape_interval: 10s

scheme: https

tls_config:

cert_file: "localhost.crt"

key_file: "localhost.key"

static_configs:

- targets: ['127.0.0.1:4443']

labels:

instance: '** test **'

Those settings don't match the scenario you described.

If you want to connect to the exporter without using client cert authentication, prometheus is still going to validate the exporter's cert.  So you need:

tls_config:

ca_file: "ca.crt"

where ca.crt is the CA which signed the exporter's cert.  In addition, the cert will have to have SAN 127.0.0.1 (because that's the target you're scraping) or prometheus will drop the connection, unless:

1. You apply "server_name:" setting to the config, in which case the cert must match whatever is in "server_name" (and that's a static value, so if there are multiple targets in the scrape they have to have the same certificate identity); or

2. You apply "insecure_skip_verify:" which turns off all validation, both the name in the certificate and the authority it's signed by (so any imposter could provide any junk self-signed cert and it would be accepted)

I still don't know why you want to use TLS to talk to a server on 127.0.0.1.  The connection cannot be intercepted except by someone who has "root" on the box, in which case you're already toast.  It's just a waste of CPU time.

 

2.) For node_exporter apart from using a proxy for HTTPS is there an alternate way to setup HTTPS?

node_exporter 1.0.0rc1 has built-in support for HTTPS.  It works fine.

 

3.) I am trying to understand 'server_name' setting. If possible can you please share any link or info that can help in understanding more about its usage?

Imagine you're a web browser (client) connecting to a web server with HTTPS, say to www.google.com

When you open the connection: the first thing that happens is that a TLS exchange takes place and the other side presents their certificate, with proof that they own the corresponding private key.

At your side, the client verifies three things before going any further:

1. The certificate is signed by a trusted CA (directly, or via a chain of CA certs)

2. The current time is within the valid from and valid to dates in the certificate

3. The name in the certificate matches the name you were connecting to (in this case "www.google.com")

Prometheus is the same: if you specify

scheme: https

...
targets:
- foo.example.com
- bar.example.com

then upon connecting to foo.example.com it will check that the name in the exporter's certificate is "foo.example.com"; and for bar.example.com it will check the certificate identity is "bar.example.com".  If this check fails, it means it's talking to an imposter, and it drops the connection on the floor without sending any further data.

"server_name" changes the name check.  With

scheme: https

tls_config:

  server_name: exporter.local
...

targets:
- foo.example.com
- bar.example.com

then when connecting to both foo.example.com and bar.example.com, it will check that the exporter's certificate has identity "exporter.local"

This can be useful, for example when you're connecting to a target by IP address, but the certificate has a name.  But only really useful when you have a single target in the scrape job, or you're using the same key and cert across all targets (which is easy to deploy)

[H T]

Thanks Brian for all your help. I am now able to connect to remote extractor using server_name and through TLS on that particular server.