Monitoring Prometheus itself with https

[Hank Huang]

Hi all!

So I setup Prometheus to monitor itself.

Now I want to test with https, so I followed the doc and generated example.com.crt and example.com.key, and referenced them in the config file and web config file.

I also double clicked on the example.com.crt to install the cert onto my machine.

Then I launch the Prometheus with the two config files：

.\prometheus.exe --config.file=prometheus.yml --web.config.file=web.yml

When I query "up" from Prometheus, it's always 0, the response status is 200 though. Also there's a "TLS handshake error" in the console.

I think maybe it's because I didn't install the cert correctly. Any insight is appreciated.

config file (prometheus.yml):

web config file (web.yml):

syntax wise looks fine:

[Julius Volz]

Hi,

I don't know how TLS certs work on Windows, but you should at least be able to see the exact scrape error on the /targets page of your Prometheus server - what does it say?

Cheers,

Julius

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/51ceb9ba-58e0-4149-b199-e49f21661b1cn%40googlegroups.com.

--

Julius Volz

PromLabs - promlabs.com

[Brian Candler]

Please don't paste graphical screenshots: they are hard to read, and it's impossible to copy-paste them to make corrections.

First thing is, you're scraping port 9090 but you haven't told it to use HTTPS. You need setting "scheme: https" in the scrape job.

Secondly, you've set up TLS wrongly, although it may work given that you have "insecure_skip_verify: true".

- At the *server* side you need tls_server_config with cert_file and key_file, which is as you have it.

- At the *client* side (which in this case is prometheus making an outbound scrape connection to itself), you don't want cert_file or key_file; you need ca_file.  This points to the certificate file of the certificate authority which signed the example.com.crt certificate.  If this is a self-signed certificate, then this is the same certificate, i.e.  "ca_file: example.com.crt"

Thirdly, you're connecting to the host using name "localhost", but this will only verify successfully if the certificate contains "localhost" as one of its SubjectAltNames.  You should connect using whatever name you signed for the certificate.  Or, you can use the "server_name: ..." setting in tls_config to say what name to expect in the certificate presented by the server.  Again, "insecure_skip_verify" will probably skip this check.

(But of course, really you don't want to use "insecure_skip_verify". Why are you deploying TLS at all, if you're doing it in an insecure way?)

Fourthly, you didn't show how you generated the certificates.  With modern versions of Go (and hence recent versions of Prometheus), the certificate CommonName is ignored.  The server *must* have a certificate with at least one SubjectAltName.  So if you followed an out-of-date how-to for signing certificates, you probably made a bad certificate.

This is what I use:

mkdir /etc/prometheus/ssl
cd /etc/prometheus/ssl
openssl genpkey -genparam -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out p-256.param

openssl req -x509 -newkey ec:p-256.param -keyout prometheus_key.pem -out prometheus_cert.pem -days 29220 -nodes -subj /commonName=prometheus/ -addext "subjectAltName=DNS:prometheus"

In "/commonName=prometheus/" and "DNS:prometheus", replace "prometheus" with the hostname you want in the certificate. "localhost" would work, but apart from self-scraping, normally your clients are connecting to the prometheus server using some real fully-qualified domain name not "localhost", so you should use that FQDN.

[Brian Candler]

(I should have said: that's what I use *when making test deployments with self-signed certificates*.  For production use, I use real certificates from LetsEncrypt)

[Hank Huang]

Thanks for the advice! The /targets page is very helpful in my debugging!

[Hank Huang]

Thank you for the info, they are very helpful! I get it working basicaly following your advice one-by-one.

I am testing to query from a Prometheus DB, so I am setting up a local test environment. I will switch to querying a remote Promethues DB in the future.

For the record, This is what I have changed:

1. use https scheme in the config file

2. use example.com.crt as ca_file

3. removed insecure_skip_verify

4. regenerated the cert with SubjectAltName

openssl req -x509 -newkey rsa:4096 -nodes -keyout example.com.key -out example.com.crt -subj /commonName=example.com/ -addext "subjectAltName=DNS:example.com, DNS:localhost"

[Brian Candler]

Glad you got it working.  For tidiness, you can avoid the DNS:localhost in the certificate by changing your scrape job, either to

  - job_name: prometheus

    scheme: https
    static_configs:
      - targets: ['example.com:9090']

    tls_config:
      ...

(assuming that example.com resolves to an IP address that prometheus can use to connect to itself); or:

  - job_name: prometheus
    scheme: https
    static_configs:
      - targets: ['localhost:9090']

    tls_config:
      server_name: example.com
      ...

The latter case still connects to localhost (127.0.0.1), but verifies the certificate subject is "example.com"