Get probe_ssl_earliest_cert_expiry metric for http probes with client certificate file similar to curl

[Saikat Ray]

n this case we are trying to probe a site that requires client certificate, i am "intentionally" not sending the certificate via tls_config due to some security constraints. The only thing i am interested in this probe is the certificate expiry date
similar to what we get when we execute "curl - iv https://myservice.sss.aaa.net/"

Host operating system: output of uname -a

Linux blackbox-one-prometheus-blackbox-exporter-9cdd9d67d-d456w 5.10.102.1-microsoft-standard-WSL2 #1

blackbox_exporter version: output of blackbox_exporter --version

version 0.24.0

What is the blackbox.yml module config.

modules:
http_2xx:
prober: http
http:
enable_http2: true
fail_if_not_ssl: true
follow_redirects: true
method: HEAD
preferred_ip_protocol: ip4
valid_http_versions:
- HTTP/2.0
tls_config:
insecure_skip_verify: true
http_2xx2:
prober: http
http:
enable_http2: true
fail_if_not_ssl: false
follow_redirects: true
method: GET
preferred_ip_protocol: ip4
valid_http_versions:
- HTTP/2.0
tls_config:
insecure_skip_verify: true

What is the prometheus.yml scrape config.

just try out by running blackexporter probe function

What logging output did you get from adding &debug=true to the probe URL?

Logs for the probe:
ts=2023-06-22T13:29:31.809938012Z caller=main.go:181 module=http_2xx2 myservice.sss.aaa.net level=info msg="Beginning probe" probe=http timeout_seconds=119.5
ts=2023-06-22T13:29:31.810004165Z caller=http.go:328 module=http_2xx2 myservice.sss.aaa.net level=info msg="Resolving target address" myservice.sss.aaa.net ip_protocol=ip4
ts=2023-06-22T13:29:31.858580912Z caller=http.go:328 module=http_2xx2 myservice.sss.aaa.net level=info msg="Resolved target address" myservice.sss.aaa.net ip=XX.XXX.XXX.XXX
ts=2023-06-22T13:29:31.858787491Z caller=client.go:252 module=http_2xx2 myservice.sss.aaa.net level=info msg="Making HTTP request" url=http://XX.XXX.XX.XXX host=myservice.sss.aaa.net.net
ts=2023-06-22T13:29:32.312484166Z caller=client.go:498 module=http_2xx2 myservice.sss.aaa.net level=info msg="Received redirect" location=https://myservice.sss.aaa.net/
ts=2023-06-22T13:29:32.312617354Z caller=client.go:252 module=http_2xx2 myservice.sss.aaa.net level=info msg="Making HTTP request" url=https://myservice.sss.aaa.net/ host=
ts=2023-06-22T13:29:32.312636522Z caller=client.go:252 module=http_2xx2 myservice.sss.aaa.net level=info msg="Address does not match first address, not sending TLS ServerName" first=XX.XXX.XXX.XXX address=myservice.sss.aaa.net.net
ts=2023-06-22T13:29:32.994268596Z caller=handler.go:120 module=http_2xx2 myservice.sss.aaa.net level=error msg="Error for HTTP request" err="Get "https://myservice.sss.aaa.net/\": remote error: tls: certificate required"
ts=2023-06-22T13:29:32.994326223Z caller=handler.go:120 module=http_2xx2 myservice.sss.aaa.net level=info msg="Response timings for roundtrip" roundtrip=0 start=2023-06-22T13:29:31.858945616Z dnsDone=2023-06-22T13:29:31.858945616Z connectDone=2023-06-22T13:29:32.08670928Z gotConn=2023-06-22T13:29:32.08673468Z responseStart=2023-06-22T13:29:32.312278076Z tlsStart=0001-01-01T00:00:00Z tlsDone=0001-01-01T00:00:00Z end=0001-01-01T00:00:00Z
ts=2023-06-22T13:29:32.994357725Z caller=handler.go:120 module=http_2xx2 myservice.sss.aaa.net level=info msg="Response timings for roundtrip" roundtrip=1 start=2023-06-22T13:29:32.312771806Z dnsDone=2023-06-22T13:29:32.313804149Z connectDone=2023-06-22T13:29:32.54102004Z gotConn=2023-06-22T13:29:32.769238002Z responseStart=0001-01-01T00:00:00Z tlsStart=2023-06-22T13:29:32.541058519Z tlsDone=2023-06-22T13:29:32.769132722Z end=0001-01-01T00:00:00Z
ts=2023-06-22T13:29:32.994381499Z caller=main.go:181 module=http_2xx2 myservice.sss.aaa.net level=error msg="Probe failed" duration_seconds=1.18441594

Metrics that would have been returned:

HELP probe_dns_lookup_time_seconds Returns the time taken for probe dns lookup in secondsTYPE probe_dns_lookup_time_seconds gauge

probe_dns_lookup_time_seconds 0.04864438

HELP probe_duration_seconds Returns how long the probe took to complete in secondsTYPE probe_duration_seconds gauge

probe_duration_seconds 1.18441594

HELP probe_failed_due_to_regex Indicates if probe failed due to regexTYPE probe_failed_due_to_regex gauge

probe_failed_due_to_regex 0

HELP probe_http_content_length Length of http content responseTYPE probe_http_content_length gauge

probe_http_content_length 0

HELP probe_http_duration_seconds Duration of http request by phase, summed over all redirectsTYPE probe_http_duration_seconds gauge

probe_http_duration_seconds{phase="connect"} 0.455004955
probe_http_duration_seconds{phase="processing"} 0.225543471
probe_http_duration_seconds{phase="resolve"} 0.049676727000000004
probe_http_duration_seconds{phase="tls"} 0.228074226
probe_http_duration_seconds{phase="transfer"} 0

HELP probe_http_redirects The number of redirectsTYPE probe_http_redirects gauge

probe_http_redirects 1

HELP probe_http_ssl Indicates if SSL was used for the final redirectTYPE probe_http_ssl gauge

probe_http_ssl 0

HELP probe_http_status_code Response HTTP status codeTYPE probe_http_status_code gauge

probe_http_status_code 0

HELP probe_http_uncompressed_body_length Length of uncompressed response bodyTYPE probe_http_uncompressed_body_length gauge

probe_http_uncompressed_body_length 0

HELP probe_http_version Returns the version of HTTP of the probe responseTYPE probe_http_version gauge

probe_http_version 0

HELP probe_ip_addr_hash Specifies the hash of IP address. It's useful to detect if the IP address changes.TYPE probe_ip_addr_hash gauge

probe_ip_addr_hash 3.872600863e+09

HELP probe_ip_protocol Specifies whether probe ip protocol is IP4 or IP6TYPE probe_ip_protocol gauge

probe_ip_protocol 4

HELP probe_success Displays whether or not the probe was a successTYPE probe_success gauge

probe_success 0

What did you do that produced an error?What did you expect to see?

probe_ssl_earliest_cert_expiry

What did you see instead?

probe_ssl_earliest_cert_expiry is not present

[Brian Candler]

(Aside: your text is almost unreadable in that font/colour combination. Try viewing it at https://groups.google.com/g/prometheus-users/c/hpNrqYrHeiQ to see what I mean)

In this case we are trying to probe a site that requires client certificate, i am "intentionally" not sending the certificate via tls_config due to some security constraints. The only thing i am interested in this probe is the certificate expiry date

similar to what we get when we execute "curl - iv https://myservice.sss.aaa.net/"
image

I think you'd have to modify blackbox_exporter, or write your own exporter, for this use case.

The Go HTTP(S) client is returning an error because it fails to complete the TLS negotiation: the server requires you to authenticate with a client private key and certificate, and you didn't provide one. But you're only interested in whether it got as far as the server authenticating itself with its own certificate.  I haven't tested it, but I expect you'd need to configure your own net/http Transport and crypto.tls Config with your own VerifyPeerCertificate to stash the cert expiry time somewhere, then ignore the error from the rest of the negotiation.

Or you could write a shell script that does what you want, run it periodically from cron, and stash the results in a metrics file for node_exporter textfile collector to pick up.

Incidentally, you'd be much better off not using "insecure_skip_verify", but instead provide the certificate of the root CA which signed the server's certificate (which would be the server's certificate itself, if it's self-signed).  This does not have any security implication; the certificate contains only a public key.