ssl cert monitoring with blackbox exporter

[nina guo]

Hi,

I'm using following module to monitor ssl cert, but failed following issue.

tcp_cert:
        prober: tcp
        timeout: 5s
        tcp:
          tls: true
          preferred_ip_protocol: "ip4"
          tls_config:
            insecure_skip_verify: true

curl -g 'http://1.1.1.1:9115/probe?module=tcp_cert&target=dns:587&debug=true'
Logs for the probe:
ts=2022-08-09T01:23:27.020158973Z caller=main.go:320 module=tcp_cert target=dns:587 level=info msg="Beginning probe" probe=tcp timeout_seconds=5
ts=2022-08-09T01:23:27.020211226Z caller=tcp.go:40 module=tcp_cert target=dns:587 level=info msg="Resolving target address" ip_protocol=ip4
ts=2022-08-09T01:23:27.024779773Z caller=tcp.go:40 module=tcp_cert target=dns:587 level=info msg="Resolved target address" ip=2.2.2.2
ts=2022-08-09T01:23:27.024809314Z caller=tcp.go:121 module=tcp_cert target=dns:587 level=info msg="Dialing TCP with TLS"
ts=2022-08-09T01:23:27.02673954Z caller=main.go:130 module=tcp_cert target=dns:587 level=error msg="Error dialing TCP" err="tls: first record does not look like a TLS handshake"
ts=2022-08-09T01:23:27.026778527Z caller=main.go:320 module=tcp_cert target=dns:587 level=error msg="Probe failed" duration_seconds=0.00659328

[Brian Candler]

Sigh.  We've been through all this with you before in great detail.

https://groups.google.com/g/prometheus-users/c/LZbihDncIig/m/cqAA-UtdAQAJ

Port 587 is SMTP submission, and it does not perform TLS on connection. Try "telnet dns 587" and you'll see it responds in plain text, which is just what the error message told you: "tls: first record does not look like a TLS handshake"

To get it to do TLS, you need to send the "starttls" command before starting the TLS negotiation, like this example and in the thread linked above.

[nina guo]

Thank you Brian. I know.

I already use smtp_starttls to check the connection with mail server. I found smtp_starttls also exposed ssl cert related metrics.

But now I want to take ssl cert check out from the job which is for checking SMTP. Because I want to set different scrape_interval and evaluation inverval for the cert check.

Any good suggestion?

[Brian Candler]

Do you mean, you want one probe that tests TCP connection and the "SMTP" banner only; and another job that tests further including STARTTLS and the certificate?

Then just make two blackbox tests, one for each of those cases.

You want different scrape_interval for them?  Then just make two prometheus scrape jobs, and put one test under the first job, and the other test under the second job.

You want to use different evaluation_interval? Then make two different alerting rule groups, with different evaluation intervals, and put one alerting rule under each.

[nina guo]

smtp_banner:
        prober: tcp
        timeout: 20s
        tcp:
          preferred_ip_protocol: ip4
          query_response:
          - expect: "^220 ([^ ]+) ESMTP( .+)?$"
            send: "EHLO prober"
          - expect: "^250 "
            send: "QUIT\r"   

 
      smtp_starttls:
        prober: tcp
        timeout: 20s
        tcp:
          tls_config:
            insecure_skip_verify: true
          query_response:
            - expect: "^220 ([^ ]+) ESMTP( .+)?$"
              send: "EHLO prober\r"
            - expect: "^250-STARTTLS"
            - expect: "^250 .*$"
              send: "STARTTLS\r"
            - expect: "^220"
              starttls: true
            - send: "EHLO prober\r"
            - expect: "^250 .*$"
              send: "QUIT\r"

Can I design as above then 2 jobs

- job_name: Mail Server
        metrics_path: /probe
        params:
          module: [smtp_banner]
        file_sd_configs:
        - files:
          - '/etc/prometheus/mail' 
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: prometheus-blackbox-exporter:9115

- job_name: Mail Server TLS
        metrics_path: /probe
        params:
          module: [smtp_starttls]
        file_sd_configs:
        - files:
          - '/etc/prometheus/mail' 
        relabel_configs:
          - source_labels: [__address__]
            target_label: __param_target
          - source_labels: [__param_target]
            target_label: instance
          - target_label: __address__
            replacement: prometheus-blackbox-exporter:9115

[Brian Candler]

Yes.