blackbox exporter's probe_ssl_earliest_cert_expiry giving negative values

[deln...@gmail.com]

I understand there's an ongoing discussion on this issue. How do you prevent false(or true) alerts when one of the applications is providing multiple certs and one of these has expired? Silencing these is not solution.

[Brian Candler]

Filter it to ignore values which are "too negative":

expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14 > -365

[Nemanja Delic]

Hi Brian,

In this case the cert has expired on Oct 13 22:12:47 2020 GMT. Since it's not last one in chain it's not breaking things. I can set the filter accordingly ( with smth like > -7),  just thought there might be some other solution.

On Tue, Oct 20, 2020 at 12:20 PM Brian Candler <b.ca...@pobox.com> wrote:

Filter it to ignore values which are "too negative":

expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 14 > -365

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/3ef5f08e-fca0-479e-96f1-b8a7e2c2514bo%40googlegroups.com.

[Brian Candler]

Any certificate in the chain which is expired or invalid will "break things": i.e. clients will not accept the certificate as valid, if they cannot establish a chain of trust back to the root.

From the linked github discussion about negative time to expire, I understood these would be hugely negative values - effectively that the certificate "expired" in the year 1AD.  You can filter out those by ignoring alerts where the time to expire is enormously negative.

Also, that condition will only occur if you set "insecure_skip_verify: true" which means "don't perform any validation of the certificate".  You can avoid this but not setting insecure_skip_verify.  If the certificate was signed by a private CA, then provide the private CA's root certificate in the tls_config section, so that blackbox_exporter can validate it.

[Nemanja Delic]

Brian,

Using pretty much standard Blackbox job config:

- job_name: blackbox
  honor_timestamps: true
  params:
    module:
    - http_2xx
  scrape_interval: 1m
  scrape_timeout: 1m
  metrics_path: /probe
  scheme: http
  file_sd_configs:
  - files:
    - /etc/prometheus/file_sd_config.d/blackbox_*.yaml
    refresh_interval: 5m
  relabel_configs:
  - source_labels: [__address__]
    separator: ;
    regex: (.*)
    target_label: __param_target
    replacement: $1
    action: replace
  - source_labels: [__param_target]
    separator: ;
    regex: (.*)
    target_label: instance
    replacement: $1
    action: replace
  - separator: ;
    regex: (.*)
    target_label: __address__
    replacement: localhost:9115
    action: replace

All monitored web services use internal certs  
Will try to provide an internal ca path in tls_config.
All endpoints are using internal certs. 

--

You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/e1cadbe1-02c1-4aec-80bb-427ec7383b55o%40googlegroups.com.

[Harald Koch]

On Tue, Oct 20, 2020, at 08:52, Nemanja Delic wrote:

In this case the cert has expired on Oct 13 22:12:47 2020 GMT. Since it's not last one in chain it's not breaking things. I can set the filter accordingly ( with smth like > -7),  just thought there might be some other solution.

PKIX path validation requires that all certificates be valid at the time of validation: RFC 5280 Section 6.1.3. As soon as any certificate in the path expires, the entire chain is invalidated. So yeah - things are probably breaking.

--

Harald

[Christian Hoffmann]

Hi,

On 10/20/20 11:25 AM, deln...@gmail.com wrote:
> I understand there's an ongoing discussion

> <https://github.com/prometheus/blackbox_exporter/issues/653> on this

> issue. How do you prevent false(or true) alerts when one of the
> applications is providing multiple certs and one of these has expired?
> Silencing these is not solution.

You mail's subject says probe_ssl_earliest_cert_expiry while the linked
issue references the rather new
probe_ssl_last_chain_expiry_timestamp_seconds.

Which one have you been experimenting with? The latter sounds like it
might fit your usecase.
Which blackbox_exporter version are you using? There was a bugfix
related to this metric in the last release:

[1] https://github.com/prometheus/blackbox_exporter/pull/681

Kind regards,
Christian