"Sectigo AddTrust External CA Root" and the "probe_ssl_earliest_cert_expiry"

54 views
Skip to first unread message

Julian van den Berkmortel

unread,
May 12, 2020, 5:26:18 PM5/12/20
to Prometheus Users
I won't ask "why is this happening" because enough has been said about that, the reasoning behind it or what "probe_ssl_earliest_cert_expiry" does, does not or should do.

I only have one question which is, what will happen after the expiry of the "Sectigo AddTrust External CA Root" certificate on the 30th of May.
To be completely honest, my knowledge of certificates and everything around it is as much as it has to be to solve daily tasks and problems if they ever occur.
I'd expect the certificate to have to be renewed so that it's signed by a root certificate other than the "Sectigo AddTrust External CA Root", but is this the only solution?
I'm trying to reduce upcoming false positives by silencing some of the alerts beforehand but this doesn't feel like a practical "solution" and requires us to manually remind ourselves to re-issue the certificate in a year or something (some earlier than others).

I've tried using the SSL Exporter but this gave me problems when I tried to determine what certificate was an end-user certificate and should alert upon, but that is a whole other story.

I hope someone can clarify this a bit more!

Matt Doughty

unread,
May 12, 2020, 6:24:53 PM5/12/20
to Julian van den Berkmortel, Prometheus Users
If you have an up-to-date trust store, the cross signing certificates for the newer root CAs should mean you don't have to do anything.  If you don't have an updated trust store, you have work to do.

--Matt

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/c86ad060-97da-4fc5-a1df-dae0bff078b9%40googlegroups.com.


--
--Matt

Julian van den Berkmortel

unread,
May 12, 2020, 6:34:27 PM5/12/20
to Prometheus Users
It's in regards to the "probe_ssl_earliest_cert_expiry" metric which uses the date of the earliest expiring certificate in the chain as its value.
Its value at the moment is the 30th of May because the root certificate is the certificate which will expire the earliest in the certificate chain right now, even though the end-user certificate won't expire for the next couple of months and stay valid because of the cross signing certificates as you explained (thus causing the false-positives alerts).
I was curious whether there is another solution to prevent false-positives but keep alerting active for the domains in question which have the expiring certificate as their root certificate, other than completely renewing the certificate.

On Wednesday, May 13, 2020 at 12:24:53 AM UTC+2, Matt Doughty wrote:
If you have an up-to-date trust store, the cross signing certificates for the newer root CAs should mean you don't have to do anything.  If you don't have an updated trust store, you have work to do.

--Matt

On Tue, May 12, 2020 at 5:26 PM Julian van den Berkmortel <jul...@weprovide.com> wrote:
I won't ask "why is this happening" because enough has been said about that, the reasoning behind it or what "probe_ssl_earliest_cert_expiry" does, does not or should do.

I only have one question which is, what will happen after the expiry of the "Sectigo AddTrust External CA Root" certificate on the 30th of May.
To be completely honest, my knowledge of certificates and everything around it is as much as it has to be to solve daily tasks and problems if they ever occur.
I'd expect the certificate to have to be renewed so that it's signed by a root certificate other than the "Sectigo AddTrust External CA Root", but is this the only solution?
I'm trying to reduce upcoming false positives by silencing some of the alerts beforehand but this doesn't feel like a practical "solution" and requires us to manually remind ourselves to re-issue the certificate in a year or something (some earlier than others).

I've tried using the SSL Exporter but this gave me problems when I tried to determine what certificate was an end-user certificate and should alert upon, but that is a whole other story.

I hope someone can clarify this a bit more!

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.


--
--Matt

Harald Koch

unread,
May 12, 2020, 6:40:32 PM5/12/20
to Prometheus Users
On Tue, May 12, 2020, at 18:34, Julian van den Berkmortel wrote:
It's in regards to the "probe_ssl_earliest_cert_expiry" metric which uses the date of the earliest expiring certificate in the chain as its value.
Its value at the moment is the 30th of May because the root certificate is the certificate which will expire the earliest in the certificate chain right now, even though the end-user certificate won't expire for the next couple of months and stay valid because of the cross signing certificates as you explained (thus causing the false-positives alerts).
I was curious whether there is another solution to prevent false-positives but keep alerting active for the domains in question which have the expiring certificate as their root certificate, other than completely renewing the certificate.

- make sure the newer (cross-signing) certs are in your trust store, so that the blackbox exporter can find a valid chain to/through them.
- remove the expiring Root CA from your trust store.

Problem solved?

--
Harald

Matt Doughty

unread,
May 12, 2020, 7:23:47 PM5/12/20
to Harald Koch, Prometheus Users
Should solve problem with the added benefit that removing the old Root CA is probably a good test to do anyway.

--Matt

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/9e7d225d-a0a5-4250-859c-b0079905c14b%40www.fastmail.com.


--
--Matt

Sebastian Ebling

unread,
May 13, 2020, 2:07:57 AM5/13/20
to Matt Doughty, Harald Koch, Prometheus Users
Another option is to remove the intermediate certificate from the chain you provide at your TLS endpoint. Blackbox exporter will then check with the new path as long as you have the cross signed CA in your keystore.

Clients that do not have the new cross signed CA in their keystore will fail then. But they will do so anyhow after expiry of the old root certificate in some days. So this is also a good setup for testing your clients keystores. 

Best regards,
Sebastian

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CAGyBzczGirXVZs%2B8rKuV-yzVV%2BDYacYeXE5CRXbKAHunVbXJCA%40mail.gmail.com.

Julian van den Berkmortel

unread,
May 13, 2020, 5:31:22 PM5/13/20
to Prometheus Users
I've tried removing the expiring certificate from the trust store ("/etc/ca-certificates.conf" and "update-ca-certificates") and I thought it yielded the wanted results but...
I checked one of the domains which before had the expiry date of May 30th and this one worked, it gave back the proper date.
But then I checked another domain and that one was still giving back the expiry date of May 30th, so after that I reverted my changes in "/etc/ca-certificates.conf" and ran "update-ca-certificates" again expecting the domain that now worked to fail again.
But, it didn't. So, right now I'm kinda confused why the domain keeps working even though I added back the expiring root certificate to the trust store...


On Wednesday, May 13, 2020 at 1:23:47 AM UTC+2, Matt Doughty wrote:
Should solve problem with the added benefit that removing the old Root CA is probably a good test to do anyway.

--Matt

On Tue, May 12, 2020 at 6:40 PM Harald Koch <haral...@gmail.com> wrote:
On Tue, May 12, 2020, at 18:34, Julian van den Berkmortel wrote:
It's in regards to the "probe_ssl_earliest_cert_expiry" metric which uses the date of the earliest expiring certificate in the chain as its value.
Its value at the moment is the 30th of May because the root certificate is the certificate which will expire the earliest in the certificate chain right now, even though the end-user certificate won't expire for the next couple of months and stay valid because of the cross signing certificates as you explained (thus causing the false-positives alerts).
I was curious whether there is another solution to prevent false-positives but keep alerting active for the domains in question which have the expiring certificate as their root certificate, other than completely renewing the certificate.

- make sure the newer (cross-signing) certs are in your trust store, so that the blackbox exporter can find a valid chain to/through them.
- remove the expiring Root CA from your trust store.

Problem solved?

--
Harald

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to promethe...@googlegroups.com.


--
--Matt
Reply all
Reply to author
Forward
0 new messages