Upcoming Retirement of DigiCert Log Server 2

684 views
Skip to first unread message

Devon O'Brien

unread,
May 6, 2020, 12:04:06 AM5/6/20
to Certificate Transparency Policy

DigiCert announced that on 2020-05-02, there was a possible compromise of the Log signing key used by DigiCert Log Server 2 (https://ct2.digicert-ct.com/log/). The compromise was the result of an attack using recently-disclosed vulnerabilities in the Salt configuration management software used in their Log infrastructure; further details of the incident can be found in their announcement [1].


While a known key compromise may appear to be more severe than past CT Log incidents, Chrome cannot distinguish accidental misbehavior from intentional misbehavior. Chrome’s CT policy anticipates this, and requires multiple SCTs so that misbehavior in any particular Log (whether intentional or accidental) is non-catastrophic. Given this, Chrome takes the same response to nearly every form of Log incident, and treats these incidents as equivalent to intentional misbehavior.


Accordingly, DigiCert Log Server 2 will transition to Retired with the effective date detailed below. As a result, certificates that rely on embedded SCTs from DigiCert Log Server 2 will have an increased reliance on the remaining embedded SCTs. Currently, every certificate relying on embedded SCTs from DigiCert Log Server 2 contains at least one other currently qualified SCT from another Log. Should any of those other Logs retire before these certificates expire, it could cause certificate validation failures down the road; however, there will be no breakage as a result of retiring DigiCert Log Server 2.


Certificates relying on SCTs from soon-to-be Retired Logs delivered via OCSP or TLS are expected to swap these out before the Retirement goes into effect. This is because SCTs delivered via OCSP or TLS must be from Logs that are Qualified at the time of the connection, which these SCTs will no longer be once this Log is Retired.


I also wanted to briefly share why other alternative courses of action were ruled out: 

  • Rejecting this CT Log, which results in removing it entirely from Chromium, would be effectively retroactively rewriting policy to require at least two currently qualified SCTs,  and would have caused millions of certificates to fail to validate. Chrome’s CT Policy is designed to allow Logs to become Retired, even while still valid certificates include their SCTs.

  • Adding new behavior to updated CT user agents while not causing breakage would be effectively creating a new untested Log state, and would likely be functionally equivalent to Retirement regardless. 


------

Retirement Announcement


Effective 2020-05-19, DigiCert Log Server 2 Log (https://ct2.digicert-ct.com/log/) will transition to Retired, with the last ‘Qualified’ SCT having a timestamp no later than 1588550440, or 2020-05-04T00:00:40+00:00 in ISO 8601 format, which is the point at which this Log went Read Only.


No SCT from DigiCert Log Server 2 - past, present or future - will count towards the requirement that at least one SCT is Qualified at time of check and, as such, will not be appropriate for inclusion within the TLS extension or embedded within a stapled OCSP response. SCTs that are dated prior to 2020-05-04T00:00:40 and embedded within existing certificates will continue to count as “once or currently” qualified, but certificates issued on-or-after 2020-05-04T00:00:40 will need to ensure that there are a sufficient number of SCTs from Logs which are Qualified or Pending qualification at time of issuance.


What does this mean for site operators?


If you are delivering SCTs embedded in the certificate, this should require no action on your part. All certificates previously issued, and which contain SCTs issued by the DigiCert Log Server 2, that complied with the Certificate Transparency policy will continue to do so. While SCTs from the DigiCert Log Server 2 will not be considered as being from a Log Qualified at the time of check, the presence of Qualified Google Logs in the certificate will ensure there is no disruption. Your CA may make replacement certificates available, using fresh SCTs from currently Qualified Logs, as a way of providing additional protection against future incidents. If this is the case, you can replace your certificates when convenient. 


If you are delivering SCTs embedded in the OCSP response, via OCSP stapling, then your CA must take appropriate action to update their OCSP pipeline to include at least one SCT from a non-Google operated Log that is Qualified at time of check. Once done, you must refresh the OCSP response stapled to the connection. Alternatively, you may also enable the TLS extension mechanism to deliver additional SCTs from non-Google operated Logs, in order to ensure that at least one SCT from a non-Google Log, Qualified at time of check, is presented.


If you are delivering SCTs via a TLS extension, SCTs issued by DigiCert Log Server 2 will not be considered Qualified at the time of check. As such, you must begin to serve SCTs from a different, non-Google Log by 2020-05-19, in order to satisfy the CT requirements.


What does this mean for CAs?


If you are embedding SCTs in your certificates, any SCTs from DigiCert Log Server 2 for newly issued certificates will not count towards the minimum requirements after 2020-05-19, either in the number of SCTs from Logs once or currently Qualified or as a non-Google Log. You must provide SCTs from a diverse set of Logs which are Qualified at the time of issuance; SCTs from the DigiCert Log Server 2 will not count towards this requirement.


While it is not required in response to Retiring this Log, if you are a CA with existing certificates that embed SCTs from DigiCert Log Server 2, you may wish to proactively reissue affected certificates to increase the resilience of your subscribers’ certificates to possible future Log incidents.


If you are embedding SCTs in your OCSP responses, you must issue new OCSP responses before 2020-05-19, which replace the DigiCert Log Server 2 SCTs with SCTs issued from another non-Google Log. Your customers must then begin to serve this new response. Alternatively, your customers must include SCTs delivered via the TLS extension until you are able to update the SCTs included in your responses.


What does this mean for Log Operators


If you are operating a CT Log not listed in the above soon-to-be Retired Log(s), you do not need to take any action as a result of this announcement.


Corey Bonnell

unread,
May 6, 2020, 2:55:40 PM5/6/20
to Certificate Transparency Policy
Hi Devon,
Thank you for the comprehensive notification. I do have one follow-up question regarding the Qualified SCT cut-off time of 2020-05-04 midnight UTC. According to the original announcement by Digicert, the log server was compromised around midnight 2020-05-03, so there appears to be a period of approximately 24 hours where the key was compromised but SCTs generated during this time will be Qualified. Was the cut-off time decided based on the timing of Digicert's original announcement, or some other factor, such as evidence that the attacker did not access the signing key material?

Thanks,
Corey

Devon O'Brien

unread,
May 6, 2020, 4:37:19 PM5/6/20
to Certificate Transparency Policy

Hi Corey,


This is actually a very excellent question, and I'm glad you asked.


Retiring DigiCert’s CT Log on May 02 (or months earlier when the Log first deployed the vulnerable software) instead of May 04 would do one thing: retroactively rewrite CT Policy for all certificates issued after May 02 to “at least 2 SCTs must be currently qualified at time of check”. This would be inconsistent with stated policy, provide no protection to users, and be deeply unfair to CAs and site operators who issued/obtained a certificate during this period.


Choosing the retirement date (previously called disqualification date, if you’re following along with past incidents, e.g. [1], [2]) for a given CT Log has always been about ensuring CAs can reliably include sufficient SCTs at point of certificate issuance, not about whether we have faith in whether or when the Log’s key was misused. The key misuse or policy violation is why it’s being Retired in the first place. This is the very crux of why we require >= 2 embedded SCTs that were Qualified at the point of issuance, which is the only purpose the retirement date serves in CT enforcement.


In this situation, we could have picked a retirement timestamp on May 19, May 04, or May 02 and all of those dates would have provided the same protection to Chrome’s users: That is, as of the effective date of this change, certificates will need to rely on another SCT to meet the “currently qualified at time of check” requirement to ensure they are publicly logged in a Log currently believed to be intact and behaving according to policy. 


Since dates before May 04 would have been inconsistent with standing policy and caused certificate breakage, despite these certificates being demonstrably logged by a still-Qualified CT Log, we selected the earliest known timestamp that would do neither.


-Devon

[1] https://groups.google.com/a/chromium.org/d/msg/ct-policy/qOorKuhL1vA/efOTq9cWDgAJ

[2] https://groups.google.com/a/chromium.org/d/msg/ct-policy/W1Ty2gO0JNA/ZbQxlgRZAQAJ


Case Taintor

unread,
May 8, 2020, 12:59:27 PM5/8/20
to Certificate Transparency Policy
Hello,

I'm replying this to since I believe the issues we are experiencing are related to this announcement and perhaps someone watching this can fix the problem. Since Thu, 07 May 2020 23:15:00 GMT we have been experiencing issues with CT. It's been a devil to track down the problem but the root of it seems to be that one of Google's servers is responding with one state of the CT log while another server responds with another. Anyone know who can fix this? Thanks!

Full details here: https://github.com/google/certificate-transparency/issues/1476 (the different objects are related to "DigiCert Log Server 2")

Case

Devon O'Brien

unread,
May 8, 2020, 3:19:24 PM5/8/20
to Certificate Transparency Policy
Hi Case,

Thanks for bringing this to our attention. We're discussing this internally with the team that maintains the gstatic CT Log List and we'll get back to you once we get to the bottom of it. 

-Devon

Case Taintor

unread,
May 8, 2020, 4:37:50 PM5/8/20
to Devon O'Brien, Certificate Transparency Policy
Great to hear. I suspect this same issue has existed for a while (although we haven’t been using CT for long so we haven’t personally experienced it). This ticket hints at previous encounters - 
https://github.com/babylonhealth/certificate-transparency-android/issues/28. Looks like there may be a need for a strategy to definitively link the log version with its signature.

Case

--
You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msgid/ct-policy/d2e6c0e1-1953-423c-ae6d-a500018ac84d%40chromium.org.
--

Case Taintor
Domain Lead, Developer Services
Engineering

+46 700 029 423

Klarna Bank AB (publ)
Sveavägen 46, 111 34 Stockholm
Tel: +46 8 120 120 00
Reg no: 556737-0431
klarna.com

Devon O'Brien

unread,
May 15, 2020, 5:07:51 PM5/15/20
to Certificate Transparency Policy, asymm...@chromium.org
Case,

Just to circle back on this, Kat posted in response to the issue filed against the old CT Log implementation repo [1]. If you encounter this issue again after the full fix, please create a new thread on ct-p...@chromium.org and we can take another look.

Case Taintor

unread,
May 17, 2020, 3:18:56 AM5/17/20
to Devon O'Brien, Certificate Transparency Policy
Thanks for circling back Devon. I've left a note on the github issue to better understand a timeline for a full fix. I appreciate your help and am looking forward to the fix.

Case
Reply all
Reply to author
Forward
0 new messages