SECURITY RELEVANT FOR CAs: The curious case of the Dangerous Delegated Responder Cert

12459 views
Skip to first unread message

Ryan Sleevi

unread,
Jul 1, 2020, 5:06:22 PM7/1/20
to mozilla-dev-security-policy
I've created a new batch of certificates that violate 4.9.9 of the BRs,
which was introduced with the first version of the Baseline Requirements as
a MUST. This is https://misissued.com/batch/138/

A quick inspection among the affected CAs include O fields of: QuoVadis,
GlobalSign, Digicert, HARICA, Certinomis, AS Sertifitseeimiskeskus,
Actalis, Atos, AC Camerfirma, SECOM, T-Systems, WISeKey, SCEE, and CNNIC.

Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST
include an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP
Delegated Responder within Section 4.2.2.2 as indicated by the presence of
the id-kp-OCSPSigning as an EKU.

These certificates lack the necessary extension, and as such, violate the
BRs. As the vast majority of these were issued on-or-after 2013-02-01, the
Effective Date of Mozilla Root CA Policy v2.1, these are misissued. You
could also consider the effective date as 2013-05-15, described later in
[1] , without changing the results.

This batch is NOT comprehensive. According to crt.sh, there are
approximately 293 certificates that meet the criteria of "issued by a
Mozilla-trusted, TLS-capable CA, with the OCSPSigning EKU, and without
pkix-nocheck". misissued.com had some issues with parsing some of these
certificates, due to other non-conformities, so I only included a sample.

Censys.io is aware of approximately 276 certificates that meet this
criteria, as you can see at [2]. The differences in perspectives
underscores the importance of CAs needing to carefully examine the
certificates they've issued to understand.

It's important for CAs to understand this is Security Relevant. While they
should proceed with revoking these CAs within seven (7) days, as defined
under the Baseline Requirements Section 4.9.1.2, the degree of this issue
likely also justifies requiring witnessed Key Destruction Reports, in order
to preserve the integrity of the issuer of these certificates (which may
include the CA's root).

The reason for this is simple: In every case I examined, these are
certificates that appear to nominally be intended as Issuing CAs, not as
OCSP Responder Certificates. It would appear that many CAs were unfamiliar
with RFC 6960 when constructing their certificate profiles, and similarly
ignored discussion of this issue in the past [3], which highlighted the
security impact of this. I've flagged this as a SECURITY matter for CAs to
carefully review, because in the cases where a third-party, other than the
Issuing CA, operates such a certificate, the Issuing CA has delegated the
ability to mint arbitrary OCSP responses to this third-party!

For example, consider a certificate like https://crt.sh/?id=2657658699 .
This certificate, from HARICA, meets Mozilla's definition of "Technically
Constrained" for TLS, in that it lacks the id-kp-serverAuth EKU. However,
because it includes the OCSP Signing EKU, this certificate can be used to
sign arbitrary OCSP messages for HARICA's Root!

This also applies to non-technically-constrained sub-CAs. For example,
consider this certificate https://crt.sh/?id=21606064 . It was issued by
DigiCert to Microsoft, granting Microsoft the ability to provide OCSP
responses for any certificate issued by Digicert's Baltimore CyberTrust
Root. We know from DigiCert's disclosures that this is independently
operated by Microsoft.

Unfortunately, revocation of this certificate is simply not enough to
protect Mozilla TLS users. This is because this Sub-CA COULD provide OCSP
for itself that would successfully validate, AND provide OCSP for other
revoked sub-CAs, even if it was revoked. That is, if this Sub-CA's key was
maliciously used to sign a GOOD response for itself, it would be accepted.
These security concerns are discussed in Section 4.2.2.2.1 of RFC 6960, and
is tied to a reliance on the CRL. Mozilla users COULD be protected through
the use of OneCRL, although this would not protect other PKI participants
or use cases that don't use OneCRL.

A little more than a third of the affected certificates have already been
revoked, which is encouraging. However, I'm not aware of any incident
reports discussing this failure, nor am I aware of any key destruction
reports to provide assurance that these keys cannot be used maliciously.
While this seems like a benign failure of "we used the wrong profile", it
has a meaningful security impact to end users, even if it was made with the
best of intentions.

This has been a requirement of the BRs since the first version, and is
spelled out within the OCSP RFCs, and CAs are expected to be deeply
knowledgeable in both of these areas. There is no excusing such an
oversight, especially if it was (most likely) to work around an issue with
a particular CA Software Vendor's product. Recall that the same
justification (work around an issue in a vendor's product) has been used to
justify MITM interception. Ignorance and malice are, unfortunately, often
indistinguishable, and thus have to be treated the same.

While I'll be looking to create Compliance Incidents for the affected CAs,
and attempting to report through their problem reporting mechanisms, the
fact that it is the constrained CAs which are not yet required to be
disclosed, and most likely invisible to CT (e.g. S/MIME issuing CAs that do
not issue TLS), still pose substantial risk, that it requires every CA
closely examine their practices.

CAs affected MUST ensure they revoke such certificates within 7 days, as
per 4.9.1.2 (5) and 4.9.1.2 (6)

[1]
https://wiki.mozilla.org/CA:CertificatePolicyV2.1#Time_Frames_for_included_CAs_to_comply_with_the_new_policy
[2]
https://censys.io/certificates?q=%28parsed.extensions.extended_key_usage.ocsp_signing%3Atrue+and+validation.nss.valid%3Atrue+and+parsed.validity.start%3A%5B2013-05-15+TO+*%5D%29+and+not+parsed.unknown_extensions.id%3A1.3.6.1.5.5.7.48.1.5
[3]
https://groups.google.com/d/msg/mozilla.dev.security.policy/XQd3rNF4yOo/bXYjt1mZAwAJ

Ryan Sleevi

unread,
Jul 1, 2020, 10:10:26 PM7/1/20
to Ryan Sleevi, mozilla-dev-security-policy
On Wed, Jul 1, 2020 at 5:05 PM Ryan Sleevi <ry...@sleevi.com> wrote:

> While I'll be looking to create Compliance Incidents for the affected CAs,
>

This is now done, I believe. However, as mentioned, just because a
compliance bug was not filed does not mean that a CA may not be affected;
it may just be that CT does not know of the cert and the CA did not
disclose it via CCADB. I only filed incidents for CAs where the issuer is
not already revoked via OneCRL.

https://bugzilla.mozilla.org/show_bug.cgi?id=1649961 - Actalis
https://bugzilla.mozilla.org/show_bug.cgi?id=1649963 - ATOS
https://bugzilla.mozilla.org/show_bug.cgi?id=1649944 - Camerfirma
https://bugzilla.mozilla.org/show_bug.cgi?id=1649951 - DigiCert
https://bugzilla.mozilla.org/show_bug.cgi?id=1649943 - Firmaprofesional
https://bugzilla.mozilla.org/show_bug.cgi?id=1649937 - GlobalSign
https://bugzilla.mozilla.org/show_bug.cgi?id=1649945 - HARICA
https://bugzilla.mozilla.org/show_bug.cgi?id=1649947 - Microsec
https://bugzilla.mozilla.org/show_bug.cgi?id=1649938 - QuoVadis
https://bugzilla.mozilla.org/show_bug.cgi?id=1649964 - PKIoverheid
https://bugzilla.mozilla.org/show_bug.cgi?id=1649962 - SECOM
https://bugzilla.mozilla.org/show_bug.cgi?id=1649942 - SK ID
https://bugzilla.mozilla.org/show_bug.cgi?id=1649941 - T-Systems
https://bugzilla.mozilla.org/show_bug.cgi?id=1649939 - WISeKey

Peter Gutmann

unread,
Jul 1, 2020, 11:48:29 PM7/1/20
to mozilla-dev-security-policy, ry...@sleevi.com
Ryan Sleevi via dev-security-policy <dev-secur...@lists.mozilla.org> writes:

>Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST include
>an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP Delegated
>Responder within Section 4.2.2.2 as indicated by the presence of the id-kp-
>OCSPSigning as an EKU.

Unless I've misread your message, the problem isn't the presence or not of a
nocheck extension but the invalid presence of an OCSP EKU:

>I've flagged this as a SECURITY matter [...] the Issuing CA has delegated the


>ability to mint arbitrary OCSP responses to this third-party

So the problem would be the presence of the OCSP EKU when it shouldn't be
there, not the absence of the nocheck extension.

Peter.

Ryan Sleevi

unread,
Jul 2, 2020, 12:31:16 AM7/2/20
to Peter Gutmann, mozilla-dev-security-policy, ry...@sleevi.com
On Wed, Jul 1, 2020 at 11:48 PM Peter Gutmann <pgu...@cs.auckland.ac.nz>
wrote:
Not quite. It’s both.

The BR violation is caused by the lack of the extension.

The security issue is caused by the presence of the EKU.

However, since some CAs only view things through the lens of BR/program
violations, despite the sizable security risk they pose, the compliance
incident is what is tracked. The fact that it’s security relevant is
provided so that CAs understand that revocation is necessary, and that it’s
also not sufficient, because of how dangerous the issue is.

Pedro Fuentes

unread,
Jul 2, 2020, 2:19:34 AM7/2/20
to mozilla-dev-s...@lists.mozilla.org
Hello,
Our understanding when creating this SubCA was that the CA certificate should include the EKUs that would be allowed to issue, and therefore, as it would generate certificates for the OCSP responders, it should include such EKU, the same it would include the EKU for clientAuthentication, for example.
Can you please clarify why this is not correct and what is the security problem it creates?
Thanks,
Pedro

Pedro Fuentes

unread,
Jul 2, 2020, 3:11:10 AM7/2/20
to mozilla-dev-s...@lists.mozilla.org
Sorry, my message was incomplete... please read the las part as:

Can you please clarify why this is not correct and what is the security problem it creates if the CA is not operated externally?

Paul van Brouwershaven

unread,
Jul 2, 2020, 3:23:19 AM7/2/20
to ry...@sleevi.com, mozilla-dev-s...@lists.mozilla.org, Pedro Fuentes
Thanks for raising this issue Ryan, I'm trying to update
http://revocationcheck.com/ to cover this issue.

>From my understanding:

The OCSPnocheck extension is only required for a delegated OCSP responder
certificate as it can't provide answers for itself.
For a CA certificate in (CA signed responses) the OCSPnocheck extension
MUST NOT be present as it's not authorized to create a status for itself.

A CA certificate MUST NOT include the OCSPsigning EKU, even when
using CA signed responses.
When using CA signed responses the EKU digitalSignature MUST be set.
Delegated OCSP signing certificates MUST only have the OCSPsigning EKU set
(Microsoft A12)
Delegated OCSP signing certificates MUST be issued directly by the CA that
is identified in the request as the issuer of the EE certificate (RFC6960
4.2.2.2)

But as Pedro also mentioned, the EKU extension in intermediate certificates
acts as a constraint on the permitted EKU OIDs in end-entity certificates,
which means you won't be able to use delegated OCSP signing certificates
with strict EKU validation on the path? While not every client might have
strict validation on this, it would be really confusing if it's required
for one EKU and forbidden for the other.

On Thu, 2 Jul 2020 at 08:19, Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hello,
> Our understanding when creating this SubCA was that the CA certificate
> should include the EKUs that would be allowed to issue, and therefore, as
> it would generate certificates for the OCSP responders, it should include
> such EKU, the same it would include the EKU for clientAuthentication, for
> example.
> Can you please clarify why this is not correct and what is the security
> problem it creates?
> Thanks,
> Pedro
>
> El jueves, 2 de julio de 2020, 6:31:16 (UTC+2), Ryan Sleevi escribió:
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>


--
Regards,

Paul van Brouwershaven

http://linkedin.com/in/pvanbrouwershaven
http://facebook.com/p.vanbrouwershaven
http://twitter.com/vanbroup

Rob Stradling

unread,
Jul 2, 2020, 6:14:37 AM7/2/20
to mozilla-dev-security-policy, ry...@sleevi.com
> This batch is NOT comprehensive. According to crt.sh, there are approximately 293 certificates that meet the criteria of "issued by a Mozilla-trusted, TLS-capable CA, with the OCSPSigning EKU, and without pkix-nocheck". misissued.com had some issues with parsing some of these certificates, due to other non-conformities, so I only included a sample.

I just reproduced this result. I've posted my SQL query and (thanks to GitHub) a searchable TSV report of all 293 certificates here:
https://gist.github.com/robstradling/6c737c97a7a3ab843b6f24747fc9ad1f

________________________________
From: dev-security-policy <dev-security-...@lists.mozilla.org> on behalf of Ryan Sleevi via dev-security-policy <dev-secur...@lists.mozilla.org>
Sent: 01 July 2020 22:05
To: mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Subject: SECURITY RELEVANT FOR CAs: The curious case of the Dangerous Delegated Responder Cert

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


I've created a new batch of certificates that violate 4.9.9 of the BRs,
which was introduced with the first version of the Baseline Requirements as
a MUST. This is https://misissued.com/batch/138/

A quick inspection among the affected CAs include O fields of: QuoVadis,
GlobalSign, Digicert, HARICA, Certinomis, AS Sertifitseeimiskeskus,
Actalis, Atos, AC Camerfirma, SECOM, T-Systems, WISeKey, SCEE, and CNNIC.

Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST
include an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP
Delegated Responder within Section 4.2.2.2 as indicated by the presence of
While I'll be looking to create Compliance Incidents for the affected CAs,
and attempting to report through their problem reporting mechanisms, the
fact that it is the constrained CAs which are not yet required to be
disclosed, and most likely invisible to CT (e.g. S/MIME issuing CAs that do
not issue TLS), still pose substantial risk, that it requires every CA
closely examine their practices.

CAs affected MUST ensure they revoke such certificates within 7 days, as
per 4.9.1.2 (5) and 4.9.1.2 (6)

[1]
https://wiki.mozilla.org/CA:CertificatePolicyV2.1#Time_Frames_for_included_CAs_to_comply_with_the_new_policy
[2]
https://censys.io/certificates?q=%28parsed.extensions.extended_key_usage.ocsp_signing%3Atrue+and+validation.nss.valid%3Atrue+and+parsed.validity.start%3A%5B2013-05-15+TO+*%5D%29+and+not+parsed.unknown_extensions.id%3A1.3.6.1.5.5.7.48.1.5
[3]
https://groups.google.com/d/msg/mozilla.dev.security.policy/XQd3rNF4yOo/bXYjt1mZAwAJ

Peter Mate Erdosi

unread,
Jul 2, 2020, 7:04:29 AM7/2/20
to mozilla-dev-security-policy
Just for my better understanding, is the following CA certificate
"TLS-capable"?

X509v3 Basic Constraints critical:
CA:TRUE
X509v3 Key Usage critical:
Certificate Sign, CRL Sign
X509v3 Extended Key Usage:
Time Stamping, OCSP Signing


Peter

Rob Stradling

unread,
Jul 2, 2020, 7:21:51 AM7/2/20
to mozilla-dev-security-policy, Peter Mate Erdosi
Hi Peter. The "following CA certificate" (which I'll call Certificate X) is not capable of issuing id-kp-serverAuth leaf certificates that will be trusted by Mozilla, but that fact is entirely irrelevant to this discussion. Notice that Ryan wrote "issued by a Mozilla-trusted, TLS-capable CA" rather than "is a Mozilla-trusted, TLS-capable CA".

Certificate X contains the id-kp-OCSPSigning EKU. This means that it can be used as a delegated OCSP signer, to sign OCSP responses on behalf of its issuer. If its issuer is "a Mozilla-trusted, TLS-capable CA", then all of its issuer's delegated OCSP signer certificates are in scope for the BRs and for the Mozilla Root Store Policy.

Certificate X is an intermediate CA certificate, which is capable of issuing id-kp-timeStamping leaf certificates. That's all very nice, but it doesn't alter the fact that Certificate X is also a (misissued) delegated OCSP signing certificate that is in scope for the BRs and the Mozilla Root Store Policy.

________________________________
From: dev-security-policy <dev-security-...@lists.mozilla.org> on behalf of Peter Mate Erdosi via dev-security-policy <dev-secur...@lists.mozilla.org>
Sent: 02 July 2020 12:04
To: mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: SECURITY RELEVANT FOR CAs: The curious case of the Dangerous Delegated Responder Cert

Pedro Fuentes

unread,
Jul 2, 2020, 7:52:34 AM7/2/20
to mozilla-dev-s...@lists.mozilla.org
El jueves, 2 de julio de 2020, 9:23:19 (UTC+2), Paul van Brouwershaven escribió:
> But as Pedro also mentioned, the EKU extension in intermediate certificates
> acts as a constraint on the permitted EKU OIDs in end-entity certificates,
> which means you won't be able to use delegated OCSP signing certificates
> with strict EKU validation on the path? While not every client might have
> strict validation on this, it would be really confusing if it's required
> for one EKU and forbidden for the other.
>

If we look at the BR, it says:
"[^**]: Generally Extended Key Usage will only appear within end entity certificates (as highlighted in RFC 5280 (4.2.1.12)), however, Subordinate CAs MAY include the extension to further protect relying parties until the use of the extension is consistent between Application Software Suppliers whose software is used by a substantial portion of Relying Parties worldwide."

Therefore, in my humble opinion it's fully logical to understand this requirement "as it's written", which is to restrict the CA and protect relying parties... In other words, the BR is clearly saying that the meaning of the EKU in SubCAs MUST be understood as a constraint and NOT to express the EKU of the certificate itself. The same applies to other EKUs, for example the meaning of serverAuth EKU is EVIDENTLY associated to a constraint, and no one understands that the CA certificate can be used to protect a web server, as in that case the rest of the certificate profile should be also consistent with the requirements of the leaf TLS certs... I think it's not logical to consider in the BR the implications of setting some EKU and not the others.

I would consider this as two derived issues that need to be considered separately and appropriately:

#1. There's an evident "gap" in the BR in section 7.1.2.2-g that is creating a potential inconsistency with the RFC, and also creates incompatibilities with certain software solutions.

#2. This inconsistency could provoke in certain conditions a security risk, and in particular this applies in case of externally-operated subCAs. This security risk must analysed and mitigated by, maybe, revoking these SubCAs.

But I would say that just considering this as a unique problem that needs a unique solution is not appropriate.

Best,
Pedro

Peter Mate Erdosi

unread,
Jul 2, 2020, 7:59:42 AM7/2/20
to mozilla-dev-security-policy
Hi Rob, thanks for the clarification.

What will be the situation if the issuer is a Root CA instead of the "TLS
capable (intermediate or subordinate) CA"?
As far as I understood till now, it is not misissued, if the root CA cannot
be considered as an "Mozilla-trusted, TLS-capable CA".

And considering chapter 7.1.2.1 b) of CAB Forum BRG, extendedKeyUsage MUST
NOT be present in root CA certificates, but "If the Root CA Private Key is
used for signing OCSP responses, then the digitalSignature bit MUST be
set", which is the same in the 7.1.2.2 e) : ". If the Subordinate CA
Private Key is used for signing OCSP responses, then the digitalSignature
bit MUST be set."

I have not seen that the SQL query considered with digitalSignature bit,
but as I interpreted until now, the CA cannot sign OCSP responses without
setting the digitalSignature bit even the OCSPSigning EKU is used. And
Mozilla requires the BRG-conformant CAs also, isn't it?

So, I am a bit confused.


Thanks again,

Peter

On Thu, Jul 2, 2020 at 1:21 PM Rob Stradling <r...@sectigo.com> wrote:

> Hi Peter. The "following CA certificate" (which I'll call Certificate X)
> is not capable of issuing id-kp-serverAuth leaf certificates that will be
> trusted by Mozilla, but that fact is entirely irrelevant to this
> discussion. Notice that Ryan wrote "*issued by* a Mozilla-trusted,
> TLS-capable CA" rather than "*is* a Mozilla-trusted, TLS-capable CA".
>
> Certificate X contains the id-kp-OCSPSigning EKU. This means that it can
> be used as a delegated OCSP signer, to sign OCSP responses on behalf of its
> issuer. If its issuer is "a Mozilla-trusted, TLS-capable CA", then all of
> its issuer's delegated OCSP signer certificates are in scope for the BRs
> and for the Mozilla Root Store Policy.
>
> Certificate X is an intermediate CA certificate, which is capable of
> issuing id-kp-timeStamping leaf certificates. That's all very nice, but it
> doesn't alter the fact that Certificate X is also a (misissued) delegated
> OCSP signing certificate that is in scope for the BRs and the Mozilla Root
> Store Policy.
>
> ------------------------------
> *From:* dev-security-policy <dev-security-...@lists.mozilla.org>
> on behalf of Peter Mate Erdosi via dev-security-policy <
> dev-secur...@lists.mozilla.org>
> *Sent:* 02 July 2020 12:04
> *To:* mozilla-dev-security-policy <
> mozilla-dev-s...@lists.mozilla.org>
> *Subject:* Re: SECURITY RELEVANT FOR CAs: The curious case of the

Neil Dunbar

unread,
Jul 2, 2020, 8:11:22 AM7/2/20
to dev-secur...@lists.mozilla.org

On 02/07/2020 12:52, Pedro Fuentes via dev-security-policy wrote:
> If we look at the BR, it says:
> "[^**]: Generally Extended Key Usage will only appear within end entity certificates (as highlighted in RFC 5280 (4.2.1.12)), however, Subordinate CAs MAY include the extension to further protect relying parties until the use of the extension is consistent between Application Software Suppliers whose software is used by a substantial portion of Relying Parties worldwide."
>
> Therefore, in my humble opinion it's fully logical to understand this requirement "as it's written", which is to restrict the CA and protect relying parties... In other words, the BR is clearly saying that the meaning of the EKU in SubCAs MUST be understood as a constraint and NOT to express the EKU of the certificate itself.

Pedro,

I think that the problem here isn't what the BRs indicate the reading of
EKUs in a CA certificate should be.

It's that RFC 6960 (Section 4.2.2.2) states that 

> OCSP signing delegation SHALL be designated by the inclusion of
> id-kp-OCSPSigning in an extended key usage certificate extension
> included in the OCSP response signer's certificate.


In other words, if a certificate X (CA or otherwise) contains that EKU
value, by definition, it becomes a valid delegated OCSP responder
certificate, regardless of the intentions surrounding EKU interpretation
in CA certificates. Thus, OCSP responses signed by that X, on behalf of
X's issuing CA, _would_ be properly validated by compliant RP software.
If a hostile party grabs hold of the private key for the CA certificate,
their harm is not limited to the PKI described by the original CA
certificate, but extends to all of the sibling certificates of X

Now, it's true that the BRs also require the id-pkix-ocsp-nocheck
extension too, but RFC 6960 does not require it (it's just the way to
say "trust this delegated cert for as long as it is valid", and don't
consult OCSP/CRLs).

Regards,

Neil

Paul van Brouwershaven

unread,
Jul 2, 2020, 10:34:58 AM7/2/20
to dev-secur...@lists.mozilla.org
I did do some testing on EKU chaining in Go, but from my understand this
works the same for Microsoft:


An OCSP responder certificate with Extended Key Usage OCSPSigning, but an
issuing CA without the EKU (result: certificate specifies an incompatible
key usage)

https://play.golang.org/p/XSsKfxytx3O


The same chain but now the ICA includes the Extended Key Usage OCSPSigning
(result: ok)

https://play.golang.org/p/XL7364nSCe8


Microsoft requires the EKU to be present in issuing CA certificates:



*Issuing CA certificates that chain to a participating Root CA must be
constrained to a single EKU (e.g., separate Server Authentication, S/MIME,
Code Signing, and Time Stamping uses. This means that a single Issuing CA
must not combine server authentication with S/MIME, code signing or time
stamping EKU. A separate intermediate must be used for each use case.
https://docs.microsoft.com/en-us/security/trusted-root/program-requirements#a-root-requirements
<https://docs.microsoft.com/en-us/security/trusted-root/program-requirements#a-root-requirements>
(8)*


Technically constraining issuing CA’s based on the EKU as Microsoft
requires feels like a good thing to do. But if we leave out the OCSPSigning
EKU we must leave out all EKU constraints (and talk to Microsoft) or move
away from delegated OCSP signing certificates and all move to CA signed
responses.

Ryan Sleevi

unread,
Jul 2, 2020, 10:41:11 AM7/2/20
to Paul van Brouwershaven, MDSP
On Thu, Jul 2, 2020 at 10:34 AM Paul van Brouwershaven via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:

> I did do some testing on EKU chaining in Go, but from my understand this
> works the same for Microsoft:
>

Go has a bug https://twitter.com/FiloSottile/status/1278501854306095104

The understanding for Microsoft isn't correct, as linked earlier in the
reference materials.


> Microsoft requires the EKU to be present in issuing CA certificates:


> *Issuing CA certificates that chain to a participating Root CA must be
> constrained to a single EKU (e.g., separate Server Authentication, S/MIME,
> Code Signing, and Time Stamping uses. This means that a single Issuing CA
> must not combine server authentication with S/MIME, code signing or time
> stamping EKU. A separate intermediate must be used for each use case.
>
> https://docs.microsoft.com/en-us/security/trusted-root/program-requirements#a-root-requirements
> <
> https://docs.microsoft.com/en-us/security/trusted-root/program-requirements#a-root-requirements
> >
> (8)*
>

Did you paste the wrong section? This doesn't seem to be consistent with
what you're saying, and perhaps it was just a bad copy/paste? Even if
quoting Microsoft policy, how do you square this with: "A CA must
technically constrain an OCSP responder such that the only EKU allowed is
OCSP Signing." (from that same section)

Did you read the related thread where this was previously discussed on
m.d.s.p.?

Technically constraining issuing CA’s based on the EKU as Microsoft
> requires feels like a good thing to do. But if we leave out the OCSPSigning
> EKU we must leave out all EKU constraints (and talk to Microsoft) or move
> away from delegated OCSP signing certificates and all move to CA signed
> responses.


That's not correct, and is similar to the mistake I originally/previously
made, and was thankfully corrected on, which also highlighted the
security-relevant nature of it. I encourage you to give another pass at
Robin's excellent write-up, at
https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/XQd3rNF4yOo/bXYjt1mZAwAJ

Paul van Brouwershaven

unread,
Jul 2, 2020, 1:16:05 PM7/2/20
to ry...@sleevi.com, MDSP
On Thu, 2 Jul 2020 at 16:41, Ryan Sleevi <ry...@sleevi.com> wrote:

>
> On Thu, Jul 2, 2020 at 10:34 AM Paul van Brouwershaven via
> dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
>> I did do some testing on EKU chaining in Go, but from my understand this
>> works the same for Microsoft:
>>
>
> Go has a bug https://twitter.com/FiloSottile/status/1278501854306095104
>
> The understanding for Microsoft isn't correct, as linked earlier in the
> reference materials.
>

I wasn't aware that this would be for ADCS only.
The Windows certificate viewer doesn't validate the purpose but after a
quick test with the powershell command Test-Certificate, it does look to
validate the EKU path on Windows 10:

Get-ChildItem -Path
Cert:\currentUser\addressbook\63D6AEAD044E9D720930D7F814B7C74DBB541572 |
Test-Certificate -User -AllowUntrustedRoot -EKU "1.3.6.1.5.5.7.3.9"
WARNING: Chain status:
CERT_TRUST_IS_NOT_VALID_FOR_USAGE
Test-Certificate : The certificate is not valid for the requested usage.
0x800b0110 (-2146762480 CERT_E_WRONG_USAGE)
At line:1 char:94
+ ... DBB541572 | Test-Certificate -User -AllowUntrustedRoot -EKU "1.3.6.1.
...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:Certificate) [Test-Certificate],
Exception
+ FullyQualifiedErrorId :
CryptographicError,Microsoft.CertificateServices.Commands.TestCertificate


(the certificates in the test above are from the chain generated by my
previous example)


>
>> Microsoft requires the EKU to be present in issuing CA certificates:
>
>
>> *Issuing CA certificates that chain to a participating Root CA must be
>> constrained to a single EKU (e.g., separate Server Authentication, S/MIME,
>> Code Signing, and Time Stamping uses. This means that a single Issuing CA
>> must not combine server authentication with S/MIME, code signing or time
>> stamping EKU. A separate intermediate must be used for each use case.
>>
>> https://docs.microsoft.com/en-us/security/trusted-root/program-requirements#a-root-requirements
>> <
>> https://docs.microsoft.com/en-us/security/trusted-root/program-requirements#a-root-requirements
>> >
>> (8)*
>>
>
> Did you paste the wrong section? This doesn't seem to be consistent with
> what you're saying, and perhaps it was just a bad copy/paste? Even if
> quoting Microsoft policy, how do you square this with: "A CA must
> technically constrain an OCSP responder such that the only EKU allowed is
> OCSP Signing." (from that same section)
>

No, it does state that the EKU for other purposes must be set in the ussing
CA, my point here is that when you set these it does exclude OCSP signing.

The other item of the policy you refer to is confusing as it doesn't seem
to make a difference between CA signed and CA delegated responses, it might
even prohibit CA signed responses?


> Did you read the related thread where this was previously discussed on
> m.d.s.p.?
>
> Technically constraining issuing CA’s based on the EKU as Microsoft
>> requires feels like a good thing to do. But if we leave out the
>> OCSPSigning
>> EKU we must leave out all EKU constraints (and talk to Microsoft) or move
>> away from delegated OCSP signing certificates and all move to CA signed
>> responses.
>
>
> That's not correct, and is similar to the mistake I originally/previously
> made, and was thankfully corrected on, which also highlighted the
> security-relevant nature of it. I encourage you to give another pass at
> Robin's excellent write-up, at
> https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/XQd3rNF4yOo/bXYjt1mZAwAJ
>

Thanks, it's an interesting thread, but as shown above, Windows does
validate the EKU chain, but doesn't look to validate it for delegated OCSP
signing certificates?

Ryan Sleevi

unread,
Jul 2, 2020, 1:26:14 PM7/2/20
to Paul van Brouwershaven, Ryan Sleevi, MDSP
On Thu, Jul 2, 2020 at 1:15 PM Paul van Brouwershaven <
pa...@vanbrouwershaven.com> wrote:

> That's not correct, and is similar to the mistake I originally/previously
>> made, and was thankfully corrected on, which also highlighted the
>> security-relevant nature of it. I encourage you to give another pass at
>> Robin's excellent write-up, at
>> https://groups.google.com/forum/#!msg/mozilla.dev.security.policy/XQd3rNF4yOo/bXYjt1mZAwAJ
>>
>
> Thanks, it's an interesting thread, but as shown above, Windows does
> validate the EKU chain, but doesn't look to validate it for delegated OCSP
> signing certificates?
>

The problem is providing the EKU as you're doing, which forces chain
validation of the EKU, as opposed to validating the OCSP response, which
does not.

A more appropriate test is to install the test root R as a locally trusted
CA, issue an intermediate I (without the EKU/only id-kp-serverAuth), issue
an OCSP responder O (with the EKU), and issue a leaf cert L. You can then
validate the OCSP response from the responder cert (that is, an OCSP
response signed by the chain O-I-R) for the certificate L-I-R.

Pedro Fuentes

unread,
Jul 2, 2020, 2:34:22 PM7/2/20
to mozilla-dev-s...@lists.mozilla.org
Hello.
Sorry if this question is incorrect, but I’d like to know if it would acceptable that, for CAs that are owned and operated by the same entity that the Root, the CA certificate is reissued with the same key pair without the offending EKU, instead of doing a full issuance with new keys.
I consider this particular case as less risky than externally operated CAs, so I wonder if this could make possible an smoother solution.
Your comments and guidance are appreciated.
Thanks,
Pedro

Paul van Brouwershaven

unread,
Jul 2, 2020, 2:42:50 PM7/2/20
to ry...@sleevi.com, MDSP
When validating the EKU using `Test-Certificate` Windows states it's
invalid, but when using `certutil` it's accepted or not explicitly checked.
https://gist.github.com/vanbroup/64760f1dba5894aa001b7222847f7eef

When/if I have time I will try to do some further tests with a custom setup
to see if the EKU is validated at all.

Ryan Sleevi

unread,
Jul 2, 2020, 4:11:52 PM7/2/20
to Pedro Fuentes, mozilla-dev-security-policy
This is definitely a hard question, but I don't see how we can easily
resolve that. That's why the comments about Key Destruction were made.

So, first, let me say it definitely mitigates *some* of the security
concerns, particularly the most major one: a third-party being able to
arbitrarily "unrevoke" a certificate, particularly "their" certificate. In
the cases of 3P Sub-CAs, this is just so fundamentally terrifying. Even if
the Sub-CA "hasn't" abused such a capability, the mere knowledge that they
could gives them greater flexibility to "live dangerously" - or to make
them a ripe target for compromise.

Now assuming the keys are all part of the same (audited) CA infrastructure,
what does the risk perspective there look like? In effect, for every
Issuing CA that has issued one of these, Browsers/Relying Parties have no
assurance that any revocations are "correct". This is important, because
when a CA revokes a Sub-CA, even their own, we often stop worrying about
audits, for example, or stop looking for misissued certificates, because
"of course" these certificates can't be used for that purpose. The mere
existence of these certificates undermines that whole design: we have to
treat every revoked Sub-CA as if it was unrevoked, /especially/ if that
Sub-CA was the one that had the EKU.

Now, it might be tempting to say "Well, can't we audit the key usage to
make sure it never signs a delegated OCSP response"? But that shifts the
burden and the risk now onto the client software, for what was actually a
CA mistake. A new form of audit would have to be designed to account for
that, and browsers would have to think *very* carefully about what controls
were suitable, whether the auditor was qualified to examine those controls
and had the necessary knowledge. In short, it requires Browsers/the client
to work through every possible thing that could go wrong with this key
existing, and then think about how to defend against it. While the CA might
see this as "saving" a costly revocation, it doesn't really "save" anything
- it just shifts all the cost onto browsers.

It might be tempting to ask how many had the digitalSignature KU, and can
we check the KU on OCSP responses to make sure it matches? In theory,
clients wouldn't accept it, so they wouldn't be unrevocable and able to
cause shenanigans, and we're saved! But again, this is a cost transfer:
every client and relying party now needs to be updated to enforce this, and
work out the compatibility issues, and test and engineer. And even then, it
might be months or years for users to be protected, when the BRs are
supposed to provide protection "within 7 days". Even "clever" alternatives,
like "Don't allow a delegated responder to provide a response for itself"
don't fully address the issue, because it can still provide responses for
others, and that would further require mutating the revocation checking
process described in RFC 5280 to "skip" OCSP (or fall back) to CRLs. All of
this is more complexity, more testing, and contributes to the body of "dark
knowledge" needed for a secure implementation, which makes it harder to
write new browsers / new libraries to verify certificates.

This is the risk analysis we expect CAs to work through, and think about.
What is the cost of this decision on others? Often, CAs focus
(understandably) on the cost to those they've issued certificates to, but
ignore the externalized ecosystem that they're simply shifting those costs
to. The BRs try to force the CA to account for this up front, because they
*know* that if *anything* goes wrong, they have 7 days to revoke, but then
they don't design their PKIs to be resilient for that.

You can imagine a CA that was rotating issuing intermediates every year
would be in a better position, for example, if this was a "previous"
mistake, since fixed. The impact/blast radius of revoking an intermediate
is a linear decay tied to how many unexpired certificates from that
intermediate there are, which is precisely why you should rotate often.
It's a point I've made often, especially with respect to certificate
lifetimes, but it still doesn't seem to have been taken to heart yet by
many. I'm encouraged GlobalSign's new infrastructure appears to be doing
so, and although it was also affected by this issue, it's hopefully
"easier" for them to clean up versus others.

But this is what disaster recovery plans are for. The "compromise" of a
delegated signing CA is, as noted in RFC 6960, in many ways *as bad as*
compromise of the CA key. CAs have to get better at planning for things
blowing up, and I hope every incident report looks at exactly those best
practices: minimizing the pain of revoking a cert within 7d.

For CAs with non-TLS certs, this is going to be especially painful to
revoke, but it's still necessary. This underscores the "Don't use the TLS
root to issue non-TLS certs".

Pedro Fuentes

unread,
Jul 2, 2020, 5:30:29 PM7/2/20
to mozilla-dev-s...@lists.mozilla.org
Hello Ryan,
Thanks for your detailed response.

Just to be sure that we are in the same page. My question was about reissuing a new CA using the same key pair, but this implies also the revocation of the previous version of the certificate.

You elaborate the need to revoke, but this would be still done anyway.

Thanks,
Pedro

Ryan Sleevi

unread,
Jul 2, 2020, 5:33:05 PM7/2/20
to Pedro Fuentes, mozilla-dev-security-policy
On Thu, Jul 2, 2020 at 5:30 PM Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Hello Ryan,
> Thanks for your detailed response.
>
> Just to be sure that we are in the same page. My question was about
> reissuing a new CA using the same key pair, but this implies also the
> revocation of the previous version of the certificate.
>

Right, but this doesn't do anything, because the previous key pair can be
used to sign an OCSP response that unrevokes itself.

This is the problem and why "key destruction" is the best of the
alternatives (that I discussed) for ensuring that this doesn't happen,
because it doesn't shift the cost to other participants.

Pedro Fuentes

unread,
Jul 2, 2020, 6:05:16 PM7/2/20
to mozilla-dev-s...@lists.mozilla.org
I understand your rational, but my point is that this is happening in the same infrastructure where the whole PKI is operated, and under the responsibility of the same operator of the Root. In my understanding the operator of the Root has full rights to do delegated OCSP responses if those responses are produced by its own OCSP responders.

I'm failing to see what is the main problem you don't consider solved. As per your own dissertations in the related posts, there are two issues:

1. The certificate contains incorrect extensions, so it's a misissuance. This is solved by revoking the certificate, and this is done not only internally in the PKI, but also in OneCRL.

2. The operator of the SubCA could produce improper revocation responses, so this is a security risk. This risk is already difficult to find if the operator of the subCA is the same operator of the Root... If such entity wants to do a wrongdoing, there are far easier ways to do it, than overcomplicated things like unrevoking its own subCA...

Sorry, but I don't see the likeliness of the risks you evoke... I see the potential risk in externally operated CAs, but not here.

Ryan Sleevi

unread,
Jul 2, 2020, 6:25:19 PM7/2/20
to Pedro Fuentes, mozilla-dev-security-policy
On Thu, Jul 2, 2020 at 6:05 PM Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I understand your rational, but my point is that this is happening in the
> same infrastructure where the whole PKI is operated, and under the
> responsibility of the same operator of the Root. In my understanding the
> operator of the Root has full rights to do delegated OCSP responses if
> those responses are produced by its own OCSP responders.
>
> I'm failing to see what is the main problem you don't consider solved. As
> per your own dissertations in the related posts, there are two issues:
>
> 1. The certificate contains incorrect extensions, so it's a misissuance.
> This is solved by revoking the certificate, and this is done not only
> internally in the PKI, but also in OneCRL.
>

This solves *nothing* for anyone not using OneCRL. It doesn't meet the
obligations the CA warranted within its CP/CPS. It doesn't meet the BRs. It
simply "shifts" risk onto everyone else in the ecosystem, and that's a
grossly negligent and irresponsible thing to do.

"Revoking the certificate" is the minimum bar, which is already a promise
the CA made, to everyone who decides to trust that CA, that they will do
within 7 days. But it doesn't mitigate the risk.


> 2. The operator of the SubCA could produce improper revocation responses,
> so this is a security risk. This risk is already difficult to find if the
> operator of the subCA is the same operator of the Root... If such entity
> wants to do a wrongdoing, there are far easier ways to do it, than
> overcomplicated things like unrevoking its own subCA...
>
> Sorry, but I don't see the likeliness of the risks you evoke... I see the
> potential risk in externally operated CAs, but not here.


The risk is just the same! As a CA, I can understand you would say "Surely,
we would never do anything nefarious", but as a member of the community,
why should we trust what you say? Why would the risk be any different with
externally operated CAs? After all, they're audited to, like roots,
shouldn't the risk be the same? Of course you'd realize that no, they're
not the same, because the CA has no way of truly knowing the sub-CA is
being nefarious. The same is true for the Browser trusting the root: it has
no way of knowing you're not being nefarious.

Look, we've had Root CAs that have actively lied in this Forum,
misrepresenting things to the community they later admitted they knew were
false, and had previously been an otherwise CA in good standing (or at
least, no worse standing than other CAs). A CA is a CA, and the risk is
treated the same.

The line of argument being pursued here is a bit like saying "If no one
abuses this, what's the harm?" I've already shown how any attempt to
actually verify it's not abused ends up just shifting whatever cost onto
Relying Parties, when it's the CA and the Subscribers that should bear the
cost, because it's the CA that screwed up. I simply can't see how "just
trust us" is better than objective verification, especially when "just
trust us" got us into this mess in the first place. How would you provide
assurances to the community that this won't be abused? And how is the cost
for the community, in risk, better?

Tim Hollebeek

unread,
Jul 2, 2020, 6:37:19 PM7/2/20
to ry...@sleevi.com, Peter Gutmann, Mozilla
So, from our perspective, the security implications are the most important here.
We understand them, and even in the absence of any compliance obligations they
would constitute an unacceptable risk to trustworthiness of our OCSP responses,
so we have already begun the process of replacing the ICAs we are responsible for.
There are already several key ceremonies scheduled and they will continue through
the holiday weekend. We're prioritizing the ICAs that are under the control of third
parties and/or outside our primary data centers, as they pose the most risk. We are
actively working to mitigate internal ICAs as well. Expect to see revocations start
happening within the next day or two.

I understand the attraction of using a BR compliance issue to attract attention to
this issue, but honestly, that shouldn't be necessary. The BRs don't really adequately
address the risks of the OCSPSigning EKU, and there's certainly lots of room for
improvement there. I think, especially in the short term, it is more important to
focus on how to mitigate the security risks and remove the inappropriate EKU from
the affected ICAs. We can fix the BRs later.

It's also important to note that, much like SHA-1, this issue doesn't respect the
normal assumptions about certificate hierarchies. Non-TLS ICAs can have a significant
impact on their TLS-enabled siblings. This means that CA review needs to extend
beyond the certificates that would traditionally be in scope for the BRs.

I would also caution CAs to carefully analyze the implications before blindly adding the
pkix-ocsp-nocheck extension to their ICAs. That might fix the compliance issue,
but in the grand scheme of things probably makes the problem worse, as ICAs
have fairly long lifetimes, and doing so effectively makes the inadvertent delegated
responder certificate unrevokable. So while the compliance problems might be
fixed, it makes resolving the security issues much more challenging.

-Tim

> -----Original Message-----
> From: dev-security-policy <dev-security-...@lists.mozilla.org>
> On Behalf Of Ryan Sleevi via dev-security-policy
> Sent: Thursday, July 2, 2020 12:31 AM
> To: Peter Gutmann <pgu...@cs.auckland.ac.nz>
> Cc: ry...@sleevi.com; Mozilla <mozilla-dev-security-
> pol...@lists.mozilla.org>
> Subject: Re: SECURITY RELEVANT FOR CAs: The curious case of the
> Dangerous Delegated Responder Cert
>
> On Wed, Jul 1, 2020 at 11:48 PM Peter Gutmann
> <pgu...@cs.auckland.ac.nz>
> wrote:
>
> > Ryan Sleevi via dev-security-policy
> > <dev-secur...@lists.mozilla.org>
> > writes:
> >
> > >Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST
> > include
> > >an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP Delegated
> > >Responder within Section 4.2.2.2 as indicated by the presence of the
> > id-kp-
> > >OCSPSigning as an EKU.
> >
> > Unless I've misread your message, the problem isn't the presence or
> > not of a nocheck extension but the invalid presence of an OCSP EKU:
> >
> > >I've flagged this as a SECURITY matter [...] the Issuing CA has
> > >delegated
> > the
> > >ability to mint arbitrary OCSP responses to this third-party
> >
> > So the problem would be the presence of the OCSP EKU when it shouldn't
> > be there, not the absence of the nocheck extension.
>
>
> Not quite. It’s both.
>
> The BR violation is caused by the lack of the extension.
>
> The security issue is caused by the presence of the EKU.
>
> However, since some CAs only view things through the lens of BR/program
> violations, despite the sizable security risk they pose, the compliance incident
> is what is tracked. The fact that it’s security relevant is provided so that CAs
> understand that revocation is necessary, and that it’s also not sufficient,
> because of how dangerous the issue is.

Pedro Fuentes

unread,
Jul 2, 2020, 6:42:21 PM7/2/20
to mozilla-dev-s...@lists.mozilla.org
Hello Ryan,
I’m fully understanding your argumentative line, but I’d still have a question for you:

Does the operator of a root and it’s hierarchy have the right to delegate OCSP responses to its own responders?

If your answer is “No”, then I don’t have anything else to say, but if your answer is “Yes”, then I’ll be having still a hard time to see the security risk derived of this issue.

Thanks.

Ben Wilson

unread,
Jul 2, 2020, 7:13:33 PM7/2/20
to Ryan Sleevi, mozilla-dev-security-policy
All,


Thank you to Ryan for identifying this problem, and to all of you who are
earnestly investigating what this problem means and the impact to your CA
hierarchies. Mozilla::pkix requires that an OCSP responder certificate be
an end entity certificate, so we believe that Firefox and Thunderbird are
not impacted by this problem. Historically, as per
https://bugzilla.mozilla.org/show_bug.cgi?id=991209#c10, Mozilla has
allowed CA certificates to have the OCSP signing EKU because some CAs
reported that some Microsoft server software required CA certificates to
have the id-kp-OCSPSigning EKU.

The comments in the code[1] say

// When validating anything other than an delegated OCSP signing cert,

// reject any cert that also claims to be an OCSP responder, because such

// a cert does not make sense. For example, if an SSL certificate were to

// assert id-kp-OCSPSigning then it could sign OCSP responses for itself,

// if not for this check.

// That said, we accept CA certificates with id-kp-OCSPSigning because

// some CAs in Mozilla's CA program have issued such intermediate

// certificates, and because some CAs have reported some Microsoft server

// software wrongly requires CA certificates to have id-kp-OCSPSigning.

// Allowing this exception does not cause any security issues because we

// require delegated OCSP response signing certificates to be end-entity

// certificates.

Additionally, as you all know, Firefox uses OneCRL for checking the
revocation status of intermediate certificates, so as long as the revoked
intermediate certificate is in OneCRL, the third-party would not be able to
“unrevoke” their certificate (for Firefox). Therefore, Mozilla does not
need the certificates that incorrectly have the id-kp-OCSPSigning EKU to be
revoked within the next 7 days, as per section 4.9.1.2 of the BRs.

However, as Ryan has pointed out in this thread, others may still have risk
because they may not have a OneCRL equivalent, or they may have certificate
verification implementations that behave differently than mozilla::pkix in
regards to processing OCSP responder certificates. Therefore, it is
important to identify a path forward to resolve the security risk that this
problem causes to the ecosystem.

We are concerned that revoking these impacted intermediate certificates
within 7 days could cause more damage to the ecosystem than is warranted
for this particular problem. Therefore, Mozilla does not plan to hold CAs
to the BR requirement to revoke these certificates within 7 days. However,
an additional Incident Report for delayed revocation will still be
required, as per our documented process[2]. We want to work with CAs to
identify a path forward, which includes determining a reasonable timeline
and approach to replacing the certificates that incorrectly have the
id-kp-OCSPSigning EKU (and performing key destruction for them).

Therefore, we are looking forward to your continued input in this
discussion about the proper response for CAs to take to resolve the
security risks caused by this problem, and ensure that this problem is not
repeated in future certificates. We also look forward to your suggestions
on how we can improve OCSP responder requirements in Mozilla’s Root Store
Policy, and to your continued involvement in the CA/Browser Forum to
improve the BRs.

Thanks,


Ben

[1]
https://dxr.mozilla.org/mozilla-central/rev/c68fe15a81fc2dc9fc5765f3be2573519c09b6c1/security/nss/lib/mozpkix/lib/pkixcheck.cpp#858-869

[2] https://wiki.mozilla.org/CA/Responding_To_An_Incident#Revocation


On Wed, Jul 1, 2020 at 3:06 PM Ryan Sleevi via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I've created a new batch of certificates that violate 4.9.9 of the BRs,
> which was introduced with the first version of the Baseline Requirements as
> a MUST. This is https://misissued.com/batch/138/
>
> A quick inspection among the affected CAs include O fields of: QuoVadis,
> GlobalSign, Digicert, HARICA, Certinomis, AS Sertifitseeimiskeskus,
> Actalis, Atos, AC Camerfirma, SECOM, T-Systems, WISeKey, SCEE, and CNNIC.
>
> Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST
> include an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP
> Delegated Responder within Section 4.2.2.2 as indicated by the presence of

Ryan Sleevi

unread,
Jul 2, 2020, 8:10:10 PM7/2/20
to Pedro Fuentes, mozilla-dev-security-policy
On Thu, Jul 2, 2020 at 6:42 PM Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Does the operator of a root and it’s hierarchy have the right to delegate
> OCSP responses to its own responders?
>
> If your answer is “No”, then I don’t have anything else to say, but if
> your answer is “Yes”, then I’ll be having still a hard time to see the
> security risk derived of this issue.
>

Yes. But that doesn't mean we blindly trust the CA in doing so. And that's
the "security risk".

I totally appreciate that your argument is "but we wouldn't misuse the
key". The "risk" that I'm talking about is how can anyone, but the CA, know
that's true? All of the compliance obligations assume certain facts when
the CA is operating a responder. This issue violates those assumptions, and
so it violates the controls, and so we don't have any way to be confident
that the key is not misused.

I think the confusion may be from the overloading of the word "risk". Here,
I'm talking about "the possibility of something bad happening". We don't
have any proof any 3P Sub-CAs have mis-signed OCSP responses: but we seem
to agree that there's risk of that happening. It seems we disagree on
whether there is risk of the CA themselves doing it. I can understand the
view that says "Of course the CA wouldn't", and my response is that the
risk is still the same: there's no way to know, and it's still a
possibility.

I can understand that our views may differ: you may see 3P as "great risk"
and 1p as "acceptable risk". However, from the view of a browser or a
relying party, "1p" and "3p" are the same: they're both CAs. So the risk is
the same, and the risk is unacceptable for both cases.

Ryan Sleevi

unread,
Jul 2, 2020, 8:22:43 PM7/2/20
to Ben Wilson, Ryan Sleevi, mozilla-dev-security-policy
On Thu, Jul 2, 2020 at 7:13 PM Ben Wilson <bwi...@mozilla.com> wrote:

> We are concerned that revoking these impacted intermediate certificates
> within 7 days could cause more damage to the ecosystem than is warranted
> for this particular problem. Therefore, Mozilla does not plan to hold CAs
> to the BR requirement to revoke these certificates within 7 days. However,
> an additional Incident Report for delayed revocation will still be
> required, as per our documented process[2]. We want to work with CAs to
> identify a path forward, which includes determining a reasonable timeline
> and approach to replacing the certificates that incorrectly have the
> id-kp-OCSPSigning EKU (and performing key destruction for them).
>

I'm not sure I understand this. The measurement is "damage to the
ecosystem", but the justification is "Firefox is protected, even though
many others are not" (e.g. OpenSSL-derived systems, AFAICT), because
AFAICT, Firefox does a non-standard (but quite reasonable) thing.

I can totally appreciate the answer "The risk to Mozilla is low", but this
response seems... different? It also seems to place CAs that adhere to
4.9.1.2, because they designed their systems robustly, at greater
disadvantage from those that did not, and seems like it only encourages the
problem to get worse over time, not better. Regardless, I do hope that any
delay for revocation is not treated as a "mitigate the EKU incident", but
rather more specifically, "what is your plan to ensure every Sub-CA able to
be revoked as required by 4.9.1.2", which almost invariably means
automating certificate issuance and regularly rotating intermediates. If we
continue to allow CAs to place Mozilla, or broadly, browsers, as somehow
responsible for the consequences of the CA's design decisions, things will
only get worse.

Setting aside the security risk factors, which understandably for Mozilla
are seen as low, at its core, this is a design issue for any CA that can't
or doesn't meet the obligations they warranted Mozilla, and the broader
community, that they would meet. Getting to a path where this design issue,
this lack of agility, is remediated is essential, not just in the "oh no,
what if the key is compromised" risk, but within the broader "how do we
have an agile ecosystem?" Weak entropy with serial numbers "should" have
been the wake-up call on investing in this.

Ryan Sleevi

unread,
Jul 2, 2020, 9:09:36 PM7/2/20
to Tim Hollebeek, Mozilla, Peter Gutmann, ry...@sleevi.com
Thanks Tim.

It’s deeply reassuring to see DigiCert tackling this problem responsibly
and head-on.

And thank you for particularly calling attention to the fact that blindly
adding id-pkix-ocsp-nocheck to these ICAs introduces worse security
problems. This is why RFC 6960 warns so specifically on this.

What does a robust design look like?
- Omit the EKU for ICAs. You can work around the ADCS issue using Sectigo’s
guidance.
- For your actual delegated responders, omitting OCSP URLs can help “some”
clients, but not all. A sensible minimum profile is:
- basicConstraints:CA=FALSE
- extKeyUsage=id-kp-OCSPSigning (and ONLY that)
- validity period of 90 days or less (30?)
- id-pkix-ocsp-nocheck

basicConstraints is to guarantee it works with Firefox. EKU so it’s a
delegated responder (and only that). Short lived because nocheck means it’s
high risk.

Invariably, any profile (e.g. in the CABForum) would also need to ensure
that these keys are protected to the same assurance level as CA keys,
because of the similar function they pose. I had previously proposed both
the lifetime and protection requirements in CABF, but met significant
opposition. This still lives in
https://github.com/sleevi/cabforum-docs/pull/2/files , although several of
these changes have found their way in through other ballots, such as SC31
in the SCWG if the CABF.

On Thu, Jul 2, 2020 at 6:37 PM Tim Hollebeek <tim.ho...@digicert.com>
wrote:
> > > Ryan Sleevi via dev-security-policy
> > > <dev-secur...@lists.mozilla.org>
> > > writes:
> > >
> > > >Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST
> > > include
> > > >an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP Delegated
> > > >Responder within Section 4.2.2.2 as indicated by the presence of the
> > > id-kp-
> > > >OCSPSigning as an EKU.
> > >
> > > Unless I've misread your message, the problem isn't the presence or
> > > not of a nocheck extension but the invalid presence of an OCSP EKU:
> > >
> > > >I've flagged this as a SECURITY matter [...] the Issuing CA has
> > > >delegated
> > > the
> > > >ability to mint arbitrary OCSP responses to this third-party
> > >
> > > So the problem would be the presence of the OCSP EKU when it shouldn't
> > > be there, not the absence of the nocheck extension.
> >
> >
> > Not quite. It’s both.
> >
> > The BR violation is caused by the lack of the extension.
> >
> > The security issue is caused by the presence of the EKU.
> >
> > However, since some CAs only view things through the lens of BR/program
> > violations, despite the sizable security risk they pose, the compliance
> incident
> > is what is tracked. The fact that it’s security relevant is provided so
> that CAs
> > understand that revocation is necessary, and that it’s also not
> sufficient,
> > because of how dangerous the issue is.

Filippo Valsorda

unread,
Jul 3, 2020, 12:32:33 AM7/3/20
to ry...@sleevi.com, Paul van Brouwershaven, dev-security-policy
2020-07-02 10:40 GMT-04:00 Ryan Sleevi via dev-security-policy <dev-secur...@lists.mozilla.org>:
> On Thu, Jul 2, 2020 at 10:34 AM Paul van Brouwershaven via
> dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> > I did do some testing on EKU chaining in Go, but from my understand this
> > works the same for Microsoft:
> >
>
> Go has a bug https://twitter.com/FiloSottile/status/1278501854306095104

Yep. In fact, Go simply doesn't have an OCSP verifier. We should fix that! I filed an issue: https://golang.org/issues/40017 <https://github.com/golang/go/issues/40017>

The pieces are there (OCSP request serialization and response parsing, signature verification, a chain builder) but the logic stringing them together is not. That includes building the chain without requesting the EKU up the path, and then checking the EKU only on the Responder itself.

It's unfortunate that the Mozilla requirement (that the Responder must be an EE) is not standard, because that would have allowed the OCSP EKU to work like any other, nested up the chain, but that's just not how it works and it's too late to change, so it has to be special-cased out of the chain nesting requirement, or it wouldn't be possible to mint an Intermediate that can in turn mint Responders, without making the Intermediate a Responder itself.

Pedro Fuentes

unread,
Jul 3, 2020, 3:24:05 AM7/3/20
to mozilla-dev-s...@lists.mozilla.org
>
> Yes. But that doesn't mean we blindly trust the CA in doing so. And that's
> the "security risk".

But the point then is that a delegated responder that had the required "noCheck" extension wouldn't be affected by this issue and CAs wouldn't need to react, and therefore the issue to solve is the "mis-issuance" itself due to the lack of the extension, not the fact that the CA certificate could be used to do delegated responses for the same operator of the Root, which is acceptable, as you said.

In fact the side effect is that delegated responders operated externally that have the required no check extension don't seem to be affected by the issue and would be deemed acceptable, without requiring further action to CAs, while the evident risk problem is still there.

>
> I can understand that our views may differ: you may see 3P as "great risk"
> and 1p as "acceptable risk". However, from the view of a browser or a
> relying party, "1p" and "3p" are the same: they're both CAs. So the risk is
> the same, and the risk is unacceptable for both cases.

But this is not actually like that, because what is required now to CAs is to react appropriately to this incident, and you are imposing a unique approach while the situations are fundamentally different. It's not the same the derivations of this issue for CAs that had 3P delegation (or a mix of 1P and 3P), and the ones, like us, that don't have such delegation.

In our particular case, where we have three affected CAs, owned and operated by WISeKey, we are proposing this action plan, for which we request feedback:
1.- Monday, new CAs will be created with new keys, that will be used to substitute the existing ones
2.- Monday, the existing CAs would be reissued with the same keys, removing the OCSP Signing EKU and with A REDUCED VALIDITY OF THREE MONTHS
3.- The existing CAs will be disabled for any new issuance, and will only be kept operative for signing CRLs and to attend revocation requests
4.- Within the 7 days period, the previous certificate of the CAs will be revoked, updating CCADB and OneCRL
5.- Once the re-issued certificates expire, we will destroy the keys and write the appropriate report

In my humble opinion, this plan is:
- Solving the BR compliance issue by revoking the offending certificate within the required period
- Reducing even more the potential risk of hypothetical misuse of the keys by establishing a short life-time

I hope this plan is acceptable.

Best,
Pedro

Paul van Brouwershaven

unread,
Jul 3, 2020, 6:35:49 AM7/3/20
to ry...@sleevi.com, MDSP
For those who are interested, in contrast to the direct EKU validation with
Test-Certificate, certutil does validate the OCSP signing EKU on the
delegated OCSP signing certificate but doesn't validate the
certificate chain for the OCSP signing EKU.

Full test script and output can be found here:
https://gist.github.com/vanbroup/84859cd10479ed95c64abe6fcdbdf83d

Ryan Sleevi

unread,
Jul 3, 2020, 7:26:07 AM7/3/20
to Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
Hi Pedro,

I’m not sure how best to proceed here. It seems like we’ve reached a point
where you’re wanting to discuss possible ways to respond to this, as a CA,
and it feels like this should be captured on the bug.

I’m quite worried here, because this reply demonstrates that we’re at a
point where there is still a rather large disconnect, and I’m not sure how
to resolve it. It does not seem that there’s an understanding here of the
security issues, and while I want to help as best I can, I also believe
it’s appropriate that we accurately consider how well a CA understands
security issue as part of considering incident response. I want there to be
a safe space for questions, but I’m also deeply troubled by the confusion,
and so I don’t know how to balance those two goals.

On Fri, Jul 3, 2020 at 3:24 AM Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> >
> > Yes. But that doesn't mean we blindly trust the CA in doing so. And
> that's
> > the "security risk".
>
> But the point then is that a delegated responder that had the required
> "noCheck" extension wouldn't be affected by this issue and CAs wouldn't
> need to react, and therefore the issue to solve is the "mis-issuance"
> itself due to the lack of the extension, not the fact that the CA
> certificate could be used to do delegated responses for the same operator
> of the Root, which is acceptable, as you said.


I don’t understand why this is difficult to understand. If you have the
noCheck extension, then as RFC 6960, you need to make that certificate
short lived. The BRs require the cert have the extension.

Similarly, if something goes wrong with such a responder, you also have to
consider revoking the root, because it as-bad-as a root key compromise.

In fact the side effect is that delegated responders operated externally
> that have the required no check extension don't seem to be affected by the
> issue and would be deemed acceptable, without requiring further action to
> CAs, while the evident risk problem is still there.


The “nocheck” discussion extension here is to highlight the compliance
issue.

The underlying issue is a security issue: things capable of providing OCSP
responses that shouldn’t be.

It seems you understand the security issue when viewing external sub-CAs:
they can now impact the security of the issuer.

It seems we’re at an impasse for understanding the issue for
internally-operates Sub CAs: this breaks all of the auditable controls and
assurance frameworks, and breaks the security goals of a “correctly”
configured delegated responder, as discussed in the security considerations
throughout RFC 6960.


>
> >
> > I can understand that our views may differ: you may see 3P as "great
> risk"
> > and 1p as "acceptable risk". However, from the view of a browser or a
> > relying party, "1p" and "3p" are the same: they're both CAs. So the risk
> is
> > the same, and the risk is unacceptable for both cases.
>
> But this is not actually like that, because what is required now to CAs is
> to react appropriately to this incident, and you are imposing a unique
> approach while the situations are fundamentally different. It's not the
> same the derivations of this issue for CAs that had 3P delegation (or a mix
> of 1P and 3P), and the ones, like us, that don't have such delegation.


The burden is for your CA to establish that, in the incident response. I’ve
seen nothing from you to reasonably establish that; you just say “but it’s
different”. And that worries me, because it seems you don’t recognize that
all of the controls and tools and expectations we have, both in terms of
audits but also in all of the checks we make (for example, with crt.sh)
*also* lose their credibility for as long as this exists.

Again, I understand and appreciate the view that you seem to be advocating:
“If nothing goes wrong, no one is harmed. If third-parties were involved,
things could go wrong, so we understand that. But we won’t let anything go
wrong ourselves.”

But you seem to be misunderstanding what I’m saying: “If anything goes
wrong, we will not be able to detect it, and all of our assumptions and
safety features will fail. We could try and design new safety features, but
now we’re having to literally pay for your mistake, which never should have
happened in the first place.”

That is a completely unfair and unreasonable thing for WISeKey to ask of
the community: for everyone to change and adapt because WISeKey failed to
follow the expectations.

The key destruction is the only way I can see being able to provide some
assurance that “things won’t go wrong, because it’s impossible for them to
go wrong, here’s the proof”

Anything short of that is asking the community to either accept the
security risk that things can go wrong, or for everyone to go modify their
code, including their tools to do things like check CT, to appropriately
guard against that. Which is completely unreasonable. That’s how
fundamental this breaks the assumptions here.


In our particular case, where we have three affected CAs, owned and
> operated by WISeKey, we are proposing this action plan, for which we
> request feedback:
> 1.- Monday, new CAs will be created with new keys, that will be used to
> substitute the existing ones
> 2.- Monday, the existing CAs would be reissued with the same keys,
> removing the OCSP Signing EKU and with A REDUCED VALIDITY OF THREE MONTHS


To match your emphasis, THIS DOES NOTHING to solve the security problem. It
doesn’t *matter* the short validity of the new, what matters is the *old*
certificates and whether the *old* certificates private keys still exist.
Which is the thing you’re still continuing.


> 3.- The existing CAs will be disabled for any new issuance, and will only
> be kept operative for signing CRLs and to attend revocation requests
> 4.- Within the 7 days period, the previous certificate of the CAs will be
> revoked, updating CCADB and OneCRL


This does nothing to address the security risk. It *only* addresses the
compliance issue.

5.- Once the re-issued certificates expire, we will destroy the keys and
> write the appropriate report


This ignores the security issues and just focuses on the compliance issues.
Which is, I think, an extremely poor response for a CA. If that was the
response, then like I said, the mitigation I would encourage clients to do
is remove trust in those intermediates. Since that turns out to be rather
difficult, I think the safest option would be to remove trust in the root.

The most important thing I want CAs to solve is the security issue. That’s
why I highlighted *every* CA, not just those I reported, needs to examine
their systems. The point isn’t to focus on compliance and ignore security:
it’s to solve the problem at its core.

Because of the nature of this issue, anything short of revoking the
intermediates with a key destruction is asking “trust us to not screw up
but we have no way to prove we didn’t and detecting it is now even harder
and oh if we do you have to revoke the issuer CA/Root anyways”. Rather than
wait for something to go wrong, and accept that risk, I’d rather we just
tackle it head on and start removing trust in whatever is needed. That’s
how big the risk is, and how just hoping things won’t go wrong isn’t a
strategy.

In my humble opinion, this plan is:
> - Solving the BR compliance issue by revoking the offending certificate
> within the required period
> - Reducing even more the potential risk of hypothetical misuse of the keys
> by establishing a short life-time
>
> I hope this plan is acceptable.


This plan still highlights a misunderstanding about the security issues,
and as a consequence, doesn’t seem to understand that it doesn’t reduce the
potential risk. Revoking the old intermediate does nothing for the security
risk, unless and until the key is no longer usable, or everyone in the
world changes their code to defend against these situations. That’s just
how this bug works.

Pedro Fuentes

unread,
Jul 3, 2020, 8:06:19 AM7/3/20
to mozilla-dev-s...@lists.mozilla.org
Ryan,
I don’t think I’m failing to see the security problem, but we evidently have different perception of the risk level for the particular case of internal delegation.
Anyway I will just cease in my intent and just act as it’s expected, looking as guidance to the reaction of other CAs where possible.

I would just have a last request for you. I would appreciate if you can express your views on Ben’s message about Mozilla’s position, in particular about the 7-day deadline.
I think it’s of extreme benefit for all if the different browsers are aligned.

Thanks,
Pedro

Rob Stradling

unread,
Jul 3, 2020, 8:28:59 AM7/3/20
to ry...@sleevi.com, Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
On 03/07/2020 12:24, Ryan Sleevi via dev-security-policy wrote:
<snip>
> The key destruction is the only way I can see being able to provide some
> assurance that “things won’t go wrong, because it’s impossible for them to
> go wrong, here’s the proof”

Ryan, distrusting the root(s) would be another way to provide this
assurance (for up-to-date clients anyway), although I'd be surprised if
any of the affected CAs would prefer to go that route!

--
Rob Stradling
Senior Research & Development Scientist
Sectigo Limited

Arvid Vermote

unread,
Jul 3, 2020, 10:04:58 AM7/3/20
to ry...@sleevi.com, mozilla-dev-security-policy
GlobalSign recognizes the reported security issue and associated risk, and
is working on a plan to remediate the impacted CA hierarchies with first
priority on terminating those branches that include issuing CA with private
keys outside of GlobalSign's realm. We will soon share an initial plan on
our Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1649937.

One question we have for the root store operators specifically is what type
of assurance they are looking for on the key destruction activities. In the
past we've both done key destruction ceremonies without and with (e.g. in
the case of addressing a compliance issue like
https://bugzilla.mozilla.org/show_bug.cgi?id=1591005) an external auditor
witnessing the destruction and issuing an independent ISAE3000 witnessing
report.

> -----Original Message-----
> From: dev-security-policy <dev-security-...@lists.mozilla.org>
On
> Behalf Of Ryan Sleevi via dev-security-policy
> Sent: woensdag 1 juli 2020 23:06
> To: mozilla-dev-security-policy
<mozilla-dev-s...@lists.mozilla.org>
> Subject: SECURITY RELEVANT FOR CAs: The curious case of the Dangerous
> Delegated Responder Cert
>
> I've created a new batch of certificates that violate 4.9.9 of the BRs,
which was
> introduced with the first version of the Baseline Requirements as a MUST.
This is
> https://misissued.com/batch/138/
>
> A quick inspection among the affected CAs include O fields of: QuoVadis,
> GlobalSign, Digicert, HARICA, Certinomis, AS Sertifitseeimiskeskus,
Actalis,
> Atos, AC Camerfirma, SECOM, T-Systems, WISeKey, SCEE, and CNNIC.
>
> Section 4.9.9 of the BRs requires that OCSP Delegated Responders MUST
> include an id-pkix-ocsp-nocheck extension. RFC 6960 defines an OCSP
> Delegated Responder within Section 4.2.2.2 as indicated by the presence of
the
> in the past [3], which highlighted the security impact of this. I've
flagged this as a
> SECURITY matter for CAs to carefully review, because in the cases where a
> third-party, other than the Issuing CA, operates such a certificate, the
Issuing CA
> has delegated the ability to mint arbitrary OCSP responses to this

Ryan Sleevi

unread,
Jul 3, 2020, 10:43:47 AM7/3/20
to Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
On Fri, Jul 3, 2020 at 8:06 AM Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Ryan,
> I don’t think I’m failing to see the security problem, but we evidently
> have different perception of the risk level for the particular case of
> internal delegation.
> Anyway I will just cease in my intent and just act as it’s expected,
> looking as guidance to the reaction of other CAs where possible.


Again, I don’t disagree that it’s reasonable to see a difference between 1P
and 3P if you’re a CA. But I look at the risk of “what attacks are now
enabled if this is unaddressed”?

Put differently, if a CA used a long-lived delegated responder, and I found
out about it, I would absolutely be concerned and expect explanations.
However, if the “worst” thing the delegated responder could do, if
compromised, was “simply” sign OCSP responses, that’s at least better than
the situation we’re in here. Because these delegated responders can, in
many cases, cause new issuance, and because they are actively signing
things other than OCSP responses (e.g. certificates), the threat model
becomes unreasonably complex.

The BRs have clear guidance on who absorbs the risk for a CA’s mistake: the
CA and anyone who has obtained certificates from the CA. Historically,
browsers have rather nobly stepped in and absorbed that risk for the
ecosystem, introducing ever more complex solutions to try to allow site
operators to continue with business as usual with minimal disruption. But
that’s not what the “social contract” of a CA says should happen, and it’s
not what the CP/CPS of the CA says will happen.

It’s true that 4.9.1.2 “only” demands revocation, but any CA not treating
this as recognizing the security impact and the similarity to a key misuse
and/or compromise - even if “hypothetically” - does a disservice to all.

I would just have a last request for you. I would appreciate if you can
> express your views on Ben’s message about Mozilla’s position, in particular
> about the 7-day deadline.
> I think it’s of extreme benefit for all if the different browsers are
> aligned.


I participate here largely in an individual capacity, except as specified.
I’ve already shared my risk analysis with Ben on that thread, but I also
think it would be grossly negligent for a CA to try to look for “browser
alignment” as a justification for ignoring the BRs, especially when so many
platforms and libraries are put at risk by the CA’s misissuance. It sounds
like you’re looking for an explicit “exception”, and while Ben’s message
seems like a divergence from past Mozilla positions, I think it at least
maintains consistency that “this is an incident, no exceptions”.

Again, while wanting to ensure a safe space for questions, as a CA, your
responsibility is to understand these issues and act accordingly. I wholly
appreciate wanting to have an open and transparent discussion about the
facts, and I am quite sensitive to the fact that there very well can be
information being overlooked. As I said, and will say again: this is for
your incident response to demonstrate, and as with any CA, and for any
incident, you will be judged on how well and how timely you respond to the
risks. Similarly, if you fail to revoke on time, how comprehensively you
mitigate the lack of timeliness so that you can comprehensively demonstrate
it will never happen again.

Every time we encounter some non-compliance with an intermediate, CAs push
back on their 4.9.1.2 obligations, saying it would be too disruptive. Heck,
we’re still in a place where CAs are arguing even revoking the Subscriber
cert under 4.9.1.1 is too disruptive, despite the CA claiming they would do
so. This **has** to change, and so it needs to be clear that the first
order is to expect a CA to **do what they said they would**, and revoke on
the timetable defined. If there truly is an exceptional situation that
prevents this, and the CA has a second incident for complying with the BRs
for not revoking, then the only way that can or should not result in
distrust of the CA is if their incident report can show that they
understand the issue sufficiently and can commit to never delaying
revocation again by showing comprehensively the step they are taking.

There is little evidence that the majority of CAs are capable of this, but
it’s literally been a Baseline Requirement since the start. For lack of a
better analogy: it’s a borrower who is constantly asking for more and more
credit to keep things going, and so they don’t default. If they default,
you are unlikely to get your money back, but if you continue to loan them
more and more, you’re just increasing your own risk for if/when they do
default. The CA is the borrower, defaulting is being distrusted, browsers
are the lenders, and the credit being extended is how flexible they are
when misissuance events occur. Taking a delay on revocation for this issue
is asking for a *huge* loan. For some CAs, that’s beyond the credit they
have available, and the answer is no. For others, it might be yes. Just
like credit scores are used in credit, different CAs have different risks
based on how well they’ve handled past issues and, in some part, based on
how well they handle this issue.

The best way you can avoid taking on new “trust” debt, and even potentially
pay down some of any existing debt caused by past incidents, is to promptly
revoke and provide a key destruction ceremony report to that effect. Short
of that, it’s going to depend on the incident report as to what happens and
whether further credit is extended.

>

Ryan Sleevi

unread,
Jul 3, 2020, 10:47:56 AM7/3/20
to Arvid Vermote, mozilla-dev-security-policy, ry...@sleevi.com
On Fri, Jul 3, 2020 at 10:04 AM Arvid Vermote <arvid....@globalsign.com>
wrote:

> GlobalSign recognizes the reported security issue and associated risk, and
> is working on a plan to remediate the impacted CA hierarchies with first
> priority on terminating those branches that include issuing CA with private
> keys outside of GlobalSign's realm. We will soon share an initial plan on
> our Bugzilla ticket https://bugzilla.mozilla.org/show_bug.cgi?id=1649937.
>
> One question we have for the root store operators specifically is what type
> of assurance they are looking for on the key destruction activities. In the
> past we've both done key destruction ceremonies without and with (e.g. in
> the case of addressing a compliance issue like
> https://bugzilla.mozilla.org/show_bug.cgi?id=1591005) an external auditor
> witnessing the destruction and issuing an independent ISAE3000 witnessing
> report.


Since the goal here is to be able to demonstrate, with some reasonable
assurance, that this key will not come back and haunt the ecosystem, my
intent of the suggestion was yes, an independently witnessed ceremony with
an appropriate report to that fact (e.g. ISAE300)

The reason for this is that so much of our current design around controls
and audits assume that once something is revoked, the key can no longer do
harm and is not interesting if it gets compromised. This threat model
defeats that assumption, because for the lifetime of the responder
certificate(s) associated with that key, it can be misused to revoke
itself, or its siblings, and cause pain anew.

I suspect that the necessity of destruction ceremony is probably influenced
by a variety of factors, such as how long the responder cert is valid for.
This is touched on some in RFC 6960. I don’t know what the “right” answer
is, but my gut is that any responder cert valid for more than a year from
now would benefit from such a report. If it’s less than a year out, and
internally operated, that maybe is reasonable to not require a report? I’m
not sure where that line is, and this is where the CAs can share their
analysis of the facts to better inform and find the “right” balance here.

>

Peter Bowen

unread,
Jul 3, 2020, 10:58:06 AM7/3/20
to Ryan Sleevi, Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
Ryan,

I have read through this thread and am also somewhat perplexed.

I want to be clear, I'm posting only for myself, as an individual, not
on behalf of any current or former employers.

On Fri, Jul 3, 2020 at 4:26 AM Ryan Sleevi via dev-security-policy
<dev-secur...@lists.mozilla.org> wrote:
> On Fri, Jul 3, 2020 at 3:24 AM Pedro Fuentes via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> > >
> > > Yes. But that doesn't mean we blindly trust the CA in doing so. And that's
> > > the "security risk".
> >
> > But the point then is that a delegated responder that had the required
> > "noCheck" extension wouldn't be affected by this issue and CAs wouldn't
> > need to react, and therefore the issue to solve is the "mis-issuance"
> > itself due to the lack of the extension, not the fact that the CA
> > certificate could be used to do delegated responses for the same operator
> > of the Root, which is acceptable, as you said.
>
>
> I don’t understand why this is difficult to understand. If you have the
> noCheck extension, then as RFC 6960, you need to make that certificate
> short lived. The BRs require the cert have the extension.

I think this is difficult to understand because you are adding
requirements that don't currently exist.

There is nothing in the BRs or other obligations of the CA to make a
delegated OCSP responder certificate short-lived. From 6960:

"CAs may choose to issue this type of certificate with a very short
lifetime and renew it frequently."

6960 explicitly says "may" is defined in RFC 2119, which says:

"MAY This word, or the adjective "OPTIONAL", mean that an item is
truly optional."

Contrast with "should" which is "This word, or the adjective
"RECOMMENDED", mean that there may exist valid reasons in particular
circumstances to ignore a particular item, but the full implications
must be understood and carefully weighed before choosing a different
course."

While it may be viewed as best practice to have short lived responder
certificates, it must not be viewed as a hard requirement for the BRs
or for the Mozilla program. As you have pointed out previously, a
browser could make this a requirement, but I am unaware of any
publicly available requirement ot do so.

> Similarly, if something goes wrong with such a responder, you also have to
> consider revoking the root, because it as-bad-as a root key compromise.

I think we can all agree to this point.

> In fact the side effect is that delegated responders operated externally
> > that have the required no check extension don't seem to be affected by the
> > issue and would be deemed acceptable, without requiring further action to
> > CAs, while the evident risk problem is still there.
>
>
> The “nocheck” discussion extension here is to highlight the compliance
> issue.
>
> The underlying issue is a security issue: things capable of providing OCSP
> responses that shouldn’t be.
>
> It seems you understand the security issue when viewing external sub-CAs:
> they can now impact the security of the issuer.
>
> It seems we’re at an impasse for understanding the issue for
> internally-operates Sub CAs: this breaks all of the auditable controls and
> assurance frameworks, and breaks the security goals of a “correctly”
> configured delegated responder, as discussed in the security considerations
> throughout RFC 6960.

This is where I disagree. Currently CAs can create as many delegated
OCSP responder certificates as they like with as many distinct keys as
they like. There are no public requirements from any browser on the
protection of OCSP responder keys, as far as I know. The few
requirements on revocation information are to provide accurate
information and provide the information within 10 seconds "under
normal operating conditions" (no SLO if the CA determines it is not
operating under normal conditions).

For the certificates you identified in the beginning of this thread,
we know they have a certain level of key protection - they are all
required to be managed using cryptographic modules that are validated
as meeting overall Level 3 requirements in FIPS 140. We also know
that there CAs are monitoring these keys as they have an obligation in
BR 6.2.6 to "revoke all certificates that include the Public Key
corresponding to the communicated Private Key" if the "Subordinate
CA’s Private Key has been communicated to an unauthorized person or an
organization not affiliated with the Subordinate CA".

I agree with Pedro here. If the CA has control over the keys in the
certificates in question, then I do not see that there is a risk that
is greater than already exists. The CA can determine that these are
approved OCSP responders and easily assess whether they have controls
in place since the creation of the certificate that provide assurance
that all OCSP responses signed using the key were accurate (if any
such responses exist). They can also easily validate that they have
controls around these keys to provide assurance that any future OCSP
responses signed using the key will be accurate, the same that they
presumably do for their other OCSP responders.

So, from my personal view, what needs to happen here is that each CA
needs to acknowledge each of the keys in the certificats as a valid
OCSP responder key, according to their internal procedures for such,
if they have not already done so. Then they need to revoke the
certificates and issue new certificates with different serial numbers
that do not have the OCSP EKU to correct the certificate profile
issue. While not required, they should also consider having controls
in place for the lifetime of the key (until destruction) to provide
assurance that any OCSP responses it signs are accurate, as we know
some certificate status consumers may not check the validity of the
response signer certificate.

This situation does suggest that root programs should consider adding
public requirements around:
1) Delegation of OCSP response generation to third parties
2) Maximum validity period of OCSP responder certificates
3) Use of CA private keys, specifically requirements for data they sign

I also will not that any CA that determines that a key that has been
included a certificate usable for OCSP response signing but that key
is not authorized for OCSP response generation should immediately
ensure destruction of that key and should document their consideration
to revoke the CA that issued such certificate so their auditor can
verify that they considered the risks of doing so.

Thanks,
Peter

Ryan Sleevi

unread,
Jul 3, 2020, 12:18:49 PM7/3/20
to Peter Bowen, Ryan Sleevi, Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
On Fri, Jul 3, 2020 at 10:57 AM Peter Bowen <pzb...@gmail.com> wrote:

> While it may be viewed as best practice to have short lived responder
> certificates, it must not be viewed as a hard requirement for the BRs
> or for the Mozilla program. As you have pointed out previously, a
> browser could make this a requirement, but I am unaware of any
> publicly available requirement ot do so.
>

Thanks, and I think you're making a useful clarification here. The 'need'
being talked about is the logical consequence of a secure design, and the
assumptions that go with it, as well as the role the BRs play in "profiling
down" RFC 5280-and-related into sensible subsets appropriate for their use
case.

I think we're in agreement here that nocheck is required, and that the
consequences of nocheck's presence, namely:
CAs issuing such a certificate should realize that a compromise of
the responder's key is as serious as the compromise of a CA key
used to sign CRLs, at least for the validity period of this
certificate. CAs may choose to issue this type of certificate with
a very short lifetime and renew it frequently.

As BR-consuming clients, the majority of implementations I looked at don't
recursively check revocation for the delegated responder. That is, rather
than "nocheck" determining client behaviour (vs its absence), "nocheck" is
used to reflect what the clients will do regardless. This can be
understandably seen as 'non-ideal', but gets back into some of the
discussion with Corey regarding profiles and client behaviours.

"need" isn't an explicitly spelled out requirement, I agree, but falls from
the logical consequences of designing such a system and ensuring equivalent
security properties. For example, consider 4.9.10's requirement that OCSP
responses for subordinate CA certificates not have a validity greater than
1 year (!!). We know that's a clear security goal, and so a CA needs to
ensure they can meet that property. If issuing a Delegated Responder, it
logically follows that its validity period should be a year or less,
because if that Delegated Responder is compromised, the objective of 4.9.10
can't be fulfilled. I agree that a CA might argue "well, we published a new
response, and 4.9.10 doesn't say anything about having to result, just
perform the action", but I think we can agree that such an approach, even
if technically precise, calls into question much of their overall
interpretation of the BRs.


> > It seems we’re at an impasse for understanding the issue for
> > internally-operates Sub CAs: this breaks all of the auditable controls
> and
> > assurance frameworks, and breaks the security goals of a “correctly”
> > configured delegated responder, as discussed in the security
> considerations
> > throughout RFC 6960.
>
> This is where I disagree. Currently CAs can create as many delegated
> OCSP responder certificates as they like with as many distinct keys as
> they like. There are no public requirements from any browser on the
> protection of OCSP responder keys, as far as I know. The few
> requirements on revocation information are to provide accurate
> information and provide the information within 10 seconds "under
> normal operating conditions" (no SLO if the CA determines it is not
> operating under normal conditions).
>

I suspect this is the reopening of the discussion about "the CA
organization" or "the CA certificate"; does 6.2.7 apply to all Private Keys
that logically make up the CA "the organization"'s services, or is 6.2.7
only applicable to keys with CA:TRUE. Either extreme is an unsatisfying
answer: do the TLS keys need to be on FIPS modules? (no, ideally not). Does
this only apply to CA keys and not to delegated responders (no, ideally
not).

Going back to 6960 and the requirement of pkix-nocheck, we know that such a
responder certificate is 'as powerful as' the Private Key associated with
the CA Certificate for which the responder is a responder for. Does the
short-lived validity eliminate the need for protection?

I suspect when you disagree, is with respect to the auditable
controls/assurance frameworks, and less with respect to security goals
captured in 6960, since we seemed to agree on those above.


> For the certificates you identified in the beginning of this thread,
> we know they have a certain level of key protection - they are all
> required to be managed using cryptographic modules that are validated
> as meeting overall Level 3 requirements in FIPS 140. We also know
> that there CAs are monitoring these keys as they have an obligation in
> BR 6.2.6 to "revoke all certificates that include the Public Key
> corresponding to the communicated Private Key" if the "Subordinate
> CA’s Private Key has been communicated to an unauthorized person or an
> organization not affiliated with the Subordinate CA".
>

Sure, but we know that such revocation is largely symbolic in the existence
of these certificates for the vast majority of clients, and so the security
goal cannot be reasonably demonstrated while that Private Key still exists.

Further, once this action is performed according to 6.2.6, it disappears
with respect to the obligations under the existing auditing/reporting
frameworks. This is a known deficiency of the BRs, which you rather
comprehensively tried to address when representing a CA member of the
Forum, in your discussion about object hierarchies and signing actions. A
CA may have provisioned actions within 6.2.10 of their CPS, but that's not
a consistent baseline that they can rely on.

At odds here is how to square with the CA performing the action, but not
achieving the result of that action?


> I agree with Pedro here. If the CA has control over the keys in the
> certificates in question, then I do not see that there is a risk that
> is greater than already exists. The CA can determine that these are
> approved OCSP responders and easily assess whether they have controls
> in place since the creation of the certificate that provide assurance
> that all OCSP responses signed using the key were accurate (if any
> such responses exist). They can also easily validate that they have
> controls around these keys to provide assurance that any future OCSP
> responses signed using the key will be accurate, the same that they
> presumably do for their other OCSP responders.
>

But that's not an assurance "we", the relying party, have. We don't have a
way to know what process the CA used to assess what controls they have in
place, and whether those controls were reliable. After all, CAs were
supposed to have controls in place with respect to Delegated Responders,
and this incident is, in part, because some CAs assessed those controls as
compliant, but they were not. This is the breakdown of assurance I spoke to
above. You yourself are familiar with the fact that 5.4.1 of the BRs
doesn't actually require that CAs maintain logs of the messages they've
signed via the HSM, so that's not a guarantee we can simply take at face
value.

Obviously, we're talking about a spectrum of responses, and I am of course
interested if there are options other than what I've outlined, based on
facts not yet considered.

I would hope we would agree that if a CA simply revoked the certificate,
and did nothing beyond that (no reissuance, no destruction), it would not
provide the necessary assurance regarding that key.
- The key would be outside the scope of much of the audited activities,
which are presently oriented around "certificates"
- For the lifetime of the certificates that were revoked, we have to
worry that they, or their siblings, may "come back to haunt us"

Pedro's option is to reissue a certificate for that key, which as you point
out, keeps the continuity of CA controls associated with that key within
the scope of the audit. I believe this is the heart of Pedro's risk
analysis justification.
- However, controls like you describe are not ones that are audited, nor
consistent between CAs
- They ultimately rely on the CA's judgement, which is precisely the
thing an incident like this calls into question, and so it's understandable
not to want to throw "good money after bad"


> So, from my personal view, what needs to happen here is that each CA
> needs to acknowledge each of the keys in the certificats as a valid
> OCSP responder key, according to their internal procedures for such,
> if they have not already done so. Then they need to revoke the
> certificates and issue new certificates with different serial numbers
> that do not have the OCSP EKU to correct the certificate profile
> issue. While not required, they should also consider having controls
> in place for the lifetime of the key (until destruction) to provide
> assurance that any OCSP responses it signs are accurate, as we know
> some certificate status consumers may not check the validity of the
> response signer certificate.
>

Is this from the point of view of "what meets the compliance obligations"
or "what demonstrates the CA understands the risks, has sufficient
safeguards, and can demonstrate them other than 'trust us'"?

I readily admit, I framed the security issue within a compliance issue
precisely because CAs were having trouble recognizing the security issue. I
would hate to see the response predicated on treating it like a compliance
issue, though. When you talk about "while not required", that is indeed the
heart of what I'm talking about in terms of how do we mitigate the security
risks? "It's a security risk, but we're not required to address it" is, I
think, a reasonable ground to criticize the incident response and the CA's
handling of it.

The question of controls in place for the lifetime of the certificate is
the "cost spectrum" I specifically talked about in
https://www.mail-archive.com/dev-secur...@lists.mozilla.org/msg13530.html
, namely "Well, can't we audit the key usage to make sure it never signs a
delegated OCSP response".

As I mentioned, I think the response to these incidents is going to have to
vary on a CA-by-CA basis, because there isn't a one-size-fits-all category
for mitigating these risk factors. Some CAs may be able to demonstrate a
sufficient number of controls that mitigate these concerns in a way that
some browsers will accept. But I don't think we can treat this as "just" a
compliance failure, particularly when the failure means that the intended
result of a significant number of obligations cannot be met because of it.


> This situation does suggest that root programs should consider adding
> public requirements around:
> 1) Delegation of OCSP response generation to third parties
> 2) Maximum validity period of OCSP responder certificates
> 3) Use of CA private keys, specifically requirements for data they sign
>

Whole heartedly agree that this reveals greater clarity of profiles are
needed, and will be working on resurrecting/rebasing
https://github.com/sleevi/cabforum-docs/pull/2 to try and account for some
of this. Those are "prevent" solutions, and we're focused here on "detect
and mitigate" solutions.

Peter Bowen

unread,
Jul 3, 2020, 4:19:22 PM7/3/20
to Ryan Sleevi, Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
On Fri, Jul 3, 2020 at 9:18 AM Ryan Sleevi <ry...@sleevi.com> wrote:
>
>
>
> On Fri, Jul 3, 2020 at 10:57 AM Peter Bowen <pzb...@gmail.com> wrote:
>>
>> While it may be viewed as best practice to have short lived responder
>> certificates, it must not be viewed as a hard requirement for the BRs
>> or for the Mozilla program. As you have pointed out previously, a
>> browser could make this a requirement, but I am unaware of any
>> publicly available requirement ot do so.
>
>
> Thanks, and I think you're making a useful clarification here. The 'need' being talked about is the logical consequence of a secure design, and the assumptions that go with it, as well as the role the BRs play in "profiling down" RFC 5280-and-related into sensible subsets appropriate for their use case.
>
> I think we're in agreement here that nocheck is required, and that the consequences of nocheck's presence, namely:
> CAs issuing such a certificate should realize that a compromise of
> the responder's key is as serious as the compromise of a CA key
> used to sign CRLs, at least for the validity period of this
> certificate. CAs may choose to issue this type of certificate with
> a very short lifetime and renew it frequently.
>
> As BR-consuming clients, the majority of implementations I looked at don't recursively check revocation for the delegated responder. That is, rather than "nocheck" determining client behaviour (vs its absence), "nocheck" is used to reflect what the clients will do regardless. This can be understandably seen as 'non-ideal', but gets back into some of the discussion with Corey regarding profiles and client behaviours.

So we are in agreement that is a certificate consumer bug if it fails
to check revocation on certificates that do not have nocheck set.
(Yes, I know that is a massive set of negations, sorry.). Good news is
that all the certificates are in the category of "need revocation
checking".


>> > It seems we’re at an impasse for understanding the issue for
>> > internally-operates Sub CAs: this breaks all of the auditable controls and
>> > assurance frameworks, and breaks the security goals of a “correctly”
>> > configured delegated responder, as discussed in the security considerations
>> > throughout RFC 6960.
>>
>> This is where I disagree. Currently CAs can create as many delegated
>> OCSP responder certificates as they like with as many distinct keys as
>> they like. There are no public requirements from any browser on the
>> protection of OCSP responder keys, as far as I know. The few
>> requirements on revocation information are to provide accurate
>> information and provide the information within 10 seconds "under
>> normal operating conditions" (no SLO if the CA determines it is not
>> operating under normal conditions).
>
>
> I suspect this is the reopening of the discussion about "the CA organization" or "the CA certificate"; does 6.2.7 apply to all Private Keys that logically make up the CA "the organization"'s services, or is 6.2.7 only applicable to keys with CA:TRUE. Either extreme is an unsatisfying answer: do the TLS keys need to be on FIPS modules? (no, ideally not). Does this only apply to CA keys and not to delegated responders (no, ideally not).

As I read the BRs today, the requirement only applies to CA keys and
not to keys for delegated responders. However that does not matter in
this case, because all the certificates you identified are for CAs, so
we know their keys are in HSMs.

> Going back to 6960 and the requirement of pkix-nocheck, we know that such a responder certificate is 'as powerful as' the Private Key associated with the CA Certificate for which the responder is a responder for. Does the short-lived validity eliminate the need for protection?
>
> I suspect when you disagree, is with respect to the auditable controls/assurance frameworks, and less with respect to security goals captured in 6960, since we seemed to agree on those above.
>
>>
>> For the certificates you identified in the beginning of this thread,
>> we know they have a certain level of key protection - they are all
>> required to be managed using cryptographic modules that are validated
>> as meeting overall Level 3 requirements in FIPS 140. We also know
>> that there CAs are monitoring these keys as they have an obligation in
>> BR 6.2.6 to "revoke all certificates that include the Public Key
>> corresponding to the communicated Private Key" if the "Subordinate
>> CA’s Private Key has been communicated to an unauthorized person or an
>> organization not affiliated with the Subordinate CA".
>
>
> Sure, but we know that such revocation is largely symbolic in the existence of these certificates for the vast majority of clients, and so the security goal cannot be reasonably demonstrated while that Private Key still exists.
>
> Further, once this action is performed according to 6.2.6, it disappears with respect to the obligations under the existing auditing/reporting frameworks. This is a known deficiency of the BRs, which you rather comprehensively tried to address when representing a CA member of the Forum, in your discussion about object hierarchies and signing actions. A CA may have provisioned actions within 6.2.10 of their CPS, but that's not a consistent baseline that they can rely on.
>
> At odds here is how to square with the CA performing the action, but not achieving the result of that action?

As long as the key is a CA key, the obligations stand.

>> I agree with Pedro here. If the CA has control over the keys in the
>> certificates in question, then I do not see that there is a risk that
>> is greater than already exists. The CA can determine that these are
>> approved OCSP responders and easily assess whether they have controls
>> in place since the creation of the certificate that provide assurance
>> that all OCSP responses signed using the key were accurate (if any
>> such responses exist). They can also easily validate that they have
>> controls around these keys to provide assurance that any future OCSP
>> responses signed using the key will be accurate, the same that they
>> presumably do for their other OCSP responders.
>
>
> But that's not an assurance "we", the relying party, have. We don't have a way to know what process the CA used to assess what controls they have in place, and whether those controls were reliable. After all, CAs were supposed to have controls in place with respect to Delegated Responders, and this incident is, in part, because some CAs assessed those controls as compliant, but they were not. This is the breakdown of assurance I spoke to above. You yourself are familiar with the fact that 5.4.1 of the BRs doesn't actually require that CAs maintain logs of the messages they've signed via the HSM, so that's not a guarantee we can simply take at face value.
>
> Obviously, we're talking about a spectrum of responses, and I am of course interested if there are options other than what I've outlined, based on facts not yet considered.
>
> I would hope we would agree that if a CA simply revoked the certificate, and did nothing beyond that (no reissuance, no destruction), it would not provide the necessary assurance regarding that key.
> - The key would be outside the scope of much of the audited activities, which are presently oriented around "certificates"
> - For the lifetime of the certificates that were revoked, we have to worry that they, or their siblings, may "come back to haunt us"

Agreed, simply revoking doesn't solve the issue; arguably it makes it
worse than doing nothing.

> Pedro's option is to reissue a certificate for that key, which as you point out, keeps the continuity of CA controls associated with that key within the scope of the audit. I believe this is the heart of Pedro's risk analysis justification.
> - However, controls like you describe are not ones that are audited, nor consistent between CAs
> - They ultimately rely on the CA's judgement, which is precisely the thing an incident like this calls into question, and so it's understandable not to want to throw "good money after bad"

To be clear, I don't necessarily see this as a bad judgement on the
CA's part. Microsoft explicitly documented that _including_ the OCSP
EKU was REQURIED in the CA certificate if using a delegated OCSP
responder (see https://support.microsoft.com/en-us/help/2962991/you-cannot-enroll-in-an-online-certificate-status-protocol-certificate).
Using a delegated OCSP responder can be a significant security
enhancement in some CA designs, such as when the CA key itself is
stored offline.

>> So, from my personal view, what needs to happen here is that each CA
>> needs to acknowledge each of the keys in the certificats as a valid
>> OCSP responder key, according to their internal procedures for such,
>> if they have not already done so. Then they need to revoke the
>> certificates and issue new certificates with different serial numbers
>> that do not have the OCSP EKU to correct the certificate profile
>> issue. While not required, they should also consider having controls
>> in place for the lifetime of the key (until destruction) to provide
>> assurance that any OCSP responses it signs are accurate, as we know
>> some certificate status consumers may not check the validity of the
>> response signer certificate.
>
>
> Is this from the point of view of "what meets the compliance obligations" or "what demonstrates the CA understands the risks, has sufficient safeguards, and can demonstrate them other than 'trust us'"?
>
> I readily admit, I framed the security issue within a compliance issue precisely because CAs were having trouble recognizing the security issue. I would hate to see the response predicated on treating it like a compliance issue, though. When you talk about "while not required", that is indeed the heart of what I'm talking about in terms of how do we mitigate the security risks? "It's a security risk, but we're not required to address it" is, I think, a reasonable ground to criticize the incident response and the CA's handling of it.
>
> The question of controls in place for the lifetime of the certificate is the "cost spectrum" I specifically talked about in https://www.mail-archive.com/dev-secur...@lists.mozilla.org/msg13530.html , namely "Well, can't we audit the key usage to make sure it never signs a delegated OCSP response".
>
> As I mentioned, I think the response to these incidents is going to have to vary on a CA-by-CA basis, because there isn't a one-size-fits-all category for mitigating these risk factors. Some CAs may be able to demonstrate a sufficient number of controls that mitigate these concerns in a way that some browsers will accept. But I don't think we can treat this as "just" a compliance failure, particularly when the failure means that the intended result of a significant number of obligations cannot be met because of it.

As you pointed out, I pushed for a number of improvements in WebTrust
audits when I was involved with operation of a publicly trusted CA.
One of those resulted in practitioner guidance that is relevant here.
In the guidance at
https://www.cpacanada.ca/en/business-and-accounting-resources/audit-and-assurance/overview-of-webtrust-services/practitioner-qualification-and-guidance
, you can see two relevant items:

- Reporting When Certain Criteria Not Applicable as Services Not Performed by CA
- Disclosure of Changes in Scope or Roots with no Activity

This makes it clear that auditors can report on the absence of
activity. In an attestation engagement, the auditor can also provide
an opinion on statements in the attestation that can be tested,
regardless of whether they match a specific criteria in the audit
criteria.

Given this, I believe relying parties and root programs could
determine there are sufficient controls via audit reporting. The CA
can include statements in the assertion that (as applicable) the keys
were not used to sign any OCSP responses or that the keys were not
used to sign OCSP responses for certificates issued by CAs other than
the CA identified as the subject of the CA certificate where key is
bound to that subject.

I think (but an auditor would need to confirm) that an auditor could
be in a position to make statements about past periods based on the
controls they observed at the time, as recorded in their work papers.
For example, a CA might have controls that a given key is only used
during specific ceremonies where all the ceremonies are known to not
contain inappropriate OCSP response signing. Alternatively the
configuration of the HSM and attached systems that the auditor
validated at the time may clearly show that OCSP signing is not
possible and the auditor may have observed controls that the key is
restricted to only be used with the observed system configuration.

I agree that we cannot make blanket statements that apply to all CAs,
but these are some examples where it seems like there are alternatives
to key destruction.

Thanks,
Peter

Ryan Sleevi

unread,
Jul 3, 2020, 5:30:47 PM7/3/20
to Peter Bowen, Ryan Sleevi, Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
On Fri, Jul 3, 2020 at 4:19 PM Peter Bowen <pzb...@gmail.com> wrote:

> >> For the certificates you identified in the beginning of this thread,
> >> we know they have a certain level of key protection - they are all
> >> required to be managed using cryptographic modules that are validated
> >> as meeting overall Level 3 requirements in FIPS 140. We also know
> >> that there CAs are monitoring these keys as they have an obligation in
> >> BR 6.2.6 to "revoke all certificates that include the Public Key
> >> corresponding to the communicated Private Key" if the "Subordinate
> >> CA’s Private Key has been communicated to an unauthorized person or an
> >> organization not affiliated with the Subordinate CA".
> >
> >
> > Sure, but we know that such revocation is largely symbolic in the
> existence of these certificates for the vast majority of clients, and so
> the security goal cannot be reasonably demonstrated while that Private Key
> still exists.
> >
> > Further, once this action is performed according to 6.2.6, it disappears
> with respect to the obligations under the existing auditing/reporting
> frameworks. This is a known deficiency of the BRs, which you rather
> comprehensively tried to address when representing a CA member of the
> Forum, in your discussion about object hierarchies and signing actions. A
> CA may have provisioned actions within 6.2.10 of their CPS, but that's not
> a consistent baseline that they can rely on.
> >
> > At odds here is how to square with the CA performing the action, but not
> achieving the result of that action?
>
> As long as the key is a CA key, the obligations stand.
>

Right, but we're in agreement that these obligations don't negate the risk
posed - e.g. of inappropriate signing - right? And those obligations
ostensibly end once the CA has revoked the certificate, since the status
quo today doesn't require key destruction ceremonies for CAs. I think that
latter point is something discussed in the CA/Browser Forum with respect to
"Audit Lifecycle", but even then, that'd be difficult to work through for
CA keys that were compromised.

Put differently: If the private key for such a responder is compromised, we
lose the ability to be assured it hasn't been used for shenanigans. As
such, either we accept the pain now, and get assurance it /can't/ be used
for shenanigans, or we have hope that it /won't/ be used for shenanigans,
but are in an even worse position if it is, because this would necessitate
revoking the root.

It is, at the core, a gamble on the order of "Pay me now vs pay me later".
If we address the issue now, it's painful up-front, but consistent with the
existing requirements and mitigates the potential-energy of a security
mistake. If we punt on the issue, we're simply storing up more potential
energy for a more painful revocation later.

I recognize that there is a spectrum here on options. In an abstract sense,
the calculus of "the key was destroyed a month before it would have been
compromised" is the same as "the key was destroyed a minute before it would
have been compromised" - the risk was dodged. But "the key was destroyed a
second after it was compromised" is doom.


> > Pedro's option is to reissue a certificate for that key, which as you
> point out, keeps the continuity of CA controls associated with that key
> within the scope of the audit. I believe this is the heart of Pedro's risk
> analysis justification.
> > - However, controls like you describe are not ones that are audited,
> nor consistent between CAs
> > - They ultimately rely on the CA's judgement, which is precisely the
> thing an incident like this calls into question, and so it's understandable
> not to want to throw "good money after bad"
>
> To be clear, I don't necessarily see this as a bad judgement on the
> CA's part. Microsoft explicitly documented that _including_ the OCSP
> EKU was REQURIED in the CA certificate if using a delegated OCSP
> responder (see
> https://support.microsoft.com/en-us/help/2962991/you-cannot-enroll-in-an-online-certificate-status-protocol-certificate
> ).
> Using a delegated OCSP responder can be a significant security
> enhancement in some CA designs, such as when the CA key itself is
> stored offline.
>

Oh, I agree on the value of delegated responders, for precisely that
reason. I think the bad judgement is not trying to find an alternative
solution. Some CAs did, and I think that highlights strength. Other CAs, no
doubt, simply said "Sorry, we can't do it" or "You need to run a different
platform"

I'm not trying to suggest bad /intentions/, but I am trying to say that
it's bad /judgement/, no different than the IP address discussions had in
the CA/B Forum or internal server names. The intentions were admirable, the
execution was inappropriate.
Right, and you know I'm familiar with these options :) I'm thoroughly glad
WebTrust has been responsive to the needs of the ecosystem.

That said, I think it's reasonable to question what exactly the controls
are used, because different auditors' judgement as to the level of
assurance needed to report on those criteria is going to differ, just like
we see different CAs' responses differ.

Let's assume an "ideal" CA behaviour, which would be to ensure that they
use the Detailed Controls Report to include the controls tested as part of
forming the basis of the opinion. Could this provide a level of assurance
appropriate? Very possibly! But ingesting and examining that is simply a
cost that is passed on, from the CA, onto all the Relying Parties, for
which Browsers generally do the bulk of representation of those interests.
Using your example, of including these statements, means that for the
lifetime of the CA associated with that key, browsers would have to be
fastidious in their inspection of reports.

And what happens if things go wrong? Well, we're back to revoking the root.
All we've done is incur greater cost / "trust debt" along the way.


> Given this, I believe relying parties and root programs could
> determine there are sufficient controls via audit reporting. The CA
> can include statements in the assertion that (as applicable) the keys
> were not used to sign any OCSP responses or that the keys were not
> used to sign OCSP responses for certificates issued by CAs other than
> the CA identified as the subject of the CA certificate where key is
> bound to that subject.
>
> I think (but an auditor would need to confirm) that an auditor could
> be in a position to make statements about past periods based on the
> controls they observed at the time, as recorded in their work papers.
> For example, a CA might have controls that a given key is only used
> during specific ceremonies where all the ceremonies are known to not
> contain inappropriate OCSP response signing. Alternatively the
> configuration of the HSM and attached systems that the auditor
> validated at the time may clearly show that OCSP signing is not
> possible and the auditor may have observed controls that the key is
> restricted to only be used with the observed system configuration.
>

CAs have proposed this before, but this again falls into an area of whether
or not it provides a suitable degree of assurance. This is, for example,
part of why new CAs coming in to Mozilla's program need to have keys
subjected to the BRs from their creation; because the level of assurance
with respect to these "pre-verification" events is questionable.


> I agree that we cannot make blanket statements that apply to all CAs,
> but these are some examples where it seems like there are alternatives
> to key destruction.
>

Right, and I want to acknowledge, there are some potentially viable paths
specific to WebTrust, for which I have no faith with respect to ETSI
precisely because of the nature and design of ETSI audits, that, in an
ideal world, could provide the assurance desired. However, the act of that
assurance is to shift the cost of supervision and maintenance onto the
Browser, and I think that's reasonable to be concerned. At best, it would
only be a temporary solution, and the cost would only be justified if there
was a clear set of actionable systemic improvements attached with it that
demonstrated why this situation was exceptional, despite being a BR
violation.

This is the cost calculus for CAs to demonstrate in their incident report,
but I think the baseline expectation should be set that it's asking for
significant cost to be taken on by consumers/browsers, as a way of
offsetting the cost to any of the Subscribers of that CA. In some cases, it
may be a justified tradeoff, particularly for CAs that have been exemplary
models of best practices and/or systemic improvements. In other cases, I
think it's asking too much to be taken on faith, and too much credit to be
extended, in what is fundamentally an "unjust" model in which a CA commits
to something, but then when asked to actually deliver on that commitment,
reneges on it

Pedro Fuentes

unread,
Jul 4, 2020, 6:22:04 AM7/4/20
to mozilla-dev-s...@lists.mozilla.org
El viernes, 3 de julio de 2020, 18:18:49 (UTC+2), Ryan Sleevi escribió:
> Pedro's option is to reissue a certificate for that key, which as you point
> out, keeps the continuity of CA controls associated with that key within
> the scope of the audit. I believe this is the heart of Pedro's risk
> analysis justification.

I didn't want to participate here for now and just learn from other's opinions, but as my name has been evoked, I'd like to make a clarification.

My proposal was not JUST to reissue the certificate with the same key. My proposal was to reissue the certificate with the same key AND a short lifetime (3 months) AND do a proper key destruction after that period.

As I said, this:
- Removes the offending EKU
- Makes the certificate short-lived, for its consideration as delegated responder
- Ensures that the keys are destroyed for peace of mind of the community

And all that was, of course, pondering the security risk based on the fact that the operator of the key is also operating the keys of the Root and is also rightfully operating the OCSP services for the Root.

I don't want to start another discussion, but I just feel necessary making this clarification, in case my previous message was unclear.

Best.

Ryan Sleevi

unread,
Jul 4, 2020, 10:03:44 AM7/4/20
to Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
On Sat, Jul 4, 2020 at 6:22 AM Pedro Fuentes via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> El viernes, 3 de julio de 2020, 18:18:49 (UTC+2), Ryan Sleevi escribió:
> > Pedro's option is to reissue a certificate for that key, which as you
> point
> > out, keeps the continuity of CA controls associated with that key within
> > the scope of the audit. I believe this is the heart of Pedro's risk
> > analysis justification.
>
> I didn't want to participate here for now and just learn from other's
> opinions, but as my name has been evoked, I'd like to make a clarification.
>
> My proposal was not JUST to reissue the certificate with the same key. My
> proposal was to reissue the certificate with the same key AND a short
> lifetime (3 months) AND do a proper key destruction after that period.
>
> As I said, this:
> - Removes the offending EKU
> - Makes the certificate short-lived, for its consideration as delegated
> responder
> - Ensures that the keys are destroyed for peace of mind of the community
>
> And all that was, of course, pondering the security risk based on the fact
> that the operator of the key is also operating the keys of the Root and is
> also rightfully operating the OCSP services for the Root.
>
> I don't want to start another discussion, but I just feel necessary making
> this clarification, in case my previous message was unclear.


Thanks! I really appreciate you clarifying, as I had actually missed that
you proposed key destruction at the end of this. I agree, this is a
meaningfully different proposal that tries to balance the risks of
compliance while committing to a clear transition date.

>

Pedro Fuentes

unread,
Jul 4, 2020, 10:22:06 AM7/4/20
to mozilla-dev-s...@lists.mozilla.org
Thanks, Ryan.
I’m happy we are now in understanding to this respect.

Then I’d change the literally ongoing plan. We should have the new CAs hopefully today. Then I would do maybe also today the reissuance of the bad ones and I’ll revoke the offending certificates during the period.

Best.

Ryan Sleevi

unread,
Jul 4, 2020, 11:10:51 AM7/4/20
to Pedro Fuentes, mozilla-dev-s...@lists.mozilla.org
Pedro: I said I understood you, and I thought we were discussing in the
abstract.

I encourage you to reread this thread to understand why such a response
varies on a case by case basis. I can understand your *attempt* to balance
things, but I don’t think it would be at all appropriate to treat your
email as your incident response.

You still need to holistically address the concerns I raised. As I
mentioned in the bug: either this is a safe space to discuss possible
options, which will vary on a CA-by-CA basis based on a holistic set of
mitigations, or this was having to repeatedly explain to a CA why they were
failing to recognize a security issue.

I want to believe it’s the former, and I would encourage you, that before
you decide to delay revocation, you think very carefully. Have you met the
Mozilla policy obligations on a delay to revocation? Perhaps it’s worth
re-reading those expectations, before you make a decision that will also
fail to uphold community expectations.

Pedro Fuentes

unread,
Jul 4, 2020, 11:27:54 AM7/4/20
to mozilla-dev-s...@lists.mozilla.org
Ryan,
I'm moving our particular discussions to Bugzilla.

I just want to clarify, again, that I'm not proposing to delay the revocation of the offending CA certificate, what I'm proposing is to give more time to the key destruction. Our position right now, is that the certificate would be revoked in any case during the 7 day period.

Thanks,
Pedro

mark.a...@gmail.com

unread,
Jul 4, 2020, 12:52:20 PM7/4/20
to mozilla-dev-s...@lists.mozilla.org
On Friday, July 3, 2020 at 5:30:47 PM UTC-4, Ryan Sleevi wrote:
> On Fri, Jul 3, 2020 at 4:19 PM Peter Bowen wrote:
>

I feel compelled to respond here for the first time even though I have never participated in CA/B forum proceeding and have never read through a single one of the 55 BRs that have been published over the last 8 years.

I was informed yesterday that I would have to replace just over 300 certificates in 5 days because my CA is required by rules from the CA/B forum to revoke its subCA certificate.

This is insane!
Those 300 certificates are used to secure healthcare information systems at a time when the global healthcare system is strained by a global pandemic. I have to coordinate with more than 30 people to make this happen. This includes three subsidiaries and three contract partner organizations as well as dozens of managers and systems engineers. One of my contract partners follows the guidance of an HL7 specification that requires them to do certificate pinning. When we replace these certificates we must give them 30 days lead time to make the change.

After wading through this very long chain of messages I see little discussion of the impact this will have on end users. Ryan Sleevi, in the name of Google, is purporting to speak for the end users, but it is obvious that Ryan does not understand the implication of applying these rules.

Peter Bowen says
> ... simply revoking doesn't solve the issue; arguably it makes it
> worse than doing nothing.

You are absolutely right, Peter. Doctors will not be able to communicate with each other effectively and people could die if the CA/B forum continues to blindly follow its rules without consideration for the greater impact this will have on the security of the internet.

In the CIA triad Availability is as important as Confidentiality. Has anyone done a threat model and a serious risk analysis to determine what a reasonable risk mitigation strategy is?

Mark Arnott

unread,
Jul 4, 2020, 12:53:04 PM7/4/20
to mozilla-dev-s...@lists.mozilla.org
On Friday, July 3, 2020 at 5:30:47 PM UTC-4, Ryan Sleevi wrote:
> On Fri, Jul 3, 2020 at 4:19 PM Peter Bowen wrote:
>
I feel compelled to respond here for the first time even though I have never participated in CA/B forum proceeding and have never read through a single one of the 55 BRs that have been published over the last 8 years.

I was informed yesterday that I would have to replace just over 300 certificates in 5 days because my CA is required by rules from the CA/B forum to revoke its subCA certificate.

This is insane!
Those 300 certificates are used to secure healthcare information systems at a time when the global healthcare