Certificate issues

[Jeremy Rowley]

Hi everyone,



On Friday at 1:00 pm, we accidently introduced a bug into our issuance
system that resulted in five serverAuth-code signing certificates that did
not comply with the Baseline Requirements. The change modified a handful of
code signing certificates into a pseudo- SSL profile. Because they were
intended to be code signing certificates, the certificates issued off a
code-signing intermediate (with code-signing as the sole EKU). The
certificates contain a servauth EKU despite the intermediate's EKU
restriction. The certificates also lack a domain name. Instead, the CN and
dNSName include the code signing applicant's name. Because the certs lack a
domain name and there is an EKU mismatch between the issuer and end entity
certs, the certs can't be misused.



Our systems detected the issue shortly after the change. We corrected the
code, and revoked the certificates. We already scanned our entire
certificate database to ensure these are only the certificates affected by
the bug.



The certificates in question are:

* 02CD2F16F3CA4FCC7378C917FFD5F6A0

* 09A88902AF0698841167E814DB8B3FB8

* 0D7C350D52821BFD2326270B9215DCE5

* 0356D3A74CFA29BB5E65569E0532F134

* 089FBE93D335ADB8BDFCDCF492083B68



The bug was introduced, ironically, in code we deployed to detect potential
errors in cert profiles. This error caused the specified code signing
certificates to think they needed dNSnames and serverAuth. Let me know if
you have questions.



Thanks,

Jeremy

[Ryan Sleevi]

Thanks for posting this, Jeremy.

Are these certificates logged to Certificate Transparency? While not
wanting to suggest I'm doubting you, being able to demonstrate that all
intermediates they chain to are restricted from the serverAuth EKU is
useful.

I realize that's asking you to go above and beyond what you've disclosed so
far. I think if/once we can add clarity to the Baseline Requirements
regarding the scope, it would likely be clearer that these would be out of
scope of the Baseline Requirements, and thus any such disclosure only be
relative to root programs that recognize those paths as code-signing
capable.

[Jeremy Rowley]

They are not currently logged to CT (because they were supposed to be code
signing certificates). We can add them to our log though.



Jeremy

From: Ryan Sleevi [mailto:ry...@sleevi.com]
Sent: Tuesday, April 18, 2017 10:22 AM
To: Jeremy Rowley <jeremy...@digicert.com>
Cc: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Certificate issues







On Tue, Apr 18, 2017 at 12:09 PM, Jeremy Rowley via dev-security-policy
<dev-secur...@lists.mozilla.org

[Nick Lamb]

Hi Jeremy

Given the small number of certificates involved, it might make sense to just convert them to text and mention them inline, or put them somewhere we can all see them - if it's inconvenient to put them into the CT logs.

I think this situation will be useful as evidence of the value of not putting all ones eggs in one basket when it comes to subCA EKUs. If these had come from a subCA which wasn't constrained, we'd have a bigger (though at five specific certificates still manageable) issue.

On Tuesday, 18 April 2017 17:10:39 UTC+1, Jeremy Rowley wrote:
> The bug was introduced, ironically, in code we deployed to detect potential
> errors in cert profiles. This error caused the specified code signing
> certificates to think they needed dNSnames and serverAuth. Let me know if
> you have questions.

This irony might perhaps have been avoided if the code in question didn't have permission (never mind intent) to alter anything. I appreciate that what one would ideally like is to never issue a bad certificate, and to achieve that any checks must happen in a trusted part of the system. But it seems to me that there's a good deal of benefit, at practically zero risk, in building tooling that focuses on monitoring stuff which has already been signed, outside of the protected signing environment; and only subsequently after proving it there giving it the earlier, more powerful (and thus risky) role.

Of course I know nothing of your specific circumstances, this is just a general observation about how I think I'd approach the question of improving issuance quality with minimal risk.

[Gervase Markham]

On 18/04/17 17:22, Ryan Sleevi wrote:
> On Tue, Apr 18, 2017 at 12:09 PM, Jeremy Rowley via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>> code signing certificates into a pseudo- SSL profile. Because they were
>> intended to be code signing certificates, the certificates issued off a
>> code-signing intermediate (with code-signing as the sole EKU).

If this is true, I am not particularly concerned. So as Ryan notes, a
demonstration of this fact would satisfy me that this was not a serious
incident. Thank you for reporting it so promptly.

Gerv

[Jeremy Rowley]

Okay - they are all logged to both Google's CT log and DigiCert's CT log. I
can also send the PEM files shortly.

-----Original Message-----
From: dev-security-policy
[mailto:dev-security-policy-bounces+jeremy.rowley=digice...@lists.mozilla
.org] On Behalf Of Gervase Markham via dev-security-policy
Sent: Tuesday, April 18, 2017 10:59 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Certificate issues

On 18/04/17 17:22, Ryan Sleevi wrote:

> On Tue, Apr 18, 2017 at 12:09 PM, Jeremy Rowley via
> dev-security-policy < dev-secur...@lists.mozilla.org> wrote:
>> code signing certificates into a pseudo- SSL profile. Because they
>> were intended to be code signing certificates, the certificates
>> issued off a code-signing intermediate (with code-signing as the sole
EKU).

If this is true, I am not particularly concerned. So as Ryan notes, a
demonstration of this fact would satisfy me that this was not a serious
incident. Thank you for reporting it so promptly.

Gerv

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Jakob Bohm]

On 18/04/2017 18:47, Nick Lamb wrote:
> Hi Jeremy
>
> Given the small number of certificates involved, it might make sense to just convert them to text and mention them inline, or put them somewhere we can all see them - if it's inconvenient to put them into the CT logs.
>
> I think this situation will be useful as evidence of the value of not putting all ones eggs in one basket when it comes to subCA EKUs. If these had come from a subCA which wasn't constrained, we'd have a bigger (though at five specific certificates still manageable) issue.
>
>
> On Tuesday, 18 April 2017 17:10:39 UTC+1, Jeremy Rowley wrote:
>> The bug was introduced, ironically, in code we deployed to detect potential
>> errors in cert profiles. This error caused the specified code signing
>> certificates to think they needed dNSnames and serverAuth. Let me know if
>> you have questions.
>
> This irony might perhaps have been avoided if the code in question didn't have permission (never mind intent) to alter anything. I appreciate that what one would ideally like is to never issue a bad certificate, and to achieve that any checks must happen in a trusted part of the system. But it seems to me that there's a good deal of benefit, at practically zero risk, in building tooling that focuses on monitoring stuff which has already been signed, outside of the protected signing environment; and only subsequently after proving it there giving it the earlier, more powerful (and thus risky) role.
>

I believe the point was to check the prospective contents of the
TBSCertificate *before* CT logging (noting that Ryan Sleevi has been
violently insisting that failing to do that shall be punished as
harshly as actual misissuance) and *before* certificate signing.

Thus the checks would have to occur before signing, but it would still
be useful (architecturally) to run the checks without the ability to
change the request (other than to reject it with an error message).
Such separation will however have non-zero cost as the prospective
TBSCertificate or its description needs to be passed between additional
processes.

> Of course I know nothing of your specific circumstances, this is just a general observation about how I think I'd approach the question of improving issuance quality with minimal risk.
>

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Ryan Sleevi]

On Tue, Apr 18, 2017 at 1:32 PM, Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> I believe the point was to check the prospective contents of the
> TBSCertificate *before* CT logging (noting that Ryan Sleevi has been
> violently insisting that failing to do that shall be punished as
> harshly as actual misissuance) and *before* certificate signing.
>

While I appreciate the explicit callout as much as anyone, I think it's a
mischaracterization to state "violently". Have I suggested actual violence?

Whether you personally agree with it or not, I should note
https://wiki.mozilla.org/CA:Symantec_Issues#Issue_J:_SHA-1_Issuance_After_Deadline.2C_Again_.28February_2016.29

"(The CT RFC states that issuance of a pre-certificate is considered
equivalent to issuance of the certificate, and so Mozilla considers that
pre-certificate misissuance is misissuance.)"

[Nick Lamb]

On Tuesday, 18 April 2017 18:33:29 UTC+1, Jakob Bohm wrote:
> I believe the point was to check the prospective contents of the
> TBSCertificate *before* CT logging (noting that Ryan Sleevi has been
> violently insisting that failing to do that shall be punished as
> harshly as actual misissuance) and *before* certificate signing.

I come to this as always as someone focused on prevention of future harm. I can't speak for Ryan but I'm not interested in "punishing" anybody because retribution does not avoid future harm in itself. For example distrust of a CA is not a "punishment" of that CA, but a step taken to protect relying parties from certificates which shouldn't exist.

Detecting already bad situations still counts as prevention of future harm, this is because almost always the bad situation might get worse if undetected. This is why we fit smoke alarms - it would be bad if my flat was on fire, but it would be much worse if in the absence of an alarm it simply burned down with me inside it.

If some CA comes to m.d.s.policy twice a year with a problem where a certificate was issued that shouldn't have been, but they've cured it and altered their systems so that won't happen again - I can't say I'm ecstatic to see that, but at least they're paying attention.

In contrast if they're here twice a year because an independent researcher found a year-old certificate that shouldn't exist, and Gerv has to ask them for comment, then they investigate what went wrong and promise to cure it, I have to say I look on that much less kindly, and I suspect Ryan does too.

[Jakob Bohm]

I wrote this in the context of your previous post about why this
prevention code would have the ability to accidentally alter the
certificate before it was signed. My point was to explain, in
general, why such code is forced to run before signing and in a
context where preventing the checking code from altering the
certificate has a (small but) non-zero cost, unlike a check done
after issuance and/or after CT submission.

As for Ryan, I think his own response was quite illustrative.