Certificate issued with OU > 64

[in...@izenpe.com]

We have found that we've issued the following certificate with an OU > 64 characters.

https://crt.sh/?id=1202714390&opt=cablint

The certificate has been revoked some minutes later. We're retrieving all the information, we'll publish an incident report as soon as we have all the details.

Thanks
IZENPE

[in...@izenpe.com]

Hi, this is the incident report:

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.

We have controls to detect any misissuance before and after the issuance of the certificate. The certificate was issued at 11:52, detected in the following minute, and revoked at 12:07

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.

Feb 14th 11:52 -> the certificate was issued
Feb 14th 11:53 -> the misissuance was detected
Feb 14th 12:07 -> the certificate was revoked
Feb 14th 13:28 -> reported the incident to our PKI software manufacturer
Feb 14th 15:24 -> received the answer from the manufacturer. They tell us that there’s a bug in the preventive filter with the OU, and that they have a hotfix to solve it.
Feb 14th 17:21 -> Izenpe reports to mozilla.dev.security.policy list

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We’ll do a dual manual check until we have the hotfix correctly applied

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

There’s just one certificate affected

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.

https://crt.sh/?id=1202714390

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

It was a bug in the filter of the PKI software

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.

We hope to have the product hotfix applied by March 3rd

[Ryan Sleevi]

(Sending from the right e-mail)

Thanks for providing the additional details. However, I largely don't think
this meets the bar for the necessary level of detail - with the exception
of the timestamps and the remark your vendor had a bug, we've not really
learned anything about what went wrong or how to effectively prevent it.

Please work with your PKI software manufacturer, if necessary, to provide a
more substantive incident report - understanding when the bug was
introduced, what the bug is and how it functions, and how it's being
hotfixed, are all areas that are key to understanding how we can better
apply this knowledge across the CA ecosystem.

If you're not sure what an 'necessary' level of detail looks like, please
consider something like
https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ
,
listed as the "Good Practice" examples in the
https://wiki.mozilla.org/CA/Responding_To_An_Incident

Consider that it was stated that controls exist before and after issuance,
and yet it was still issued - this suggests the controls before issuance
are faulty.
Consider that we don't have details about what preventitive filters exist
or are configured, how they're supposed to work, how they failed to work,
and how the hotfix is meant to correct these issues.

I do want to acknowledge that it *is* good that y'all proactively detected
post-issuance and filed a report - this is absolutely what CAs should be
doing, and I'm glad to see Izenpe doing this. I similarly don't want to
discourage reporting as information comes in - I think that the details
provided are a good example of an 'interim' report in which the CA
continues to provide updates and transparency to the community while they
gather information. However, I don't think it represents a good 'final'
report - there's nothing of substance here other than "We found a problem
and fixed it". The goal of these is to understand the problem in
substantial technical detail, so that we can understand how the fixes will
address, and so that the community at large can be aware of systemic risks
or patterns and ensure that, regardless of what PKI software they use, so
that the ecosystem can itself improve.

Please continue to provide more details regarding this incident

[Wayne Thayer]

Thank you for the incident report. I have created a bug for tracking:
https://bugzilla.mozilla.org/show_bug.cgi?id=1528290

- Wayne

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Jakob Bohm]

Indeed, the report states that the bug was in the pre-issuance checking
software.

One good community lesson from this is that it is prudent to use a
different brand and implementation of the software that does post-
issuance checking than the software doing pre-issuance checking, as a
bug in the latter can be quickly detected by the former due to not
having the same set of bugs.

> I do want to acknowledge that it *is* good that y'all proactively detected
> post-issuance and filed a report - this is absolutely what CAs should be
> doing, and I'm glad to see Izenpe doing this. I similarly don't want to
> discourage reporting as information comes in - I think that the details
> provided are a good example of an 'interim' report in which the CA
> continues to provide updates and transparency to the community while they
> gather information. However, I don't think it represents a good 'final'
> report - there's nothing of substance here other than "We found a problem
> and fixed it". The goal of these is to understand the problem in
> substantial technical detail, so that we can understand how the fixes will
> address, and so that the community at large can be aware of systemic risks
> or patterns and ensure that, regardless of what PKI software they use, so
> that the ecosystem can itself improve.
>
> Please continue to provide more details regarding this incident
>

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Ryan Sleevi]

On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Indeed, the report states that the bug was in the pre-issuance checking
> software.
>

I believe you may have misread the report. I do not have the same
impression - what is stated is a failure of the control, not the linting.

> One good community lesson from this is that it is prudent to use a
> different brand and implementation of the software that does post-
> issuance checking than the software doing pre-issuance checking, as a
> bug in the latter can be quickly detected by the former due to not
> having the same set of bugs.

While I can understand the position here - and robustness is always good -
I do not believe this is wise or prudent advice, especially from this
incident. While it is true that post-issuance may catch things pre-issuance
misses, the value in post-issuance is both as a means of ensuring that
internal records are consistent (i.e. that the set of pre-issuance
certificates match what was issued - ensuring no rogue or unauthorized
certificates) as well as detecting if preissuance was not run.

There's significant more risk for writing a second implementation "just
because" - and far greater value in ensuring robustness. Certainly,
post-issuance is far less important than pre-issuance.

[Nick Lamb]

On Fri, 15 Feb 2019 05:05:16 -0800 (PST)

info--- via dev-security-policy <dev-secur...@lists.mozilla.org>
wrote:

> Feb 14th 13:28 -> reported the incident to our PKI software
> manufacturer
> Feb 14th 15:24 -> received the answer from the
> manufacturer. They tell us that there’s a bug in the preventive
> filter with the OU, and that they have a hotfix to solve it.
> Feb 14th 17:21 -> Izenpe reports to mozilla.dev.security.policy list

One value from incident reports is that other participants can learn
from what has happened rather than having to learn only from their own
experiences.

With that in mind, two thoughts

1. This incident report doesn't tell me whether the "PKI software
manufacturer" has other customers in the Web PKI. We should definitely
want not only Izenpe but any other participating CAs using the same
software to apply a fix for this issue or verify that they're
unaffected.

The most trivial way to achieve that would be for Izenpe to tell us the
name of this manufacturer and any other info (e.g. patched build
numbers, manufacturer's internal bug tracker codes) other CAs (which
are obliged to watch m.d.s.policy by Mozilla rules) should follow.

However it may be that this is too commercially sensitive, and if that
is the case Izenpe (and other CAs who find themselves in a similar
situation) should I think make sure the "PKI software manufacturer"
tells any other customers who may be affected, and add comments to this
effect in the Incident follow-up so that those following along know it
was taken care of, without it being made public which exact CAs are
using the same vendor's software.


2. Where third party software is essential to the Web PKI, I'd
encourage openness from this "PKI software manufacturer" and anybody
else in the like business. That could mean participating here (you
needn't mention specific customers if that's a problem) or it could mean
setting up some a continuing discussion elsewhere, especially if it's
going to inevitably drift far from Mozilla's focus.

There's plenty to discuss about security of these products without
straying into commercially sensitive issues like pricing or
non-security features, but it feels as though in reality a lot of the
time the makers don't talk to one another, which can mean the same
problem recurs in different software since lessons were not passed on,
the exact thing m.d.s.policy Incident reports are intended to prevent.


Nick.

[Jakob Bohm]

On 15/02/2019 19:33, Ryan Sleevi wrote:
> On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>> Indeed, the report states that the bug was in the pre-issuance checking
>> software.
>>
>
> I believe you may have misread the report. I do not have the same
> impression - what is stated is a failure of the control, not the linting.
>
>

The report refers to the pre-checking code in question as the "filter".

As the certificate hasn't been signed yet, pre-checking code needs to
either generate a dummy certificate (signed with a dummy untrusted key)
as input to a general cert-linter, or check the elements of the intended
certificate in a disembodied form.

>> One good community lesson from this is that it is prudent to use a
>> different brand and implementation of the software that does post-
>> issuance checking than the software doing pre-issuance checking, as a
>> bug in the latter can be quickly detected by the former due to not
>> having the same set of bugs.
>
>
> While I can understand the position here - and robustness is always good -
> I do not believe this is wise or prudent advice, especially from this
> incident. While it is true that post-issuance may catch things pre-issuance
> misses, the value in post-issuance is both as a means of ensuring that
> internal records are consistent (i.e. that the set of pre-issuance
> certificates match what was issued - ensuring no rogue or unauthorized
> certificates) as well as detecting if preissuance was not run.
>
> There's significant more risk for writing a second implementation "just
> because" - and far greater value in ensuring robustness. Certainly,
> post-issuance is far less important than pre-issuance.
>

Of cause. I was not suggesting that approach (although it can be a
high reliability technique). I was merely suggesting to obtain a
cert linter from a different vendor than the main CA suite. And by
all means run multiple checkers that purport to check the same
things.

Also note that pre-issuance checking is not the same as checking CT
pre-certs. Pre-issuance checks need to happen before signing the
pre-certs.

[Ryan Sleevi]

On Mon, Feb 18, 2019 at 2:49 PM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On 15/02/2019 19:33, Ryan Sleevi wrote:

> > On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy <
> > dev-secur...@lists.mozilla.org> wrote:
> >
> >> Indeed, the report states that the bug was in the pre-issuance checking
> >> software.
> >>
> >
> > I believe you may have misread the report. I do not have the same
> > impression - what is stated is a failure of the control, not the linting.
> >
> >
>

> The report refers to the pre-checking code in question as the "filter".
>
> As the certificate hasn't been signed yet, pre-checking code needs to
> either generate a dummy certificate (signed with a dummy untrusted key)
> as input to a general cert-linter, or check the elements of the intended
> certificate in a disembodied form.

Indeed it does. It’s common form to check in a disembodied form as fields
are input, typically from an RA endpoint. In fact, this is how every COTS
CA product works. This emphasizes why it’s important to not speculate, and
in particular, why it’s important and valuable to allow CAs to respond to
the questions themselves in a timely fashion. This allows for a more
meaningful understanding.

>
> >> One good community lesson from this is that it is prudent to use a
> >> different brand and implementation of the software that does post-
> >> issuance checking than the software doing pre-issuance checking, as a
> >> bug in the latter can be quickly detected by the former due to not
> >> having the same set of bugs.
> >
> >
> > While I can understand the position here - and robustness is always good
> -
> > I do not believe this is wise or prudent advice, especially from this
> > incident. While it is true that post-issuance may catch things
> pre-issuance
> > misses, the value in post-issuance is both as a means of ensuring that
> > internal records are consistent (i.e. that the set of pre-issuance
> > certificates match what was issued - ensuring no rogue or unauthorized
> > certificates) as well as detecting if preissuance was not run.
> >
> > There's significant more risk for writing a second implementation "just
> > because" - and far greater value in ensuring robustness. Certainly,
> > post-issuance is far less important than pre-issuance.
> >
>

> Of cause. I was not suggesting that approach (although it can be a
> high reliability technique). I was merely suggesting to obtain a
> cert linter from a different vendor than the main CA suite.

There’s no reason to believe this is an appropriate suggestion for this
incident, and certainly, not a necessary one. Such advice can easily lead
to confusion and problematic designs, which is why I feel it important to
point this out.

And by
> all means run multiple checkers that purport to check the same
> things.

While I realize there is a tendency to speak in the abstract here, I think
it’s both valuable and appropriate to highlight that there are no such
linters in the market, just as there is no “linter market” or “linter
vendors”. None of the open-source projects purport to cover the same set of
checks - each represents a different and complementary effort to examine
different elements of the issuance pipeline.

Also note that pre-issuance checking is not the same as checking CT
> pre-certs. Pre-issuance checks need to happen before signing the
> pre-certs.

I’m not sure if you were directly this reply at me or others. I don’t
believe there was any element to the replies or discussion to date that
would need this clarification, not have I seen any past or present incident
report that seems to have conflated these. It does seem an otherwise
unnecessary clarification, even if accurate.

>
>

[Wayne Thayer]

Izenpe posted the following response to the bug [1]:

My apologies for the delayed follow up response. First we must say that we
don't see any benefit for the community of publishing the name and version
of our PKI software, regardless of security issues.
As previously stated, we have two filters for each SSL certificate
issuance. The "pre" one is integrated into the PKI software and it's
obviously developed by the PKI manufacturer. The "post" one are really the
three filters used in crt.sh, that is cablint, x509lint and zlint.
Therefore we have different manufacturers for both filters. The previous
one is supposed to review the TBS with the same conditions as the post one.
In case of this misissued certificate it was immediately detected by the
post filters.
After contacting the manufacturer we knew that the hotfix was available
since last November. In this case we’ve already installed the hotfix in the
development environment, and it’ll be in the production environment before
next March 3rd. Meanwhile we’re reviewing manually with dual control all
requests we receive.
We have defined some improvement actions, which could help other CAs to
detect and fix these issues:

1. Apply cablint, x509lint and zlint also in the development
environment, as the post-issuance filter
2. Request our manufacturer to categorize all patches and hotfixes they
develop. At least there must be two categories: high if it applies to
security issues or RFC/CABForum misissuances, and the rest. In case of
patches classified as high, they must contact us immediately. We’re going
to update our policy to require to put those patches in production in 15
days as a maximum. We’ll also suggest our manufacturer to communicate it
also to all their customers.

All these actions will be applied in February.


[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1528290#c2

On Mon, Feb 18, 2019 at 4:58 PM Ryan Sleevi via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Mon, Feb 18, 2019 at 2:49 PM Jakob Bohm via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
>

[Wayne Thayer]

Ryan,

On Mon, Feb 18, 2019 at 4:58 PM Ryan Sleevi via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> On Mon, Feb 18, 2019 at 2:49 PM Jakob Bohm via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>
> > On 15/02/2019 19:33, Ryan Sleevi wrote:

> > > On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy <
> > > dev-secur...@lists.mozilla.org> wrote:
>

> And by
> > all means run multiple checkers that purport to check the same
> > things.
>
>
> While I realize there is a tendency to speak in the abstract here, I think
> it’s both valuable and appropriate to highlight that there are no such
> linters in the market, just as there is no “linter market” or “linter
> vendors”. None of the open-source projects purport to cover the same set of
> checks -

certlint, x509lint, and zlint all detect the problem with the Izenpe
certificate [1]. While I realize that none of these linters perform the
exact same set of checks, there is significant overlap that is in no way
abstract.

each represents a different and complementary effort to examine
> different elements of the issuance pipeline.
>

> If you are referring to certlint, x509lint, and zlint, can you explain
this statement?

[1] https://crt.sh/?id=1202714390&opt=cablint,x509lint,zlint

[Ryan Sleevi]

On Tue, Feb 19, 2019 at 9:56 PM Wayne Thayer <wth...@mozilla.com> wrote:

> Ryan,
>
> On Mon, Feb 18, 2019 at 4:58 PM Ryan Sleevi via dev-security-policy <
> dev-secur...@lists.mozilla.org> wrote:
>

>> On Mon, Feb 18, 2019 at 2:49 PM Jakob Bohm via dev-security-policy <
>> dev-secur...@lists.mozilla.org> wrote:
>>
>> > On 15/02/2019 19:33, Ryan Sleevi wrote:

>> > > On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy <
>> > > dev-secur...@lists.mozilla.org> wrote:
>>

>> And by
>> > all means run multiple checkers that purport to check the same
>> > things.
>>
>>
>> While I realize there is a tendency to speak in the abstract here, I think
>> it’s both valuable and appropriate to highlight that there are no such
>> linters in the market, just as there is no “linter market” or “linter
>> vendors”. None of the open-source projects purport to cover the same set
>> of
>> checks -
>
>
> certlint, x509lint, and zlint all detect the problem with the Izenpe
> certificate [1]. While I realize that none of these linters perform the
> exact same set of checks, there is significant overlap that is in no way
> abstract.
>
> each represents a different and complementary effort to examine
>> different elements of the issuance pipeline.
>>
>> If you are referring to certlint, x509lint, and zlint, can you explain
> this statement?
>

Sure! certlint’s strength is that it checks ASN.1 by virtue of asn1c, and
while it has a number of secondary checks for BR compliance, they’re not as
aggressively present as with zlint. Zlint is extremely well documented in
both its checks of 5280, but particularly excels in its BR compliance
aspects - especially with compliance dates.

X509lint is the less mature of the three linters, but more broadly targeted
5280 compliance without necessarily emphasizing the BR aspect.

There is indeed overlap between the three, but particularly zlint and
cablint excel in ways that the other does not. You absolutely would not
want one pre and one post - you will miss things between them.


> [1] https://crt.sh/?id=1202714390&opt=cablint,x509lint,zlint
>

[Wayne Thayer]

On Tue, Feb 19, 2019 at 10:51 AM Ryan Sleevi <ry...@sleevi.com> wrote:

>
>
> On Tue, Feb 19, 2019 at 9:56 PM Wayne Thayer <wth...@mozilla.com> wrote:
>
>> Ryan,
>>
>> On Mon, Feb 18, 2019 at 4:58 PM Ryan Sleevi via dev-security-policy <
>> dev-secur...@lists.mozilla.org> wrote:
>>

>>> On Mon, Feb 18, 2019 at 2:49 PM Jakob Bohm via dev-security-policy <
>>> dev-secur...@lists.mozilla.org> wrote:
>>>
>>> > On 15/02/2019 19:33, Ryan Sleevi wrote:

>>> > > On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy <
>>> > > dev-secur...@lists.mozilla.org> wrote:
>>>

> Interesting. In my experience, both certlint and zlint do a good job of
detecting BR violations.

X509lint is the less mature of the three linters, but more broadly targeted
> 5280 compliance without necessarily emphasizing the BR aspect.
>

> Agree on the 5280 focus of x509lint.

There is indeed overlap between the three, but particularly zlint and
> cablint excel in ways that the other does not. You absolutely would not
> want one pre and one post - you will miss things between them.
>

> I'm still not clear on the meaning of your statement "...examine
different elements of the issuance pipeline." It sounds like you're
recommending using multiple/all of these linters both pre- and
post-issuance, which makes sense and is what I understood Jakob to be
suggesting as well.

>
>> [1] https://crt.sh/?id=1202714390&opt=cablint,x509lint,zlint
>>
>

[Ryan Sleevi]

On Tue, Feb 19, 2019 at 11:05 PM Wayne Thayer <wth...@mozilla.com> wrote:

> On Tue, Feb 19, 2019 at 10:51 AM Ryan Sleevi <ry...@sleevi.com> wrote:
>
>>
>>
>> On Tue, Feb 19, 2019 at 9:56 PM Wayne Thayer <wth...@mozilla.com> wrote:
>>
>>> Ryan,
>>>
>>> On Mon, Feb 18, 2019 at 4:58 PM Ryan Sleevi via dev-security-policy <
>>> dev-secur...@lists.mozilla.org> wrote:
>>>

>>>> On Mon, Feb 18, 2019 at 2:49 PM Jakob Bohm via dev-security-policy <
>>>> dev-secur...@lists.mozilla.org> wrote:
>>>>
>>>> > On 15/02/2019 19:33, Ryan Sleevi wrote:

>>>> > > On Fri, Feb 15, 2019 at 12:01 PM Jakob Bohm via dev-security-policy
>>>> <
>>>> > > dev-secur...@lists.mozilla.org> wrote:
>>>>

I should have been clearer - ZLint is more timely in its adoption of
forward looking dates, and more comprehensive in how it evaluates historic
certs (based on the notBefore), due to its origins in studying historic
ecosystem misissuance. It also better documents the source of the
compliance requirement or interpretation. That’s not to say certlint
doesn’t catch those, but if you were a CA examining for, say, historic
misissuance as a post-issuance lint, Zlint is far more robust. They each
have strengths.

> X509lint is the less mature of the three linters, but more broadly
>> targeted 5280 compliance without necessarily emphasizing the BR aspect.
>>
>> Agree on the 5280 focus of x509lint.
>
> There is indeed overlap between the three, but particularly zlint and
>> cablint excel in ways that the other does not. You absolutely would not
>> want one pre and one post - you will miss things between them.
>>
>> I'm still not clear on the meaning of your statement "...examine
> different elements of the issuance pipeline." It sounds like you're
> recommending using multiple/all of these linters both pre- and
> post-issuance, which makes sense and is what I understood Jakob to be
> suggesting as well.
>

I don’t think Jakob’s original suggestion that elicited that reply -
namely, that “it is prudent to use a different brand and implementation of
the software that does post-issuance checking than the software doing
pre-issuance checking” seems to be saying the same thing as what you or I
are saying. It is not that you want a difference than pre- and post-, but
you want to make sure you’re comprehensive. You can be comprehensive with a
single linter (in the abstract), although none of the linters are at that
point.

It also conflates software controls on the disembodied fields - which are
not to be confused with pre-issuance linting, but instead is about the CA’s
implementation and design of controls themselves. Put differently, linting,
whether pre-issuance (i.e. a tbsCertificate fully formed, as the majority
of CAs have done) or post-issuance (with a signed certificate), is itself a
check on the CAs controls and design, not a substitute or equivalent for.

>