Certificates with less than 64 bits of entropy

[Jonathan Rudenberg]

Baseline Requirements section 7.1 says:

> Effective September 30, 2016, CAs SHALL generate non‐sequential Certificate serial numbers greater than zero (0) containing at least 64 bits of output from a CSPRNG.

There are 1027 unexpired unrevoked certificates known to CT with a notBefore date greater than or equal to 2016-09-30 that are trusted by NSS for server authentication and have a serial number that has less than 64 bits of entropy.

The full list can be found here: https://misissued.com/batch/6/

Some of these were brought up in a previous thread[0], but I though a comprehensive picture of this issue would be helpful.

I’ve included a breakdown at the end of this email, and here are a few things that stood out to me while researching this:

- The "Cihaz Sertifikası Hizmet Sağlayıcı - Sürüm 4” intermediate appears to use randomly generated 48-bit numbers.
- Three intermediates, "TeleSec ServerPass Class 2 CA”, "Go Daddy Secure Certificate Authority - G2”, and "Starfield Secure Certificate Authority - G2”, (which are not in this list) appear to issue certificates with serial numbers that are based on exactly 64 bits of entropy. This means that a small percentage of the certificates that they issue have serial numbers that are smaller than 8 bytes, requiring additional filtering to avoid false positives. It would be helpful if the policy was adjusted to require serial numbers always be at least 8 bytes before DER encoding to avoid these false positives.

Jonathan

[0] https://groups.google.com/d/topic/mozilla.dev.security.policy/vl5eq0PoJxY/discussion

—

QuoVadis (560)
Siemens Issuing CA Internet Server 2016 (560)

D-TRUST (224)
D-TRUST SSL Class 3 CA 1 2009 (178)
D-TRUST SSL Class 3 CA 1 EV 2009 (45)
D-TRUST Root Class 3 CA 2 EV 2009 (1)

DigiCert (85)
Siemens Issuing CA Class Internet Server 2013 (82)
InfoCert Web Certification Authority (3)

Izenpe S.A. (62)
EAEko Herri Administrazioen CA - CA AAPP Vascas (2) (62)

Government of The Netherlands, PKIoverheid (Logius) (55)
Digidentity Services CA - G2 (55)

Government of Turkey, Kamu Sertifikasyon Merkezi (Kamu SM) (38)
Cihaz Sertifikası Hizmet Sağlayıcı - Sürüm 4 (38)

[Jonathan Rudenberg]

> On Aug 10, 2017, at 11:20, Jonathan Rudenberg via dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
> QuoVadis (560)
> Siemens Issuing CA Internet Server 2016 (560)
>
> D-TRUST (224)
> D-TRUST SSL Class 3 CA 1 2009 (178)
> D-TRUST SSL Class 3 CA 1 EV 2009 (45)
> D-TRUST Root Class 3 CA 2 EV 2009 (1)
>
> DigiCert (85)
> Siemens Issuing CA Class Internet Server 2013 (82)
> InfoCert Web Certification Authority (3)
>
> Izenpe S.A. (62)
> EAEko Herri Administrazioen CA - CA AAPP Vascas (2) (62)
>
> Government of The Netherlands, PKIoverheid (Logius) (55)
> Digidentity Services CA - G2 (55)
>
> Government of Turkey, Kamu Sertifikasyon Merkezi (Kamu SM) (38)
> Cihaz Sertifikası Hizmet Sağlayıcı - Sürüm 4 (38)

It looks like my summary missed one QuoVadis intermediate:

Bayerische SSL-CA-2016-01 (3)

[Nick Lamb]

On Thursday, 10 August 2017 16:20:56 UTC+1, Jonathan Rudenberg wrote:
- Three intermediates, "TeleSec ServerPass Class 2 CA”, "Go Daddy Secure Certificate Authority - G2”, and "Starfield Secure Certificate Authority - G2”, (which are not in this list) appear to issue certificates with serial numbers that are based on exactly 64 bits of entropy. This means that a small percentage of the certificates that they issue have serial numbers that are smaller than 8 bytes, requiring additional filtering to avoid false positives. It would be helpful if the policy was adjusted to require serial numbers always be at least 8 bytes before DER encoding to avoid these false positives.

Mmmm. I previously spoke out in favour of the practice of calling out non-compliant certificates because we need CAs to be doing their best, but I think there's also an allied element that when we're looking for problems we too need to put the effort in.

The truth is that there is no positive test for randomness, any work in this area is going to end up needing a judgement call, so I think inconveniencing the CAs even a small amount with such a policy change just to make automated testing easier isn't the right trade off. If there happens to be some future work in this policy area and the opportunity is taken to incorporate Jonathan's wording I have no problem with that, but I definitely don't think Mozilla should insist on it for its own sake.

[Jeremy Rowley]

Hi Jonathan,

InfoCert's sub CA was revoked on August 1, 2017. We'll reach out to Siemens. They moved to Quovadis a while ago and are no longer issuing from that Sub CA.

Jeremy

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Matthew Hardeman]

On Thursday, August 10, 2017 at 11:27:53 AM UTC-5, Nick Lamb wrote:

> The truth is that there is no positive test for randomness, any work in this area is going to end up needing a judgement call, so I think inconveniencing the CAs even a small amount with such a policy change just to make automated testing easier isn't the right trade off. If there happens to be some future work in this policy area and the opportunity is taken to incorporate Jonathan's wording I have no problem with that, but I definitely don't think Mozilla should insist on it for its own sake.

Further to the point that Nick made, merely ensuring that the serial number field is represented as at least eight bytes prior to DER encoding still does not tell you whether or not the CA truly incorporated 64 bits of randomness in the serial number versus, for example, packing together 32 bits of randomness and 32 bits of sequence number.

[Ben Wilson]

With regard to Siemens, given the large number of certificates and the disruption that massive revocations will have on their infrastructure, what does this community expect them to do?

[Alex Gaynor]

Have they fixed whatever issue there is with their PKI infrastructure that
leads to this issue? From skimming, I see this pool contains certs issued
as recently as one month ago.

Alex

[Ben Wilson]

Apparently they haven’t yet, but we’ll assume that they will.

Does the community expect a remediation plan for their code and then a revocation-and-replacement plan?



Ben Wilson, JD, CISA, CISSP

VP Compliance

+1 801 701 9678

From: Alex Gaynor [mailto:aga...@mozilla.com]
Sent: Friday, August 11, 2017 8:31 AM
To: Ben Wilson <ben.w...@digicert.com>
Cc: Jeremy Rowley <jeremy...@digicert.com>; Jonathan Rudenberg <jona...@titanous.com>; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Certificates with less than 64 bits of entropy

Have they fixed whatever issue there is with their PKI infrastructure that leads to this issue? From skimming, I see this pool contains certs issued as recently as one month ago.



Alex

On Fri, Aug 11, 2017 at 10:26 AM, Ben Wilson via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > wrote:

With regard to Siemens, given the large number of certificates and the disruption that massive revocations will have on their infrastructure, what does this community expect them to do?


-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+ben <mailto:dev-security-policy-bounces%2Bben> =digice...@lists.mozilla.org <mailto:digice...@lists.mozilla.org> ] On Behalf Of Jeremy Rowley via dev-security-policy
Sent: Thursday, August 10, 2017 12:01 PM
To: Jonathan Rudenberg <jona...@titanous.com <mailto:jona...@titanous.com> >; mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org>
Subject: RE: Certificates with less than 64 bits of entropy

Hi Jonathan,

InfoCert's sub CA was revoked on August 1, 2017. We'll reach out to Siemens. They moved to Quovadis a while ago and are no longer issuing from that Sub CA.

Jeremy

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+jeremy.rowley <mailto:dev-security-policy-bounces%2Bjeremy.rowley> =digice...@lists.mozilla.org <mailto:digice...@lists.mozilla.org> ] On Behalf Of Jonathan Rudenberg via dev-security-policy
Sent: Thursday, August 10, 2017 9:26 AM
To: mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Certificates with less than 64 bits of entropy

> On Aug 10, 2017, at 11:20, Jonathan Rudenberg via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > wrote:
>
> QuoVadis (560)
> Siemens Issuing CA Internet Server 2016 (560)
>
> D-TRUST (224)
> D-TRUST SSL Class 3 CA 1 2009 (178)
> D-TRUST SSL Class 3 CA 1 EV 2009 (45)
> D-TRUST Root Class 3 CA 2 EV 2009 (1)
>
> DigiCert (85)
> Siemens Issuing CA Class Internet Server 2013 (82)
> InfoCert Web Certification Authority (3)
>
> Izenpe S.A. (62)
> EAEko Herri Administrazioen CA - CA AAPP Vascas (2) (62)
>
> Government of The Netherlands, PKIoverheid (Logius) (55)
> Digidentity Services CA - G2 (55)
>
> Government of Turkey, Kamu Sertifikasyon Merkezi (Kamu SM) (38)
> Cihaz Sertifikası Hizmet Sağlayıcı - Sürüm 4 (38)

It looks like my summary missed one QuoVadis intermediate:

Bayerische SSL-CA-2016-01 (3)

_______________________________________________
dev-security-policy mailing list

dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org>

https://lists.mozilla.org/listinfo/dev-security-policy


_______________________________________________
dev-security-policy mailing list

dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy

[Jeremy Rowley]

They are no longer issuing from the digicert cross. The issue is within their PKI but there should be no additional certificates chained to DigiCert roots

On Aug 11, 2017, at 8:33 AM, Ben Wilson <ben.w...@digicert.com<mailto:ben.w...@digicert.com>> wrote:

Apparently they haven’t yet, but we’ll assume that they will.
Does the community expect a remediation plan for their code and then a revocation-and-replacement plan?

Ben Wilson, JD, CISA, CISSP
VP Compliance
+1 801 701 9678

<image001.jpg>

[Ben Wilson]

QuoVadis Enterprise Trust CA 2 G3 signed the Siemens Issuing CA Internet Server 2016.

[David E. Ross]

On 8/11/2017 7:26 AM, Ben Wilson wrote:
>
> With regard to Siemens, given the large number of certificates and
> the disruption that massive revocations will have on their
> infrastructure, what does this community expect them to do?
>

Each violation of published requirements for the operation of a
certification authority -- no matter how minor -- brings into question
whether there are more serious violations. In this case, Siemens has
two choices. They can revoke and replace all affected certificates, or
else they can suffer the loss of trust. No certification authority
should ever be considered to big to fail.

[Eric Mill]

If they're not going to revoke within 24 hours and willingly violate that
part of the policy, I would at least expect them to, within that 24 hours,
produce a description of why this happened, what they're doing to fix it,
and when they expect the certificates to be replaced (along with an
expectation of when a hard revocation deadline would be regardless of
customer responsiveness). Once the underlying issue is fixed, I would
expect them to ring in to say that it's fixed and what they did to fix it.

That's just basic good-faith engagement that demonstrates that the issuing
CA at least takes the issue as seriously as the community does, and
engenders trust that the issue is being addressed.

Let's Encrypt just responded this week to an encoding compliance failure
with a live production code fix (including code review and sign off) within
6 hours of being notified.

While not every issuing CA may take security seriously enough to employ
engineers on staff who can research, author and deploy a production code
fix in a 24 hour period, every issuing CA should be able to muster the
strength to keep the community informed of their plans and progress in
however long it takes to address the issue.

-- Eric

On Fri, Aug 11, 2017 at 10:33 AM, Ben Wilson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Apparently they haven’t yet, but we’ll assume that they will.
>
> Does the community expect a remediation plan for their code and then a
> revocation-and-replacement plan?
>
>
>
> Ben Wilson, JD, CISA, CISSP
>
> VP Compliance
>
> +1 801 701 9678
>
>
>
>
>

> From: Alex Gaynor [mailto:aga...@mozilla.com]
> Sent: Friday, August 11, 2017 8:31 AM
> To: Ben Wilson <ben.w...@digicert.com>
> Cc: Jeremy Rowley <jeremy...@digicert.com>; Jonathan Rudenberg <
> jona...@titanous.com>; mozilla-dev-s...@lists.mozilla.org
> Subject: Re: Certificates with less than 64 bits of entropy
>
>
>

> Have they fixed whatever issue there is with their PKI infrastructure that
> leads to this issue? From skimming, I see this pool contains certs issued
> as recently as one month ago.
>
>
>
> Alex
>
>
>
> On Fri, Aug 11, 2017 at 10:26 AM, Ben Wilson via dev-security-policy <

> dev-secur...@lists.mozilla.org <mailto:dev-security-policy@li

> sts.mozilla.org> > wrote:
>
> With regard to Siemens, given the large number of certificates and the
> disruption that massive revocations will have on their infrastructure, what
> does this community expect them to do?
>
>

> dev-secur...@lists.mozilla.org <mailto:dev-security-policy@li

> sts.mozilla.org> > wrote:
> >
> > QuoVadis (560)
> > Siemens Issuing CA Internet Server 2016 (560)
> >
> > D-TRUST (224)
> > D-TRUST SSL Class 3 CA 1 2009 (178)
> > D-TRUST SSL Class 3 CA 1 EV 2009 (45)
> > D-TRUST Root Class 3 CA 2 EV 2009 (1)
> >
> > DigiCert (85)
> > Siemens Issuing CA Class Internet Server 2013 (82)
> > InfoCert Web Certification Authority (3)
> >
> > Izenpe S.A. (62)
> > EAEko Herri Administrazioen CA - CA AAPP Vascas (2) (62)
> >
> > Government of The Netherlands, PKIoverheid (Logius) (55)
> > Digidentity Services CA - G2 (55)
> >
> > Government of Turkey, Kamu Sertifikasyon Merkezi (Kamu SM) (38)
> > Cihaz Sertifikası Hizmet Sağlayıcı - Sürüm 4 (38)
>
> It looks like my summary missed one QuoVadis intermediate:
>
> Bayerische SSL-CA-2016-01 (3)
>
> _______________________________________________
> dev-security-policy mailing list

> dev-secur...@lists.mozilla.org <mailto:dev-security-policy@li

> sts.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-security-policy
>
>
> _______________________________________________
> dev-security-policy mailing list

> dev-secur...@lists.mozilla.org <mailto:dev-security-policy@li

> sts.mozilla.org>
> https://lists.mozilla.org/listinfo/dev-security-policy
>
>
>
>
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org

> https://lists.mozilla.org/listinfo/dev-security-policy
>
>


--
konklone.com | @konklone <https://twitter.com/konklone>

[Ben Wilson]

They are working on the issue and preparing a report.

From: Eric Mill [mailto:er...@konklone.com]
Sent: Saturday, August 12, 2017 9:03 PM
To: Ben Wilson <ben.w...@digicert.com>
Cc: Alex Gaynor <aga...@mozilla.com>; Jonathan Rudenberg <jona...@titanous.com>; mozilla-dev-s...@lists.mozilla.org; Jeremy Rowley <jeremy...@digicert.com>
Subject: Re: Certificates with less than 64 bits of entropy

If they're not going to revoke within 24 hours and willingly violate that part of the policy, I would at least expect them to, within that 24 hours, produce a description of why this happened, what they're doing to fix it, and when they expect the certificates to be replaced (along with an expectation of when a hard revocation deadline would be regardless of customer responsiveness). Once the underlying issue is fixed, I would expect them to ring in to say that it's fixed and what they did to fix it.



That's just basic good-faith engagement that demonstrates that the issuing CA at least takes the issue as seriously as the community does, and engenders trust that the issue is being addressed.



Let's Encrypt just responded this week to an encoding compliance failure with a live production code fix (including code review and sign off) within 6 hours of being notified.



While not every issuing CA may take security seriously enough to employ engineers on staff who can research, author and deploy a production code fix in a 24 hour period, every issuing CA should be able to muster the strength to keep the community informed of their plans and progress in however long it takes to address the issue.



-- Eric

On Fri, Aug 11, 2017 at 10:33 AM, Ben Wilson via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > wrote:

Apparently they haven’t yet, but we’ll assume that they will.

Does the community expect a remediation plan for their code and then a revocation-and-replacement plan?



Ben Wilson, JD, CISA, CISSP

VP Compliance

+1 801 701 9678 <tel:%2B1%20801%20701%209678>

From: Alex Gaynor [mailto:aga...@mozilla.com <mailto:aga...@mozilla.com> ]
Sent: Friday, August 11, 2017 8:31 AM
To: Ben Wilson <ben.w...@digicert.com <mailto:ben.w...@digicert.com> >
Cc: Jeremy Rowley <jeremy...@digicert.com <mailto:jeremy...@digicert.com> >; Jonathan Rudenberg <jona...@titanous.com <mailto:jona...@titanous.com> >; mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Certificates with less than 64 bits of entropy



Have they fixed whatever issue there is with their PKI infrastructure that leads to this issue? From skimming, I see this pool contains certs issued as recently as one month ago.



Alex

On Fri, Aug 11, 2017 at 10:26 AM, Ben Wilson via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> <mailto:dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > > wrote:

With regard to Siemens, given the large number of certificates and the disruption that massive revocations will have on their infrastructure, what does this community expect them to do?


-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+ben <mailto:dev-security-policy-bounces%2Bben> <mailto:dev-security-policy-bounces%2Bben <mailto:dev-security-policy-bounces%252Bben> > =digice...@lists.mozilla.org <mailto:digice...@lists.mozilla.org> <mailto:digice...@lists.mozilla.org <mailto:digice...@lists.mozilla.org> > ] On Behalf Of Jeremy Rowley via dev-security-policy
Sent: Thursday, August 10, 2017 12:01 PM
To: Jonathan Rudenberg <jona...@titanous.com <mailto:jona...@titanous.com> <mailto:jona...@titanous.com <mailto:jona...@titanous.com> > >; mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org> <mailto:mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org> >
Subject: RE: Certificates with less than 64 bits of entropy

Hi Jonathan,

InfoCert's sub CA was revoked on August 1, 2017. We'll reach out to Siemens. They moved to Quovadis a while ago and are no longer issuing from that Sub CA.

Jeremy

-----Original Message-----
From: dev-security-policy [mailto:dev-security-policy-bounces+jeremy.rowley <mailto:dev-security-policy-bounces%2Bjeremy.rowley> <mailto:dev-security-policy-bounces%2Bjeremy.rowley <mailto:dev-security-policy-bounces%252Bjeremy.rowley> > =digice...@lists.mozilla.org <mailto:digice...@lists.mozilla.org> <mailto:digice...@lists.mozilla.org <mailto:digice...@lists.mozilla.org> > ] On Behalf Of Jonathan Rudenberg via dev-security-policy
Sent: Thursday, August 10, 2017 9:26 AM
To: mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org> <mailto:mozilla-dev-s...@lists.mozilla.org <mailto:mozilla-dev-s...@lists.mozilla.org> >
Subject: Re: Certificates with less than 64 bits of entropy

> On Aug 10, 2017, at 11:20, Jonathan Rudenberg via dev-security-policy <dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> <mailto:dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> > > wrote:
>
> QuoVadis (560)
> Siemens Issuing CA Internet Server 2016 (560)
>
> D-TRUST (224)
> D-TRUST SSL Class 3 CA 1 2009 (178)
> D-TRUST SSL Class 3 CA 1 EV 2009 (45)
> D-TRUST Root Class 3 CA 2 EV 2009 (1)
>
> DigiCert (85)
> Siemens Issuing CA Class Internet Server 2013 (82)
> InfoCert Web Certification Authority (3)
>
> Izenpe S.A. (62)
> EAEko Herri Administrazioen CA - CA AAPP Vascas (2) (62)
>
> Government of The Netherlands, PKIoverheid (Logius) (55)
> Digidentity Services CA - G2 (55)
>
> Government of Turkey, Kamu Sertifikasyon Merkezi (Kamu SM) (38)
> Cihaz Sertifikası Hizmet Sağlayıcı - Sürüm 4 (38)

It looks like my summary missed one QuoVadis intermediate:

Bayerische SSL-CA-2016-01 (3)

_______________________________________________
dev-security-policy mailing list

dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> <mailto:dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> >

https://lists.mozilla.org/listinfo/dev-security-policy


_______________________________________________
dev-security-policy mailing list

dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> <mailto:dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org> >

https://lists.mozilla.org/listinfo/dev-security-policy




_______________________________________________
dev-security-policy mailing list

dev-secur...@lists.mozilla.org <mailto:dev-secur...@lists.mozilla.org>
https://lists.mozilla.org/listinfo/dev-security-policy







--

konklone.com <https://konklone.com> | @konklone <https://twitter.com/konklone>

[Nick Lamb]

On Sunday, 13 August 2017 04:04:45 UTC+1, Eric Mill wrote:
> While not every issuing CA may take security seriously enough to employ
> engineers on staff who can research, author and deploy a production code
> fix in a 24 hour period, every issuing CA should be able to muster the
> strength to keep the community informed of their plans and progress in
> however long it takes to address the issue.

In my opinion the correct incentive structure here is: We don't care whether you ever start issuing again but if you have a limited time to stop the problem, if you can't fix it quickly that will be by ceasing issuance.

Switching off the issuance pipeline in a timely fashion when a problem is uncovered (so that things stop getting worse) needs to be something every CA can do. It should always be within the skill set of personnel available "on call" when things go wrong. But whether they have engineers able to actually fix a problem the same day, the next day or a month later is an operational detail for the CA leadership. For commercial CAs there is presumably some trade-off between the need to be seen as a reliable supplier for repeat subscribers and the cost of having on-call engineers. But it needn't concern m.d.s.policy where they think best to draw the line, so long as they prevent the problem recurring by switching off an affected issuance pipeline until it's fixed.

I am minded to draw comparison to "emergency plumber" services. Despite it being an "emergency" the plumber will be no more quickly able to source parts from a discontinued product line, or plan and install complex new systems than a non-emergency plumber. Those things may still take weeks. But what they can always do immediately is switch off supply of water or gas so as to stop things getting worse.

[Ben Wilson]

As previously noted on this list, there are two Siemens CAs that have issued
certificates with less than 64 bits of entropy. See
https://misissued.com/batch/6/
The Siemens Issuing CA Internet 2013 is subordinate to a DigiCert-owned
root, and the Siemens Issuing CA Internet 2016 is signed by Quo Vadis.

This email is meant to only address and respond as to certificates issued by
the Siemens Issuing CA Internet 2013, which ceased issuing certificates in
2016, although it also contains information regarding the Siemens Issuing CA
Internet 2016. We suspect that further response regarding the 2016 CA will
be forthcoming from either QuoVadis or Siemens.

Siemens reported the following to us earlier today:

---------

We have discovered an implementation error in our in-house CA software which
meant it was using 32bit serials, although they were non sequential.
Siemens Issuing CA Internet 2013 was then already offline, because we
started on 2nd November 2016 the issuing with the Siemens Issuing CA
Internet 2016 which is cross-signed by QuoVadis.
Furthermore, the Siemens Issuing CA Internet 2016 was taken offline
immediately upon report to us and this was reported by Stephen Davidson from
QuoVadis to the listserv on 7/20.
The Issuing CA Internet 2016 remains offline and we issue now certificates
from the external market. We also informed our external Auditor about that
issue immediately.

For the future, all problems are fixed. Originally it was planned to start
on 4th of October 2017 with a new CA System (EJBCA) with full CT Logging.
When we noticed the serial number issue, we doubled up our resources and
moved the Go Live roughly one month earlier now to 8th of September 2017.

In the past, we issued 137 certificates from the Siemens Issuing CA 2013 and
the validity period of the certificates ends:

1 in September 2017
120 in October 2017
15 in November 2017

Except these, there are three certificates with a longer validity period:

CN=circuit.siemens.com,L=Muenchen,SP=Bayern,C=DE,O=Siemens-Internet,OU=GS IT
IN SPLM SDC US 2018-02-20 13:32:43.000

We are discussing with the customer to exchange the certificate, here some
technical points must be clarified.

The other two certificates are infrastructure certificates and cannot be
replaced due the need of the function of the OCSP Responder and the CA Test
site which is an ETSI Requirement

CN=2013-internet-valid.catestsite.siemens.com,L=Muenchen,SP=Bayern,C=DE,O=Si
emens-Internet,OU=GS IT HR 74,SN=ETSI0002 2018-02-20 14:01:05.000
CN=Siemens PKI OCSP Signer ZZZZZZY9,O=Siemens,C=DE 2018-04-10
10:24:31.000

The replacement is complicated as, although the PKI is centralized, the
certificates are issued to Siemens group companies around the world.
We´re working on a replacement plan to do this in a proper time. As the
certificates are already reaching their end of life, for most of them the
renewal process is already ongoing (60 days before expirations the
certificate holders are informed to renew).

--------

Obviously we will continue to evaluate DigiCert's response to this
information from Siemens, but we figured that interim disclosure of this
information to this list was important.

Sincerely yours,

Ben Wilson

-----Original Message-----
From: dev-security-policy
[mailto:dev-security-policy-bounces+ben=digice...@lists.mozilla.org] On
Behalf Of Nick Lamb via dev-security-policy
Sent: Sunday, August 13, 2017 2:07 AM
To: mozilla-dev-s...@lists.mozilla.org
Subject: Re: Certificates with less than 64 bits of entropy

On Sunday, 13 August 2017 04:04:45 UTC+1, Eric Mill wrote:

> While not every issuing CA may take security seriously enough to
> employ engineers on staff who can research, author and deploy a
> production code fix in a 24 hour period, every issuing CA should be
> able to muster the strength to keep the community informed of their
> plans and progress in however long it takes to address the issue.

In my opinion the correct incentive structure here is: We don't care whether
you ever start issuing again but if you have a limited time to stop the
problem, if you can't fix it quickly that will be by ceasing issuance.

Switching off the issuance pipeline in a timely fashion when a problem is
uncovered (so that things stop getting worse) needs to be something every CA
can do. It should always be within the skill set of personnel available "on
call" when things go wrong. But whether they have engineers able to actually
fix a problem the same day, the next day or a month later is an operational
detail for the CA leadership. For commercial CAs there is presumably some
trade-off between the need to be seen as a reliable supplier for repeat
subscribers and the cost of having on-call engineers. But it needn't concern
m.d.s.policy where they think best to draw the line, so long as they prevent
the problem recurring by switching off an affected issuance pipeline until
it's fixed.

I am minded to draw comparison to "emergency plumber" services. Despite it
being an "emergency" the plumber will be no more quickly able to source
parts from a discontinued product line, or plan and install complex new
systems than a non-emergency plumber. Those things may still take weeks. But
what they can always do immediately is switch off supply of water or gas so
as to stop things getting worse.

[Stephen Davidson]

Update on Siemens - Certificates with less than 64 bits of entropy
The following is regarding the topic https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/vl5eq0PoJxY regarding the “Siemens Issuing CA Internet Server 2016” that is root signed by QuoVadis and independently audited and disclosed.

At the time the issue was reported, Siemens agreed to immediately take the CA offline, and it remains offline pending resolution. This was reported to the listserv by me on 7/20.

Siemens confirmed a bug in their internally-developed CA software which meant it was issuing TLS/SSL with 32bit serial numbers, although the serial numbers were non sequential. Siemens informed their external auditors of the situation.

It was found that 1201 currently valid certificates chained to the QuoVadis root were affected. An additional 137 currently valid certificates were issued under the previous "Siemens Issuing CA Internet 2013" chained to a Digicert root, noted in an email from Ben Wilson of Digicert yesterday. In the case of the QuoVadis-chained certificates, the certificates are virtually all of one year validity with expirations balanced across the calendar months (there are a handful of two and three year certificates, similar to the Digicert-chained population). The remaining Digicert-chained certificates all expire by end of November 2017. All certificates were issued to Siemens entities and Siemens-controlled domains.

Next steps
Siemens has moved to accelerate the previously planned replacement of their existing inhouse CA platform with a well-known open source CA with which QuoVadis is well familiar. QuoVadis and Siemens' auditors are coordinating with Siemens to confirm that the new CA configuration meets Baseline Requirements. It is worth noting that some BR controls, particularly related to vetting, are imposed by the Siemens certificate lifecycle system which will continue to be used with the new CA. Siemens will not recommence their inhouse SSL issuance until the new CA is active and confirmed compliant. The new CA is expected to come online in the second week of September. Siemens commits to logging new SSL from that CA in Certificate Transparency.

Replacement
Although the Siemens PKI is centralised, the certificates are issued to a wide variety of Siemens group companies around the world and are used on both infrastructure and high traffic websites. A rushed revocation and replacement of these certificates would have a negative business impact on Siemens that they believe outweighs the risk of the lower serials entropy (particularly given that they are nonsequential).

We propose that Siemens begin the early replacement of the affected certificates as soon as the new CA infrastructure is approved, with the goal of completing the task by January 31, 2018. This will include all the affected certificates (ie those chained from both the QuoVadis and Digicert roots). While Siemens acknowledges that the affected certificates should not have occurred, we point out that they will all be replaced far in advance of the September 2019 date when industry-wide the last certificates issued before the BR change (to larger serial numbers) are scheduled to expire.

We request that Siemens be allowed this expanded scope to conduct an orderly replacement of the affected certificates.

Many thanks, Stephen Davidson
QuoVadis

[Ryan Sleevi]

Stephen,

Thanks for posting your update regarding Siemens. Unfortunately, however,
it's lacking in many critical details necessary to take appropriate next
steps.

On the positive side, it is good to see that QuoVadis immediately took (and
kept) the Siemens CA offline. This represents a minimum necessary step when
faced with misissuance from a subordinate, and is a step expected of all
CAs while they investigate the issue and its causes, to prevent future
misissuance.

However, the assessment of what went wrong, what steps are being taken, and
the risk are all deficient and, at worse, potentially demonstrate a
misunderstanding of both the risk of these certificates and the purposes of
these discussions.

To understand an appropriate level of detail, and the set of questions that
both QuoVadis and Siemens should be asking, I think a postmortem to the
level of detail provided by PKIoverheid, in
https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ
, is a _minimum_ necessary step to take. In particular, it's useful to
understand

1) Siemens has maintained it was a "bug" that caused 32-bit serial numbers.
However, it's unclear from the community whether or not Siemens actually
took steps to appropriately implement the necessary controls - meaning it
was a bug in process - or whether code was implemented, but the
implementation was faulty - a bug in software. Including sufficient detail,
by analyzing both the process and the software development methodology, is
necessary to help the community both understand the nature of the "bug" and
the processes that gave rise to it.

2) QuoVadis failed to detect these certificates from a sub-CA it is
supervising. Why is this, and what steps is QuoVadis taking to properly
oversee its independently-operated sub-CAs?

3) Given that this issue persisted in both the previous and present
hierarchy, and given Siemens' limited issuance, why was this not detected
by audits? Were the audits by a sufficiently technically skilled auditor?
Did every certificate they sampled have an appropriately random serial
number? How many certificates did they sample?

4) QuoVadis and Siemens have not disclosed all affected certificates. Can
you please provide full details for all 1201 currently valid certificates,
such as via CT? A separate document (e.g. spreadsheet) would be appropriate
to contain all of the links to this.

In short, the analysis by Siemens and QuoVadis fails to identify or address
the systemic factors that lead to this failure, and as a consequence,
restore little confidence in either Siemens' operation or QuoVadis'.
Further, the analysis of the risk of serial numbers is either misguided or
disingenuous. Serial number entropy represents a critical risk mitigation
regarding hash collisions - its criticality is about ensuring it is in
place at time of issuance, as once a certificate is issued, the harm has
been introduced. Thus, it's both inappropriate and unrelated to suggest
that other legacy certificates with inappropriate entropy exist - those
were issued in a earlier time - and what is relevant is when the process
was introduced, how long it happened, and when it stopped.

It's worth noting that Siemens made a design and business decision to use
an in-house CA, and apparently made a series of design and business
decisions that failed to oversee or detect this issue. Similarly, QuoVadis,
despite having tools available to monitor and supervise issuance, appears
to have failed to detect these issues. The impact that revoking these
certificate has can thus be argued to be through the decisions of Siemens
and QuoVadis, rather than the community, and thus appropriate to suggest
that the cost of this impact should be borne by Siemens and QuoVadis.
Further, it's worth noting that, given the future risk of further
non-compliance, it's not unreasonable to suggest that Siemens should
absolutely be prepared to replace all 1201 currently valid certificates
within 24 hours, or, at most, a week, rather than the nearly five months
being proposed here. In 2017, it is no longer defensible to suggest an
organization may need that long to replace its certificates - and if it is,
that represents a series of design and business decisions made over the
past several years that have ignored both modern deployment practices and
the changing nature of the ecosystem. It should not be the role of CAs, or
the Mozilla community, to prop up these parts of the ecosystem, which need
to support evolving to address the continued threats and necessary
ecosystem improvements.

As the Baseline Requirements require revocation of the subscriber
certificates within 24 hours (4.9.1.1 (4), (9), (15)), which Siemens has
not done, and the Baseline Requirements require revocation of the
subordinate CA certificate within 7 days (4.9.1.2 (5)), which QuoVadis has
not done, it's expected that both Siemens and QuoVadis will present
qualified audits during the next audit period due to knowing and
intentional violation of the requirements. The effect these qualifications
have on the continued trust status of past certificates, and the continued
recognition of future certificates, is predicated on the ability to provide
sufficient assurance that the systemic issues have been identified,
understood, and remediated. At present, this response does not indicate a
sufficient level of disclosure to make that conclusion. Timely and
immediate revocation is one step that can be taken to restore that
confidence, while similarly, a full, holistic, and detailed response may
help mitigate the damage to trust that this misissuance has done.

For examples of what might be considered positive and appropriate
responses, and demonstrates the CAs' awareness and understanding of the
issues:
- Let's Encrypt
-
https://community.letsencrypt.org/t/may-19-2017-ocsp-and-issuance-outage-postmortem/34922
- PKIoverheid
-
https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ

You can also consider post-mortems from related parts, such as CT logs, as
seen in Venafi's CT log post-mortem at
https://groups.google.com/a/chromium.org/d/msg/ct-policy/ohtZ64gLN3I/namq_NDmAQAJ

[Vincent Lynch]

For posterity, here is a link to a separate thread started by D-Trust
containing their response to this report:

<goog_713312620>
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/UnR98QjWQQs

-Vincent

[Jakob Bohm]

Since QuoVadis has not yet responded, let me point to a few (partial)
answers already known from previous messages from QuoVadis or others:

Note that it was (obviously) Siemens that took the SubCA offline, at the
request of QuoVadis.

Note that as you explained above the danger is caused by issuance, not
continued validity. Thus for this particular issue, revocation at any
speed provides no protection. Thus for this *particular* failure,
allowing practical considerations to override the legal 24 hour
requirement should cause no further harm (given that no further such
misissuance will occur).

> For examples of what might be considered positive and appropriate
> responses, and demonstrates the CAs' awareness and understanding of the
> issues:
> - Let's Encrypt
> -
> https://community.letsencrypt.org/t/may-19-2017-ocsp-and-issuance-outage-postmortem/34922
> - PKIoverheid
> -
> https://groups.google.com/d/msg/mozilla.dev.security.policy/vl5eq0PoJxY/W1D4oZ__BwAJ
>
> You can also consider post-mortems from related parts, such as CT logs, as
> seen in Venafi's CT log post-mortem at
> https://groups.google.com/a/chromium.org/d/msg/ct-policy/ohtZ64gLN3I/namq_NDmAQAJ
>

Enjoy

Jakob
--
Jakob Bohm, CIO, Partner, WiseMo A/S. https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark. Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded

[Ryan Sleevi]

On Fri, Aug 18, 2017 at 1:34 AM Jakob Bohm via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:

> Since QuoVadis has not yet responded, let me point to a few (partial)
> answers already known from previous messages from QuoVadis or others:

I believe it would be far more productive for this community if you allow
CAs to respond rather than attempting to speak for them. While I recognize
your desire to help, your replies unfortunately tend to introduce more
confusion and new issues. While not wanting to discourage participation on
this list, some discussions on this list are transparent and public
discussions between CAs and Root Stores, and thus your involvement is
neither beneficial nor necessary. Please allow CAs to answer for themselves.

In this case, for example, you do not actually reference any answers, as
you suggest you have, because these matters have not been answered.

I appreciate your enthusiasm on this topic, and for participation, but as
your replies attempt to speak for either CAs or root stores, they
unfortunately introduce significant harm and confusion. As I am sure you
intend to make productive contributions, it may be best in such cases to
simply observe and ask questions to better understand things, rather than
attempt to provide answers on behalf of others. This applies both on-list
and in bugs.

Thank you for your interest in learning more about these topics, and
hopefully these requests will lead to more productive discussions. As this
transparency is a valuable benefit to the community, it would be truly
unfortunate if such attempts to assist were to undermine and harm such
discussions and result in removing that transparency in order to ensure
that only the authorized representatives were responding.

[Stephen Davidson]

Thanks Ryan, and I note your further posting regarding prompt response.  Noting your desire for detail, I have hesitated to respond with partial answers as both Siemens and QuoVadis are working hard to fix the issues with the Siemens CA and to replace the certificates as quickly as possible.

However, I don’t wish to delay getting more information to you, and ask for your patience if complete information comes in iterations.

Siemens has previously indicated that the affected certificates are installed on high profile websites and infrastructure for Siemen’s group companies around the world, and that a rushed revocation would create more damage than could be expected from the serial number noncompliance.

We have been working with Siemens to dramatically advance the date by which the affected certificates can be replaced and revoked. Siemens predicts that the vast majority of the certificates can be replaced by September 30, 2017 with the few difficult cases following.

Addressing your questions:
1) The failure was one of process rather than deployed code.  QuoVadis made an indepth review of the Siemens CA, policies and practices when we took over the rootsigning, just before the BR changes which raised the serial entropy requirements. At that time it was compliant. QuoVadis formally updates Siemens on changes to applicable standards, and the Siemens PKI team independently monitors groups like the CABF public and m.d.s.p. lists. Siemens were aware of the pending change to 64-bit serials and were prepared to implement them. 

I note that at the same time planning was underway to move from the in-house CA to an OSS CA – precisely for the reason of easing compliance with the increasing pace of change in technical aspects of the BR and other standards, such as CT, CAA and serial entropy. It appears that by oversight, the update to bring the inhouse CA to 64-bits was not deployed, and our expectation was that the check would be made in the external audit. The long transition to the OSS CA is close to completion.

2)      QuoVadis has a dedicated head of compliance and risk management who, in addition to overseeing QuoVadis’ own measures, supervises its external sub-CAs including detailed discussions on evolving standards, checks on implementations, as well as ongoing monitoring of certificate issuance.  There is frequent communication with Siemens and our other root-signed customers.

Siemens has a significant and mature internal audit and external audit regime.  QuoVadis placed too much reliance on the external ETSI TS 102042 V2.4.1 NCP+ DVCP/OVCP audit report for what should have been textbook issues like the serial number entropy. Going forward, QuoVadis will increase the formality of notifications regarding BR approved ballots, requiring documented evidence of compliance by the effective date, and notification to auditors for scope.

Like many CAs, QuoVadis uses crt.sh/certlint to check certificate issuance including for external sub-CAs. This perhaps led to a false sense of security, as certlint does not highlight issues with serial number entropy. Moreover, the fleeting window of visibility in some crt.sh reports may not reveal older issues or certificates that have not appeared in CT. QuoVadis is introducing routine use of certlint in its own certificate management system, and will build an expanded view for our external subCAs (the new Siemens CA will log all SSL in CT, and we intend our other external sub-CAs to do so before the Google deadline).

3) I do not have sufficient information yet to answer your questions regarding the auditor’s practices, and cannot comment on Digicert’s (nor Verizon’s) previous practices.

4) The list of affected certificates is attached in spreadsheet form; they will be uploaded to CT as well.  You will note that the number has declined – Siemens' previous report did not take into account that some of the certificates had already previously been revoked for other reasons.  The spreadsheet also includes certificates issued during the Digicert/Verizon root signing period.

Regards, Stephen

[Matt Palmer]

On Fri, Aug 18, 2017 at 04:04:48PM +0000, Stephen Davidson via dev-security-policy wrote:
> Siemens has previously indicated that the affected certificates are
> installed on high profile websites and infrastructure for Siemen’s group
> companies around the world, and that a rushed revocation would create more
> damage than could be expected from the serial number noncompliance.

Have they considered that the potential outcome of a failure to demonstrate
an ability and willingness to abide by the BRs and Mozilla policy could
include having their intermediate CA certificate distrusted? Would that
create more damage than a "rushed revocation"?

That revocation would need to be "rushed" at all speaks volumes to Siemens'
unfamiliarity with the requirements under which they operate, or else their
unwillingness to abide by those requirements. Revoking certificates within
24 hours of notification of misissuance is a requirement, and they should
know that, have planned for it, and have their systems and processes
designed in such a way as to be able to adhere to it. If that were the
case, it would not be a "rushed" revocation and reissuance, but just
business as usual.

- Matt

[Eric Mill]

On Fri, Aug 18, 2017 at 12:04 PM, Stephen Davidson via dev-security-policy <
dev-secur...@lists.mozilla.org> wrote:
>
> 4) The list of affected certificates is attached in spreadsheet
> form; they will be uploaded to CT as well. You will note that the number
> has declined – Siemens' previous report did not take into account that some
> of the certificates had already previously been revoked for other
> reasons. The spreadsheet also includes certificates issued during the
> Digicert/Verizon root signing period.
>

Would you mind posting this to a public URL or to a Bugzilla bug? The list
doesn't transmit attachments.

-- Eric


>
> Regards, Stephen

>
> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Stephen Davidson]

Ah, my apologies.

https://bugzilla.mozilla.org/attachment.cgi?id=8898848

Regards, Stephen

________________________________________
From: dev-security-policy [dev-security-policy-bounces+s.davidson=quovadisg...@lists.mozilla.org] on behalf of Eric Mill via dev-security-policy [dev-secur...@lists.mozilla.org]
Sent: Saturday, August 19, 2017 12:06 PM
To: Stephen Davidson
Cc: ry...@sleevi.com; mozilla-dev-s...@lists.mozilla.org
Subject: Re: Certificates with less than 64 bits of entropy