dNSName containing '/' / low serial number entropy

[Charles Reiss]

https://crt.sh/?id=174827359 is a certificate issued by D-TRUST SSL
Class 3 CA 1 2009 containing the DNS SAN
'www.lbv-gis.brandenburg.de/lbvagszit' (containing a '/') with a
notBefore in April 2017.

The certificate also seems to have a short certificate serial number,
which cannot include 64 bits of entropy. Many certificates issued by
this CA appears to use large serial numbers (e.g. [1]). But there are
certificates with much shorter sequential-looking serial numbers with
notBefores shortly before [2] and after [3] this certificate's and as
recent as 4 July 2017 [4].

[1] https://crt.sh/?id=137090990 , https://crt.sh/?id=124715040
[2]
https://censys.io/certificates/4445455caca3e9cf2ab2b673304487cb220871aa6d5ac1bf03827f74609c3646
[3]
https://censys.io/certificates/8d08033efe732e8fb6c2f3257c52b500af991bd1f363ffd6e29ec1812a943cd9
[4] https://crt.sh/?id=173758922


I did a cursory check on censys.io to see if there were other cases of
short serial numbers in certificates with recent notBefores that are
trusted by Mozilla:

- Digidentity Services CA - G2 (https://crt.sh/?caid=868 ; chains to
Staat der Nederlanden Root CA - G2) has issued certificates which serial
numbers that appear to be of the form 0x10000000 + sequential counter
with notBefores as recent as 8 June 2017.

- Siemens Issuing CA Internet Server 2016 (https://crt.sh/?caid=26087 ;
chains to QuoVadis Root CA 2 G3) has issued certificates with 4-byte
serial numbers with notBefores as recent as 11 July 2017, though they do
not appear to be assigned sequentially.

D-Trust and QuoVadis both indicated no problems complying with version
2.4.1 of Mozilla's certificate policies (which requires, among other
things, 64 bits of serial number entropy) by 1 June 2017 when they
replied to Mozilla's April CA communication. The Government of the
Netherlands indicated they needed a delay for CPS translation only.

[Gervase Markham]

On 18/07/17 23:25, Charles Reiss wrote:
> https://crt.sh/?id=174827359 is a certificate issued by D-TRUST SSL

<snip>

I'm supposed to be on holiday :-), but I have emailed the 3 CAs
concerned drawing these issues to their attention, and asking them to
comment here when they have discovered the cause.

Perhaps we need a wiki page on "how to best respond to an incident
report from Mozilla"? :-)

Gerv

[Stephen Davidson]

Hello:

Siemens Issuing CA Internet Server 2016 was taken offline upon this report
while Siemens and QuoVadis investigate. It will not issue certificates
until the problem is resolved.

Kind regards, Stephen Davidson
QuoVadis

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Policy Authority PKIoverheid]

Op woensdag 19 juli 2017 00:26:16 UTC+2 schreef Charles Reiss:

> - Digidentity Services CA - G2 (https://crt.sh/?caid=868 ; chains to
> Staat der Nederlanden Root CA - G2) has issued certificates which serial
> numbers that appear to be of the form 0x10000000 + sequential counter
> with notBefores as recent as 8 June 2017.

Hi Charles,

Many thanks for bringing this to our attention. We have looked into this
matter immediately. Meanwhile the Policy Authority PKIoverheid has
prohibited Digidentity (one of the Trusted Service Providers within the
PKIoverheid/Staat der Nederlanden hierarchy) from issuing new certificates.

After investigation it emerged that a total of 777 certificates were issued
from September 30th 2016 that are not compliant with BR ballot 164
(https://cabforum.org/2016/07/08/ballot-164/) echoed by the same requirement
in version 2.4 (Compliance date: February 28, 2017) from the Mozilla CA
Certificate Policy. Digidentity will revoke and
replace these non-compliant certificates. This wil take place on or before
31 August 2017. However this action requires the cooperation from
subscribers. As you know we are in the midst of the Holiday Season so we
can't completly rule out that some certificates will be replaced a couple
of days after August the 31th. Nevertheless Digidentity will do her utmost
to revoke and replace all certs before the 31th.

As evidence that Digidentity is now compliant with regard to the certificate
serial number requirement from the BR check the new issued SSL cert on this
website: https://www.digidentity.eu/nl/home/

The Policy Authority PKIoverheid has judged that Digidentity can resume
issuing certificates now that they are in compliance with Ballot 164 and the Mozilla CA Policy.

Please let me know if you have any questions.

Further questions could also be answered by my collegaues Jorik van 't Hof
or Jochem van den Berge.

Thanks.

Regards,
Mark Janssen

[Alex Gaynor]

Hi Mark,

Are you saying you do intend to revoke all of these certificates in the
next 24 hours?

While subscribers are allowed to continue using bad certificates as long as
they desire, the BRs require CAs to revoke non-compliant certificates
within 24 hours of becoming aware of them.

Alex

On Tue, Jul 25, 2017 at 3:20 PM, Policy Authority PKIoverheid via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:

> Op woensdag 19 juli 2017 00:26:16 UTC+2 schreef Charles Reiss:

> > - Digidentity Services CA - G2 (https://crt.sh/?caid=868 ; chains to
> > Staat der Nederlanden Root CA - G2) has issued certificates which serial
> > numbers that appear to be of the form 0x10000000 + sequential counter
> > with notBefores as recent as 8 June 2017.
>
>

[Arno Fiedler]

Dear Mozilla Security Policy Community,

Thanks for the advice about the short serial numbers and apologies for the delayed response.

Since 2016, all D-TRUST TLS certificates based on electronic Certificate Requests have a certificate serial number which includes 64 bits of entropy.

Between 2012 and July 6th, 2017 we produced a small number of certificates with paper-based Certificate Registration Requests using 64 bits of entropy in the “DNqualifier” field instead of the serial number field.

Since the 7th of July, 2017, all D-TRUST TLS-Certificates have 64 bits of entropy in the serial number.

I hope this helps and please do not hesitate to contact us if there are any further questions.

Best regards
Arno Fiedler
Standardization & Consulting
Bundesdruckerei GmbH
Kommandantenstraße 18 · 10969 Berlin · Deutschland

[Policy Authority PKIoverheid]

Dear Mozilla Security Policy Community,

My apologies for the delayed follow up response.

As stated in my email from 07/25/2017, Digidentity (DDY), one of our TSP’s, issued 777 certificates from September 30th 2016 which were not compliant with BR ballot 164.

DDY has fixed the problem with the serial generation and is in the process of replacing all 777 non-compliant certificates.

Below you will find the answers on the following questions:
1. Why did Logius/Policy Authority PKIoverheid not detect, identify, disclose, and resolve this matter prior to public notification?
2. Why did DDY not implement the serial number entropy as required by the Baseline Requirements?
3. Was this detected by the auditor? If not, why not?

ANSWER ON QUESTION 1:
Logius PKIoverheid was notified by Gervase Markham to draw the issue to their attention. See the timeline for further details.

Logius PKIoverheid relies on the audits performed by external auditors to make sure that the Trusted Service Providers (TSPs) aka CAs within the PKIoverheid/Staat der Nederlanden hierarchy fully comply with applicable requirements (like the BR, ETSI and our own Program of Requirements).

For further details about the PKIoverheid architecture aka hierarchy see one of these bugs: https://bugzilla.mozilla.org/show_bug.cgi?id=529874 or https://bugzilla.mozilla.org/show_bug.cgi?id=1016568

Inform
• Our TSPs are responsible to follow relevant changes in the BR. Besides that the Policy Authority (PA) PKIoverheid informs all PKIoverheid-TSPs about (intended) relevant changes to the Baseline Requirements.

Check
• We require a yearly full ETSI EN 319 411-1 audit. In the case of DDY the most recent full audit is of November 2016.
• We require a ETSI accredited auditor. In the case of DDY the auditor is BSI and in 2016 it was accredited by the RvA (In Dutch: Raad voor Accreditatie), the Dutch accreditation body (for more information see: https://www.rva.nl/en/our-organization/about-the-rva).
• We manually take samples of the issued certificates from our TSPs using CT logs. Unfortunately DDY was not part of the latest samples (see new measure 2).

Approve
• The PA PKIoverheid reviews the audit rapports from the TSPs and if necessary will take measures to make sure the TSP conforms to the applicable audit requirements.


Timeline (all times are UTC):

19 July 2017 00:27: A posting on mozilla.dev.security.policy stating a non-compliant certificate issued by DDY.

https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/vl5eq0PoJxY

20 July 2017 16:45: Mozilla (Gerv) notifies the Policy Authority (PA) PKIoverheid on non-compliant certificates from DDY

20 July 2017 16:45: START INCIDENT

20 July 2017 17:27: PA PKIoverheid starts investigating the issue and almost immediately raises an internal incident.

21 July 2017 09:08: PA PKIoverheid issues DDY to postpone further issuing of certificates and requests an action plan from DDY how they will resolve the issue by revoking and reissuing all certificates involved.

21 July 2017 09:50: DDY confirms postponing the issuing of certificates.

21 July 2017 09:50: FURTHER CERTIFICATE ISSUING SUSPENDED

24 July 2017 08:53: DDY delivers action plan including two newly generated and compliant test certificates as proof that they fixed the issue.

24 July 2017 16:25: Based on the provided certificates the PA PKIoverheid requests DDY to start executing the action plan including the approval to restart issuing certificates.

24 July 2017 16:25: ISSUING RESTARTED

25 July 2017 14:37: DDY installs first production certificate on website (https://www.digidentity.eu/nl/home/)

25 July 2017 14:37: DDY starts revoking and replacing certificates

25 July 2017 21:20: PA PKIoverheid post a message on mozilla.dev.security.policy stating that DDY has proven to be able to generate compliant certificates and is allowed to restart the issuing of new (compliant) certificates. A link to the compliant new DDY certificates is included in this post as evidence.

26 July 2017 17:40: PA PKIoverheid requests Mozilla to honor the request to extend the
24-hour revocation time.

27 July 2017 10:31: Mozilla grants the PA PKIoverheid the request to extend the revocation time (that is, Mozilla will accept an audit qualified by this event).

27 July 2017 13:21: PA PKIoverheid requests the other Application Software Suppliers to agree with Mozilla.

28 July 2017 12:54: PA PKIoverheid informs DDY that it requests an audit statement regarding the Ballot 164 incident.

28 July 2017 12:54: PA PKIoverheid starts monitoring the resolving progress but keeps the incident open until all involved certificates are revoked or replaced.
INCIDENT UNDER CONTROL


ANSWER ON QUESTION 2:
DDY concluded wrongly that ballot 164 was not applicable for them since the use of sequential serial numbers is not a security risk when used in conjunction with the SHA-256 with RSA encryption certificate signing scheme.


ANSWER ON QUESTION 3:
Non-compliance with this requirements wasn’t noticed by the auditor because DDY didn’t include the specific requirement in their Statement of Applicability (reason: see the answer on question 2). ETSI EN 319 403 (which determines the requirements for conformity assessment bodies) is not clear about who determines the scope of an audit. The auditor’s interpretation was that the client (DDY) had to determine the scope of the audit (based on their Statement of Applicability). This will be mitigated for future audits with new measure 4.


NEW MEASURES:
The following measures are newly introduced to mitigate risks that have been identified during the evaluation of this incident.
1. The implementation of Certificate Transparancy (CT) will reduce the detection-time of non-compliant certificates to hours instead of days/weeks/months. Implementation date is set on October 1st.
2. The PA PKIoverheid will increase the rate of random certificate checks to weekly as long as CT has not been implemented. Effective immediately.
3. After a ballot has been adopted by the CABForum the PA PKIoverheid will require of it TSP’s to show proof of conformance to the ballot before the effective date. If a TSP isn't able to show proof of conformance as of the effective date, certificate issuance will be suspended until the TSP can supply evidence to the contrary. Effective 09/01/2017.
4. PA PKIoverheid will review and/or determine the scope for the TSP in question before any type of audit. Without clearance/approval of the PA PKIoverheid no audit or issuance of certificates will take place. Effective 09/01/2017.
5. Research into possible implementation of (automated) Certlint checks in the issuance process.


Best Regards,
Mark Janssen

[Ryan Sleevi]

Mark,

Thanks for providing a detailed report about this, including the steps
being taken to prevent future events like this. Your proposed remediation
plans sound like excellent steps to ensure future conformance, and
demonstrate an understanding as to the root causes and how to prevent them
in the future. More importantly, they demonstrate a level of attention that
hopefully all CAs engaging in third-party cross-signing should aspire to -
namely, the oversight and supervision of the scope of audits to ensure all
necessary controls are in place, the integration of automated checks, and
greater transparency.

[Nick Lamb]

On top of what Ryan has written, I want to specifically praise the approach of actually checking a sample of certificates as PKIoverheid describes.

I think done well this can be a very affordable yet timely and effective way to detect problems in a particular issuance pipeline or with a particular subCA. It won't catch everything, and it's not a substitute for audits, but I feel as though if every CA was doing this we'd see far fewer incidents, and more of them would be self-reported to m.d.s.policy in a timely fashion rather than being found by independent researchers months after the fact, with consequently fewer subscribers adversely affected.

[Gijs Kruitbosch]

On 11/08/2017 15:39, Policy Authority PKIoverheid wrote:
> 2. Why did DDY not implement the serial number entropy as required by the Baseline Requirements?
> 3. Was this detected by the auditor? If not, why not?

<snip>

> ANSWER ON QUESTION 2:
> DDY concluded wrongly that ballot 164 was not applicable for them since the use of sequential serial numbers is not a security risk when used in conjunction with the SHA-256 with RSA encryption certificate signing scheme.
> ANSWER ON QUESTION 3:
> Non-compliance with this requirements wasn’t noticed by the auditor because DDY didn’t include the specific requirement in their Statement of Applicability (reason: see the answer on question 2). ETSI EN 319 403 (which determines the requirements for conformity assessment bodies) is not clear about who determines the scope of an audit. The auditor’s interpretation was that the client (DDY) had to determine the scope of the audit (based on their Statement of Applicability). This will be mitigated for future audits with new measure 4.

(apologies if this is a dumb question...)

Can Mozilla / the BRs / whatever enforce making this [ie who determines
the scope of the audits] explicit so issues don't get missed because the
CA/TSP/subCA/intermediates and/or auditor mistakenly believe some items
don't apply? Could we standardize/require some of this "Statement of
Applicability" stuff to be a superset of the BRs, applicable RFCs, etc. ?

Or is that going to be useless either because whatever requirements on
audits/auditors that Mozilla / the BRs would suggest get "trumped" by
ETSI or other rules we can't (directly) influence, or because there are
so many possible permutations of applicability/scope that trying to
specify them in some way defeats the point, in that it would cause more
rather than less confusion?

(just trying to figure out if there is some way we can avoid a
reoccurrence of confusion with other issuers and/or auditors)

~ Gijs

[Kim Nguyen]

Am Mittwoch, 19. Juli 2017 00:26:16 UTC+2 schrieb Charles Reiss:

> https://crt.sh/?id=174827359 is a certificate issued by D-TRUST SSL
> Class 3 CA 1 2009 containing the DNS SAN
> 'www.lbv-gis.brandenburg.de/lbvagszit' (containing a '/') with a
> notBefore in April 2017.
>

Regarding this Topic, this incorpates the D-Trust PostMortem, Remidiation&Mitigation and Revocation Status. Regards, Kim

Issue dNSName containing '/', https://crt.sh/?id=174827359

PostMortem:
An incident was triggered by a bug in mozilla.dev.security.policy 07-08-2017.
Issuance was stopped immediately at 07-08-2017
Analysis yielded the following results:
Validation is based on both a four-eyed principle “human” approach as well as a tool based automated validation.
The GUI of our validation software backend which our team is using had some usability and visualization related issues. This implied that the way multiple SANs were displayed had potential for mistakes. We released the improvement of the backend GUI on the 2017-08-24 as previously announced.
The bug mentioned with respect to the CSR Validator was that the CSR validator didn’t filter prohibited characters correctly and was introduced by the previous release but was not recognized during test.

Mitigation/Remediation:
Existing Mitigations: Certificates require two independent parties to approve ("four eyes principle")

Remediation:
2017-08-15 - The Certificate was revoked
2017-08-24 - Hotfix to systems to validate CSR against RFC 5280
2017-08-24 - Hotfix to update validation software UI to reduce risk of mistakes
2018-03-31 - Improved software testing to consider such cases
In order to enhance the quality assurance during issuance we are setting up both manual random checks as well as automated compliance checks in our issuance system.
Also a case-related awareness training was performed.

Revocation plan:
The cert was revoked, a new BR compliant cert was issued for the costumer