Microsec: Issuance of 2 IVCP precertificates without givenName, surName, localityName fields

[Sándor dr. Szőke]

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
Microsec became aware of the problem from the new discussion at the mozilla.dev.security.policy
https://groups.google.com/forum/#!topic/mozilla.dev.security.policy/jRKOr4nvOfY

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.
2019-10-02 2 precertificates were issued for internal testing purposes with short validity
2020-03-10 Microsec was informed about the faulty precertificates
2020-03-10 Microsec checked the faulty precertificates. They were already expired, so revocation was not possible.
2020-03-10 Microsec decided to immediately stop issuance of IVCP certificates until all corrective measures are in place, to prevent similar mistakes in the future
2020-03-10 Microsec decided to develop the CA software by adding further controls regarding the required fields of IVCP certificates to guarantee compliance with the certificate profile in the future
2020-03-13 Microsec deactivated all the IVCP profiles in the CA software to prevent issuance of IVCP certificates until the controls in the CA software are in place
2020-03-13 Microsec opened this incident report

3.Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.
Two subordinate CA were affected. They haven’t been stopped, but the issuance of IVCP certificates has been suspended by informing the RA operators about the decision and by deactivating all the IVCP certificate profiles.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.
There are 2 certificates involved.
They were issued on the same day which was 2019-10-02


5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
The problematic certificates are:
https://crt.sh/?id=1947655126&opt=zlint,ocsp
https://crt.sh/?id=1947655112&opt=zlint,ocsp

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.
Microsec made an investigation and could find the following:
- the certification policy and practice statement contain the proper requirements regarding the content of the IVCP certificates
- the problem was caused by human mistakes, the RA operators could not recognize the missing data
- the CA software presently does not have automatic controls to check for the existence of the required fields in case of IVCP certificates
- the certificates have been checked by cablint, but cablint did not indicate this fault
- the certificates have never been published and used, so the fault has not been discovered until now

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.
Microsec decided the following measures to avoid having the same problem in the future.
- the problematic precertificates were issued for short period and have already expired, so revocation is not necessary
- suspended the issuance of the IVCP certificates with immediate effect
- decided to develop the CA software to add further controls and checks in case of IVCP certificates - no deadline has been set due to the COVID-19 situation
- decided to hold a training about the required IVCP profiles for the RA operators before enabling the issuance of IVCP certificates again
- the issuance of IVCP certificates may be continued only after the successful testing of the upgraded CA software
- Microsec will check all the issued IVCP certificates looking for similar issues - deadline 2020-03-20
- Microsec will review the CA software looking for possible similar problems - deadline 2020-03-31

[Wayne Thayer]

I have created bug #1622539 to track this issue.

- Wayne

> _______________________________________________
> dev-security-policy mailing list
> dev-secur...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-security-policy
>

[Sándor dr. Szőke]

>
> - Microsec will check all the issued IVCP certificates looking for similar issues - deadline 2020-03-20
>

Microsec has finished the detailed investigation on the issued TLS IVCP certificates looking for similar issues. The findings are the following:

Microsec issued altogether 9 test certificates on 2019-10-01 and 2019-10-02 with 30 days validity.
The purpose of the test was to issue DV and IV certificates from each subordinate CA which is used to issue these types of certificates.
The test covered both the RSA-based hierarchy and the ECC-based new CA hierarchy.
The test certificates were issued directly from the CA software by using the operator interface. The CA software forces the use of dual control.
The RSA-based system is configured to use Certificate Transparency to fully comply with the present requirements. 4 of the 9 test certificates were issued in the RSA-based system.
The ECC-based system was configured to issue the test certificates without Certificate Transparency, because this root is not trusted yet and none of the log servers issues SCT for this root. 5 of the 9 test certificates were issued in the ECC-based system.
The Subject DN fields were configured according to the DV profile requirements, this way the issuance of the DV certificates was successful. The 2 successfully issued RSA-based DV certificates can be found in the crt.sh as
https://crt.sh/?q=1947651733
https://crt.sh/?q=1944631156
The issuance of the test IV certificates was done using the same Subject DN fields by mistake.
None of the operators identified the missing fields in case of IV certificates.

The following RSA-based IV certificates were issued with missing fields in the Subject name:
https://crt.sh/?q=1947655126
https://crt.sh/?q=1947655112

They are already included in the incident report and there are no other IV certificates issued under the RSA root with this problem.

A similar set of 3 DV and 2 IV test certificates were issued under the ECC root, but without SCT, as explained above. Because of this, they can’t be found in the crt.sh. The 3 DV test certificates were correct. The 2 IV test certificates have the same problem: missing givenName, surName, localityName fields in the Subject DN. These certificates are already expired, so revocation is not possible.
Microsec has only issued test TLS certificates under the ECC root so far.
The cause of the problem was the same as for the RSA-based test certificates, so the remediation and preventive measures are the same too.

[Sándor dr. Szőke]

> - Microsec will review the CA software looking for possible similar problems - deadline 2020-03-31

Microsec has completed a detailed review of the automatic controls built into the CA software. The review covered all SSL/TLS certificate types and focused on the presence of required fields in the Subject DN.

Microsec first created a table with all possible Subject DN fields based on the current version of the CABF BR, EVG, and Microsec CPS documents. The following certification policies are included in the table: DVCP, IVCP, OVCP, EVCP/QWAC, EVCP/PSD2. Microsec has collected rules for each field and policy combination, which may include:
 mandatory
 forbidden
 optional

For optional fields, Microsec also identified dependencies.

After completing the complete list of requirements, Microsec reviewed the source code for the CA software. As a result of this review, Microsec determined that the scope of automated checks should be expanded for the IVCP profile, but they are complete for all other certificate profiles.

Microsec has created a work item and has already begun to upgrade the CA program with additional controls. There is no specific deadline for completing the development, but Microsec plans to do the development and test the new software version by 2020-04-15.

[Ryan Sleevi]

On Tue, Mar 31, 2020 at 4:46 PM Sándor dr. Szőke via
dev-security-policy <dev-secur...@lists.mozilla.org> wrote:
>
>
> > - Microsec will review the CA software looking for possible similar problems - deadline 2020-03-31
>
>

> Microsec has completed a detailed review of the automatic controls built into the CA software. The review covered all SSL/TLS certificate types and focused on the presence of required fields in the Subject DN.
>
> Microsec first created a table with all possible Subject DN fields based on the current version of the CABF BR, EVG, and Microsec CPS documents. The following certification policies are included in the table: DVCP, IVCP, OVCP, EVCP/QWAC, EVCP/PSD2. Microsec has collected rules for each field and policy combination, which may include:
>  mandatory
>  forbidden
>  optional

Do you plan to share the analysis?

I think saying "We compiled X" isn't nearly as useful to the community
as "We analyzed X, here's what we concluded, we're looking for
feedback and/or sharing for wider review"

This broadly fits into the picture of
https://groups.google.com/d/msg/mozilla.dev.security.policy/oP8XuNXrANw/oIYt70IiAAAJ

[dr. Szőke Sándor]

Dear Ryan,

we will translate the Excel table into English and will upload it to the discussion thread today.

It may be helpful for other CAs to learn from this issue and to help others prevent them from becoming a victim of a similar incident.

Best Regards,

Sándor


Dr. Sándor SZŐKE
dep. Director of eIDAS Trust Services



Microsec Ltd. | Ángel Sanz Briz Road 13.
Budapest, H-1033 Hungary
Graphisoft Park Southern Area, Building C, 3th floor
T: +36 1 802-4418 | +36 1 505-4477 / 488
<mailto:sandor...@microsec.com> sandor...@microsec.com
microsec.com

-----Original Message-----
From: Ryan Sleevi <ry...@sleevi.com>
Sent: Tuesday, March 31, 2020 11:57 PM
To: Sándor dr. Szőke <szoke....@microsec.hu>
Cc: mozilla-dev-security-policy <mozilla-dev-s...@lists.mozilla.org>
Subject: Re: Microsec: Issuance of 2 IVCP precertificates without givenName, surName, localityName fields

On Tue, Mar 31, 2020 at 4:46 PM Sándor dr. Szőke via

dev-security-policy < <mailto:dev-secur...@lists.mozilla.org> dev-secur...@lists.mozilla.org> wrote:
>
>
> > - Microsec will review the CA software looking for possible similar problems - deadline 2020-03-31
>
>

> Microsec has completed a detailed review of the automatic controls built into the CA software. The review covered all SSL/TLS certificate types and focused on the presence of required fields in the Subject DN.
>
> Microsec first created a table with all possible Subject DN fields based on the current version of the CABF BR, EVG, and Microsec CPS documents. The following certification policies are included in the table: DVCP, IVCP, OVCP, EVCP/QWAC, EVCP/PSD2. Microsec has collected rules for each field and policy combination, which may include:
>  mandatory
>  forbidden
>  optional

Do you plan to share the analysis?

I think saying "We compiled X" isn't nearly as useful to the community
as "We analyzed X, here's what we concluded, we're looking for
feedback and/or sharing for wider review"

This broadly fits into the picture of

<https://groups.google.com/d/msg/mozilla.dev.security.policy/oP8XuNXrANw/oIYt70IiAAAJ> https://groups.google.com/d/msg/mozilla.dev.security.policy/oP8XuNXrANw/oIYt70IiAAAJ

[Sándor dr. Szőke]

I have uploaded the Excel sheet to the following bug:

https://bugzilla.mozilla.org/show_bug.cgi?id=1622539#c3

[Sándor dr. Szőke]

I give you a brief status report on CA software development.

Microsec has already completed the development of CA software and introduced additional automatic checks for the proper configuration of the SN fields in case of different certificate types.
A number of internal tests have already been performed on the test system, but the new release is still awaiting final acceptance tests.
The new release of the CA software has not been activated in the live system yet, the scheduled activation date is 2020-05-20.
IVCP profiles are still inactive, Microsec does not issue IVCP certificates.

As Microsec informed the community in its first comment (2020-03-13), there was no deadline for CA software development due to the COVID-19 emergency situation.
This information was confirmed in comment 2020-03-19, and the date 2020-04-15 was given as a plan only and not as a strict deadline.

Microsec will immediately notify the community in case of any further change in this thread.

[Sándor dr. Szőke]

I inform you that as planned two days ago, Microsec today activated the new CA software release in the live system.

The CA sofware has been improved to support more automatic checking for the presence of SN fields for different certificate profiles.

As part of the project, Microsec has developed an automated testing tool that allows for more efficient testing of the CA program.
Test inputs and expected CA responses can be given in a configuration file, and the test runs automatically in batch mode.
The test was passed without error.
The development was approved yesterday and the installation of the new release on the live system was approved.

Despite the software upgrade, IVCP profiles are still not active, and Microsec does not issue IVCP certificates.
There is no official deadline for activation, but Microsec plans to activate IVCP profiles together with the other profile changes due to the release of new public documents on 2020-05-26.

Prior to activation, Microsec will organize a training for Registration Officiers on the proper configuration of different TSL profiles.

Microsec will immediately notify the community as soon as any changes occur.

[Sándor dr. Szőke]

We organized the training for our Registration Officers regarding the proper configuration of different TSL profiles today.

The IVCP profiles are planned to be reactivated tomorrow.

[Sándor dr. Szőke]

I inform you that Microsec activated the IVCP profiles today.


It is possible to issue IVCP certificates again.