Incident Report: Test revoked certificates created with an incorrect validity period

[Ponds-White, Trevoli]

Hi Wayne,

This is a report about the certificates used by software vendors for testing. Specifically the "revoked" ones we make for Amazon Trust Services.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
When creating the certificates that software vendors can use to test revoked certificates we set the validity period to incorrectly be 39 months and included an incorrect subject which makes the certificates appear to be EV certificates. As part of our post ceremony validation we ran cablint on all certificates on 2/1/2019. After doing this we discovered that our revoked certificates had been incorrectly formatted.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.


* 1/28/2019 - We created 5 certificates for the purpose of providing test revoked certificates. Upon creation of each certificate, they were revoked about a minute later.

* 2/1/2019 - While going through the steps to verify the ceremony artifacts we ran cablint on the certificates and discovered that the revoked certificates were being identified as EV certs and were valid for 39 months.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We will not issue test revoked certificates with an incorrect validity or subject again.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

5 certificates, all created on 1/28/2019

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
I will send out a link to the certs once we have uploaded them. All 5 have the same issue.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

When the validity period for certificates changed with ballot 193 to 825 days we didn't update the default validity period or the guardrail that limits the maximum validity period for test revoked certificate generation. This didn't impact other certificates, such as our test "good" certificates which already defaulted to 13 months and the guardrail already limited the validity period to not be longer than 825 days.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.


* We've updated the script to default to 13 months for test revoked certificates and the guardrail so that these types of test certs cannot have a validity period of longer than 825 days.

* We've updated the commands template to have the correct subject.

Thanks,
Trevoli Ponds-White | Compliance Manager | Amazon Trust Services
e: trev...@amazon.com<mailto:trev...@amazon.com> c: 425-299-6152

[Ryan Sleevi]

Thanks Trev, I've opened up
https://bugzilla.mozilla.org/show_bug.cgi?id=1525710 for this.

While this is certainly the first case for Amazon, it does match a worrying
trend we're seeing with CAs, particularly around manually generated
certificates. Google's recent case with an invalid OCSP response (
https://bugzilla.mozilla.org/show_bug.cgi?id=1522975 ) sounds similar in
nature.

Your remediation plan seems to speak about this specific incident, but it's
not clear that a full root cause analysis has been done. Specifically, from
the description, it sounds that there was a gap in reviewing these
templates for compliance - with the CA's CP/CPS and with the BRs - prior to
performing this exercise.

It seems that there's an opportunity to understand why these templates
weren't reviewed or updated, and what systemic steps are being taken to
ensure that going forward. I would similarly expect this of all CAs
participating in m.d.s.p., to review all paths for manual issuance, and
review the controls and processes around what can cause such issuance and
how compliance is assured.

Could you please perform a more thorough analysis, and update the bug with
additional analysis and remediation steps? Thanks.

[Ponds-White, Trevoli]

Thanks Ryan! I’ve linked this bug to our internal tracking and will update with more details.

From: Ryan Sleevi <ry...@sleevi.com>
Sent: Wednesday, February 6, 2019 12:51
To: Ponds-White, Trevoli <trev...@amazon.com>
Cc: dev-secur...@lists.mozilla.org
Subject: Re: Incident Report: Test revoked certificates created with an incorrect validity period

On Wed, Feb 6, 2019 at 2:41 PM Ponds-White, Trevoli via dev-security-policy <dev-secur...@lists.mozilla.org<mailto:dev-secur...@lists.mozilla.org>> wrote:
Hi Wayne,

This is a report about the certificates used by software vendors for testing. Specifically the "revoked" ones we make for Amazon Trust Services.

1. How your CA first became aware of the problem (e.g. via a problem report submitted to your Problem Reporting Mechanism, a discussion in mozilla.dev.security.policy, a Bugzilla bug, or internal self-audit), and the time and date.
When creating the certificates that software vendors can use to test revoked certificates we set the validity period to incorrectly be 39 months and included an incorrect subject which makes the certificates appear to be EV certificates. As part of our post ceremony validation we ran cablint on all certificates on 2/1/2019. After doing this we discovered that our revoked certificates had been incorrectly formatted.

2. A timeline of the actions your CA took in response. A timeline is a date-and-time-stamped sequence of all relevant events. This may include events before the incident was reported, such as when a particular requirement became applicable, or a document changed, or a bug was introduced, or an audit was done.


* 1/28/2019 - We created 5 certificates for the purpose of providing test revoked certificates. Upon creation of each certificate, they were revoked about a minute later.

* 2/1/2019 - While going through the steps to verify the ceremony artifacts we ran cablint on the certificates and discovered that the revoked certificates were being identified as EV certs and were valid for 39 months.

3. Whether your CA has stopped, or has not yet stopped, issuing certificates with the problem. A statement that you have will be considered a pledge to the community; a statement that you have not requires an explanation.

We will not issue test revoked certificates with an incorrect validity or subject again.

4. A summary of the problematic certificates. For each problem: number of certs, and the date the first and last certs with that problem were issued.

5 certificates, all created on 1/28/2019

5. The complete certificate data for the problematic certificates. The recommended way to provide this is to ensure each certificate is logged to CT and then list the fingerprints or crt.sh IDs, either in the report or as an attached spreadsheet, with one list per distinct problem.
I will send out a link to the certs once we have uploaded them. All 5 have the same issue.

6. Explanation about how and why the mistakes were made or bugs introduced, and how they avoided detection until now.

When the validity period for certificates changed with ballot 193 to 825 days we didn't update the default validity period or the guardrail that limits the maximum validity period for test revoked certificate generation. This didn't impact other certificates, such as our test "good" certificates which already defaulted to 13 months and the guardrail already limited the validity period to not be longer than 825 days.

7. List of steps your CA is taking to resolve the situation and ensure such issuance will not be repeated in the future, accompanied with a timeline of when your CA expects to accomplish these things.


* We've updated the script to default to 13 months for test revoked certificates and the guardrail so that these types of test certs cannot have a validity period of longer than 825 days.

* We've updated the commands template to have the correct subject.

Thanks,
Trevoli Ponds-White | Compliance Manager | Amazon Trust Services

e: trev...@amazon.com<mailto:trev...@amazon.com><mailto:trev...@amazon.com<mailto:trev...@amazon.com>> c: 425-299-6152

[Ponds-White, Trevoli]

Hi Ryan / Wayne,

Here is the list of the certificates:
https://crt.sh/?id=1180004125
https://crt.sh/?id=1180002617
https://crt.sh/?id=1180004198
https://crt.sh/?id=1180004200
https://crt.sh/?id=1180004199

Thanks,
Trevoli Ponds-White | Compliance Manager | Amazon Trust Services

_______________________________________________
dev-security-policy mailing list
dev-secur...@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-security-policy

[Ponds-White, Trevoli]

Thanks for following up on this Ryan.

Test certificates from these CAs were first generated in October 2015 with 39 month validity. While we had ceremonies since that time as part of the controls for WebTrust audit, we did not include renewal of these certificates until 2019. We did and do have a process for updating our CP/CPS when CA/B Forum and root program requirements change. Our CPS correctly contained the 825 day change from Ballot 193. However, we didn't have a step in our off-line issuance to review artifacts prior to signing against our CPS.

We have updated our off-line issuance process to require review of artifacts both prior to and following issuance against our CPS. We have also added a requirement to lint artifacts both prior to and following issuance. Either of these two changes would have caught the fact we had not changed the template for these test certificates to comply with Ballot 193.

I've also updated our bug with this.