Public Discussion of Google Trust Services' Request to Replace Root CA Certificates

1,230 views
Skip to first unread message

Ben Wilson

unread,
Aug 25, 2021, 5:26:24 PM8/25/21
to dev-secur...@mozilla.org

This is to announce the beginning of the public discussion phase for Google Trust Services' (GTS) request to replace five existing root CA certificates with ones that were re-signed last year, August 13, 2020.  See https://wiki.mozilla.org/CA/Application_Process#Process_Overview, (Steps 4 through 9).

The five roots are as follows:  GTS Root R1, GTS Root R2, GTS Root R3, GTS Root R4, and the GlobalSign ECC Root CA - R4.  (A sixth root CA certificate, the GlobalSign Root CA - R2, was re-signed using SHA1, and so I have removed it from this inclusion request.) The reason for their replacement is that the original CA certificates do not contain the digitalSignature key usage bit, which is required for direct OCSP signing by the CA. (See https://bugzilla.mozilla.org/show_bug.cgi?id=1652581)

GTS’ request has been tracked in the CCADB and in Bugzilla as follows:

https://ccadb-public.secure.force.com/mozilla/PrintViewForCase?CaseNumber=00000666

https://bugzilla.mozilla.org/show_bug.cgi?id=1675821

Mozilla is considering approving GTS’ request. This email begins a 3-week comment period, after which, if no concerns are raised, we will close the discussion and the request may proceed with steps required to replace the certificates in question.

Root Certificate Information:

GTS Root R1

https://crt.sh/?q=D947432ABDE7B7FA90FC2E6B59101B1280E0E1C7E4E40FA3C6887FFF57A7F4CF

    Download – https://pki.goog/repo/certs/gtsr1.pem

Replaces https://crt.sh/?id=139646520

GTS Root R2

https://crt.sh/?q=8D25CD97229DBF70356BDA4EB3CC734031E24CF00FAFCFD32DC76EB5841C7EA8

    Download – https://pki.goog/repo/certs/gtsr2.pem

Replaces https://crt.sh/?id=139646522

GTS Root R3

https://crt.sh/?q=34D8A73EE208D9BCDB0D956520934B4E40E69482596E8B6F73C8426B010A6F48

    Download – https://pki.goog/repo/certs/gtsr3.pem

Replaces https://crt.sh/?id=139646519  

GTS Root R4

https://crt.sh/?q=349DFA4058C5E263123B398AE795573C4E1313C83FE68F93556CD5E8031B3C7D

    Download – https://pki.goog/repo/certs/gtsr4.pem

Replaces https://crt.sh/?id=139646525

GlobalSign ECC Root CA - R4

https://crt.sh/?q=B085D70B964F191A73E4AF0D54AE7A0E07AAFDAF9B71DD0862138AB7325A24A2

    Download – https://pki.goog/repo/certs/gsr4.pem

Replaces https://crt.sh/?id=8644166

 

CP/CPS:   

Current CPS is Version 4.0 /  August 11, 2021

https://pki.goog/repo/cps/4.0/GTS-CPS.pdf

Repository location:   https://pki.goog/

 

Audits: 

GTS’s WebTrust auditor is Ernst & Young, and the most recent audit reports are dated November 2, 2020. These may be downloaded by clicking on the WebTrust seals on GTS’s repository page.  The WebTrust Baseline Requirements audit noted the following four incidents (closed):

1 - https://bugzilla.mozilla.org/show_bug.cgi?id=1630040 (OCSP responder issue)

2 - https://bugzilla.mozilla.org/show_bug.cgi?id=1652581 (digitalSignature KeyUsage not set – which gave rise to this inclusion request)

3 - https://bugzilla.mozilla.org/show_bug.cgi?id=1625498 (tracking possible audit delay)

4 - https://bugzilla.mozilla.org/show_bug.cgi?id=1667844 (certificates not disclosed in CCADB)

Other Incidents (Closed):

5 - https://bugzilla.mozilla.org/show_bug.cgi?id=1678183 (invalid ASN.1 encoding in OCSP response)

6 - https://bugzilla.mozilla.org/show_bug.cgi?id=1706967 (CPS stated outdated DV method from BR 3.2.2.4.10)

7 - https://bugzilla.mozilla.org/show_bug.cgi?id=1708516 (delayed incident updates)

8 - https://bugzilla.mozilla.org/show_bug.cgi?id=1709223 (SHA1 signing of GlobalSign Root CA - R2)

9 - https://bugzilla.mozilla.org/show_bug.cgi?id=1715421 (delayed revocation of end entity certificate)

Thus, this email begins a three-week public discussion period, which I’m scheduling to close on or about 15-September-2021.

A representative of GTS must promptly respond directly in the discussion thread to all questions that are posted.

Sincerely yours,

Ben Wilson

Mozilla Root Program

Andrew Ayer

unread,
Aug 31, 2021, 10:26:21 AM8/31/21
to dev-secur...@mozilla.org
Prior to 2021-08-11, Google Trust Services' CPS (version 3.4,
https://pki.goog/repo/cps/3.4/GTS-CPS.pdf) contained the following
exception to checking CAA:

"If Google is the DNS Operator (as defined in RFC 7719) of the domain's DNS."

This exception was banned by CABF Ballot SC46, which passed
on 2021-06-02 and became effective 2021-07-12. However, it
was not removed from GTS' CPS until 2021-08-11 (version 4.0,
https://pki.goog/repo/cps/4.0/GTS-CPS.pdf), with the changelog note
"Updated various sections following full CPS review".

https://bugzilla.mozilla.org/show_bug.cgi?id=1706967 describes a similar
incident from April of this year - a failure by GTS to update their CPS
to reflect BR changes. To "prevent similar issues from happening again",
GTS committed to making several changes, including:

1. "Implementing automation that monitors the Baseline Requirements
document repository and Mozilla tickets and automatically creates tickets
in [GTS'] internal tracking system", scheduled to be implemented by
2021-06-15 (https://bugzilla.mozilla.org/show_bug.cgi?id=1706967#c11)

2. Instituting a weekly meeting (previously bi-weekly) to "ensure
continual compliance with the Baseline Requirements", beginning 2021-05-03
(https://bugzilla.mozilla.org/show_bug.cgi?id=1706967#c11).

Despite detecting and removing the non-compliant DNS Operator exception
during a CPS review, GTS has not filed an incident report about the
non-compliance. https://bugzilla.mozilla.org/show_bug.cgi?id=1708516
describes another failure by GTS to file an incident report
as well as generally poor compliance practices by GTS
(https://bugzilla.mozilla.org/show_bug.cgi?id=1708516#c35).
This bug was ultimately closed on 2021-08-25 despite
lingering concerns about the adequacy of GTS' response
(https://bugzilla.mozilla.org/show_bug.cgi?id=1708516#c53).

It would appear from the above that while GTS was assuring the community
that their compliance program was improved, they were simultaneously
experiencing a recurrent compliance incident which they did not disclose.

In light of the above, I have the following questions for GTS:

1. Has Google Trust Services used the DNS Operator exception to skip
CAA checking for any certificates issued on or after 2021-07-12?

2. Why did the changes implemented in response to
https://bugzilla.mozilla.org/show_bug.cgi?id=1706967 not prevent
this incident?

3. Why did GTS not file an incident report when the CPS review led
to the non-compliant CAA exception being removed from CPS version 4.0
on 2021-08-11?

Regards,
Andrew

Brett L

unread,
Sep 1, 2021, 7:30:35 PM9/1/21
to dev-secur...@mozilla.org, Andrew Ayer

Hi Andrew,

Thank you for the questions and checking on details. 

We removed the option to use the DNS operator exception from our secondary CA platform on 2021-05-13 (60 days before the ballot changes went into effect, see timeline below). Our primary CA platform has never used it.

In May, we conducted our annual CPS review and prepared several updates including the one that removed the exception from Section 4.2.4. It was not published earlier because the update was bundled together with other changes in one revision.

We did not file an incident because removal of the DNS operator exception was identified and acted upon well ahead of the deadline. The CPS update was also started before SC46 became effective. We regret it was not published prior to the effective date. 

Timeline
2021-04-29 - Engineers are made aware of need to remove the operator exception
2021-05-04 - Code changes removing the exception from CA systems where it was allowed are proposed
2021-05-05 - CPS edits are proposed to remove the DNS Operator exception as required by SC46
2021-05-13 - Code change becomes effective and use of the operator exception is technically restricted by all of our systems
2021-07-12 - SC46 becomes effective
2021-08-11 - The GTS CPS covering removal of the operator exception is publicly published 

Brett L
Google Trust Services

Andrew Ayer

unread,
Sep 2, 2021, 11:07:27 AM9/2/21
to Brett L, dev-secur...@mozilla.org
On Wed, 1 Sep 2021 14:21:47 -0700 (PDT)
Brett L <off...@google.com> wrote:

> Hi Andrew,
>
> Thank you for the questions and checking on details.
>
> We removed the option to use the DNS operator exception from our
> secondary CA platform on 2021-05-13 (60 days before the ballot
> changes went into effect, see timeline below). Our primary CA
> platform has never used it.
>
> In May, we conducted our annual CPS review and prepared several
> updates including the one that removed the exception from Section
> 4.2.4. It was not published earlier because the update was bundled
> together with other changes in one revision.
>
> We did not file an incident because removal of the DNS operator
> exception was identified and acted upon well ahead of the deadline.
> The CPS update was also started before SC46 became effective. We
> regret it was not published prior to the effective date.

The inaccurate CPS is a compliance violation, even if the DNS
operator exception was not in use.

This is the same basic scenario as
<https://bugzilla.mozilla.org/show_bug.cgi?id=1706967> - in that case,
GTS' domain validation practices were compliant with the BRs, but GTS'
CPS did not accurately reflect GTS' practices.

It's troubling to see a recurrence of that scenario, and your email
doesn't provide much of an explanation how this happened.

1. Why was the CPS change not published in May given that
<https://bugzilla.mozilla.org/show_bug.cgi?id=1706967> made clear the
importance of having an accurate CPS?

2. When SC46 was merged into the BRs, did the automation described in
<https://bugzilla.mozilla.org/show_bug.cgi?id=1706967#c11> create an
internal ticket for it? If not, why not? If it did, how did GTS respond
to the ticket and why did the response not detect the CPS non-conformance?

Regards,
Andrew

Brett L

unread,
Sep 3, 2021, 6:34:47 PM9/3/21
to dev-secur...@mozilla.org, Andrew Ayer, dev-secur...@mozilla.org, Brett L

Hi Andrew,

As mentioned earlier the CPS update including this change was not published earlier because the update was bundled together with the annual CP/CPS update. As you know, GTS also had multiple incidents open around the time of these changes, so we were engaging a number of additional partner teams and stakeholders for reviews with the goal of ensuring we covered all updates and improvements in one pass. Wider participation in the update helped highlight additional areas to simplify and clarify language, but this also took additional time to complete. 

We flagged SC46 as an action item on 2021-04-09 and the change to our CPS was made in draft form a few days later (2021-05-13). Though the automation in question was in active development at the time, it was not completed. When SC46 was published a few weeks later (2021-06-02), the review process for requirement changes identified it as having been addressed because at the time both the associated code changes had been deployed and the CPS was in the process of being approved for release. 

Though we always try to land documentation changes with corresponding code changes sometimes it is simply not always feasible.  That being said upon review we feel the 30 day delay that was introduced as a result of the bundling of other changes together is too long and have now filed an incident report 1729097 to capture this event.

As for https://bugzilla.mozilla.org/show_bug.cgi?id=1706967 it was different in a few ways. The first is that Mozilla Root Store Policy under section 2.2 (3) has a clear expectation that "The CA's CP/CPS must clearly specify the procedure(s) that the CA employs, and each documented procedure should state which subsection of 3.2.2.4 it is complying with." In that particular case, the CPS included a forbidden validation method. Additionally we had missed that the change in numbering had not been reflected in our CPS. In this case, the changes to remove the operator exception were complete, the documentation was in progress and our tracking and automation tooling covered both. The second difference is that 4.9.1.1.7 obligated us to revoke the certificate because although our practices were conformant with current regulations, the CPS no longer had a reference to an allowed domain control verification scheme.

Brett L
Google Trust Services


Brett L

unread,
Sep 10, 2021, 3:58:35 PM9/10/21
to dev-secur...@mozilla.org, Brett L, Andrew Ayer, dev-secur...@mozilla.org

Hi Andrew,

While preparing our incident report, we reviewed all previous communications and noticed a small typo in our previous message. The SC46 action item was flagged on 2021-04-29, rather than 2021-04-09.

We apologize for any confusion this may have caused.

Brett L
Google Trust Services

Andrew Ayer

unread,
Sep 13, 2021, 11:21:50 AM9/13/21
to Brett L, dev-secur...@mozilla.org
On Fri, 3 Sep 2021 15:34:47 -0700 (PDT)
Brett L <off...@google.com> wrote:

> Though we always try to land documentation changes with corresponding
> code changes sometimes it is simply not always feasible.

The above statement is concerning, because section 2.2 of the Baseline
Requirements state:

"Section 4.2 of a CA's Certificate Policy and/or Certification Practice
Statement SHALL state the CA's policy or practice on processing CAA
Records for Fully-Qualified Domain Names; that policy shall be consistent
with these Requirements."

If a CA doesn't update their CP/CPS at the same time as deploying
changes to their CAA policy, they will be out of compliance with the
Baseline Requirements, as GTS was for 90 days.

I was expecting the incident response in
https://bugzilla.mozilla.org/show_bug.cgi?id=1729097 to address
the problem by providing a solution for ensuring GTS' CP/CPS remain
up-to-date with GTS' actual policies and practices. Instead, GTS's
response is to ensure CPS changes are published by the time of the
compliance deadline that motivated the change. This would not have
prevented GTS' non-compliance, just limited it to 60 days. (It would,
however, have prevented their non-compliance from being externally
detectable.)

GTS' response also won't address CP/CPS changes that aren't motivated
by BR/MRSP changes, such as adding another CAA Issuer Domain Name.
It appears that with GTS' current process for CP/CPS changes, such a
change could be delayed by up to a year if it gets "bundled" with GTS'
next annual CPS update. To reiterate, this would be a violation of BR
section 2.2. What is GTS doing to make it feasible for them to comply
with this requirement moving forward?

Regards,
Andrew

Ben Wilson

unread,
Sep 16, 2021, 12:58:05 PM9/16/21
to Andrew Ayer, Brett L, dev-secur...@mozilla.org

On August 25, 2021, we began a three-week public discussion[1] on GTS’ request to replace 5 GTS root CA certificates.[2] (Step 4 of the Mozilla Root Store CA Application Process[3]). 

Summary of Discussion and Completion of Action Items [Application Process, Steps 5-8]:  

On 31-Aug-2021, Andrew Ayer noted that the DNS operator exception to performing CAA had been eliminated by Ballot SC46 of the CA/Browser Forum (passed on 2021-06-02 and effective on 2021-07-12), but that it had not been removed from GTS' CPS until 2021-08-11.[4]  GTS responded that it had removed the option to use the DNS operator exception from its CA platform in May 2021, but acknowledged that it had not published its updated CPS until August--because it was including the SC46 update as part of more comprehensive CPS update and manual review-and-approval processes took longer than expected.

As a result, GTS opened up a bug in Bugzilla, #1729097[5], and filed an incident report to explain that the above processes had resulted in the delay in publishing an updated CPS. In the bug, GTS has agreed to: (a) update its processes to require that a tracking bug for compliance documentation revisions include a machine-parseable publication deadline related to the effective date of any requirements changes (and require that tracking bugs be linked to any related software changes); (b) implement, by Friday, 2021-09-17, a proactive alert of impending deadlines for documentation updates; and (c) double-check, by 2021-09-24, all ballots since SC3, and to ensure that all required changes have been made and are accurately reflected in its CPS.[6]  I believe these remediation steps are adequate and can be implemented relatively easily. I also believe they address Andrew’s concerns that GTS might continue to bundle CP/CPS changes into an annual review and not ensure that its CPS would remain up-to-date with its actual policies and practices. Any further discussion on this issue can take place in Bug #1729097.

Close of Public Discussion and Intent to Approve [Application Process, Steps 9-10]:  

GTS seeks only to replace existing CA certificates, and I conclude that in this circumstance, public discussion does not need to be placed on hold or extended further. Thus, this is notice that I am closing public discussion (Application Process, Step 9) and that it is Mozilla’s intent to approve GTS’ request to replace these five CA certificates (Step 10). 

This begins a 7-day “last call” period for any final objections.

Thanks,

Ben

[1] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/lAu1_S48RAA/m/vVnldsU4BAAJ

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1675821

[3] https://wiki.mozilla.org/CA/Application_Process

[4] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/lAu1_S48RAA/m/p-51xlT5BQAJ

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1729097

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1729097#c1


--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/20210913112012.dfd4bf784a3f58d2a45f2172%40andrewayer.name.

Brett L

unread,
Sep 17, 2021, 11:30:31 AM9/17/21
to dev-secur...@mozilla.org, bwi...@mozilla.com, Brett L, dev-secur...@mozilla.org, Andrew Ayer

Hi Andrew,

Thank you for your question and for asking for clarification. We have provided an update to https://bugzilla.mozilla.org/show_bug.cgi?id=1729097 that we hope addresses your question.

We have also started a new discussion on this list about how to better handle the synchronization between CA software and CPSes, since delays have occurred in the past for others in the ecosystem. You can see that thread at https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/BFk3q1R5gbM.


Brett L
Google Trust Services

Ben Wilson

unread,
Sep 29, 2021, 4:17:14 PM9/29/21
to dev-secur...@mozilla.org
All,
I should have noted during public discussion that GTS was seeking enablement of both the websites and the email trust bits for the roots involved. I am re-opening public comment for another week until October 6 to gather any comments about this aspect of GTS' request.
Thanks,
Ben

Rob Stradling

unread,
Nov 30, 2021, 7:39:54 PM11/30/21
to Ben Wilson, Kathleen Wilson, Clint Wilson, ryand...@google.com, ryan....@gmail.com, ry...@sleevi.com, dev-secur...@mozilla.org
On August 25th 2021, Ben Wilson wrote:
> This is to announce the beginning of the public discussion phase for Google Trust Services' (GTS) request to replace five existing root CA certificates with ones that were re-signed last year, August 13, 2020.  See https://wiki.mozilla.org/CA/Application_Process#Process_Overview, (Steps 4 through 9).
> The five roots are as follows:  GTS Root R1, GTS Root R2, GTS Root R3, GTS Root R4, and the GlobalSign ECC Root CA - R4.  (A sixth root CA certificate, the GlobalSign Root CA - R2, was re-signed using SHA1, and so I have removed it from this inclusion request.) The reason for their replacement is that the original CA certificates do not contain the digitalSignature key usage bit, which is required for direct OCSP signing by the CA. (See https://bugzilla.mozilla.org/show_bug.cgi?id=1652581)

Ben, Kathleen, Clint, Ryan, Ryan, and Ryan,

I realize that I'm posting this message quite some time after the end of the 3-week comment period.  Sorry about that, but I hope you'll agree that this needs to be discussed...

Sectigo recently announced a plan (see bug 1741777) to replace several root certificates for the same reason cited by GTS (i.e., to add the digitalSignature bit), but in https://bugzilla.mozilla.org/show_bug.cgi?id=1741777#c3 Ryan Sleevi wrote "I don't believe the proposed approach actually remediates the compliance issue".  The substance of that comment seems to apply just as much to GTS as to Sectigo, but it doesn't appear to have been discussed in relation to GTS's root replacement plan (either in this list thread or in any of the relevant GTS bugs on Bugzilla).

Since remediating the compliance issue is the sole reason that GTS are replacing their root certificates, it seems to me that GTS, the root program owners, and the wider community, should consider the potential ramifications of https://bugzilla.mozilla.org/show_bug.cgi?id=1741777#c3 on GTS's root replacement plan right away, before the December 2021 Batch of Root Changes is finalized.

Sectigo has paused its root replacement plan, pending the outcome of this discussion.

From: dev-secur...@mozilla.org <dev-secur...@mozilla.org> on behalf of Ben Wilson <bwi...@mozilla.com>
Sent: 25 August 2021 22:26
To: dev-secur...@mozilla.org <dev-secur...@mozilla.org>
Subject: Public Discussion of Google Trust Services' Request to Replace Root CA Certificates
 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

Ben Wilson

unread,
Nov 30, 2021, 9:06:58 PM11/30/21
to Rob Stradling, Kathleen Wilson, Clint Wilson, ryand...@google.com, ryan....@gmail.com, ry...@sleevi.com, dev-secur...@mozilla.org
Thanks, Rob, for raising this issue here for discussion on the list. From a root program perspective, I think we will want to narrowly scope whatever we consider and decide. I'd also like to start a new discussion thread for the issues narrowly outlined therein.  I'll attempt to paraphrase the issue(s) in that new discussion thread, which I'll get distributed shortly.
Thanks again,
Ben
Reply all
Reply to author
Forward
0 new messages