Replacement of GTS' GlobalSign Root CA – R2

[Ben Wilson]

All,

This matter involves a replacement of the GlobalSign Root CA - R2 (the “R2 Root”). The R2 Root is currently owned by Google Trust Services (“GTS”).  The R2 Root was originally signed in 2006 using SHA1WithRSA and has a validity period of 12/15/2006 to 12/15/2021. Information about the original CA certificate for the R2 Root can be obtained here:  https://crt.sh/?id=14.  The R2 Root still supports a number of legacy clients.

Of note, this original root CA certificate had key usage bits for Certificate Signing and CRL Signing (hexadecimal 06), but not the digitalSignature key usage bit. For OCSP signing, a CA certificate needs to have the digitalSignature key usage bit, along with the other two key usage bits (hexadecimal 86). (Section 7.1.2.1.b of the Baseline Requirements states, “If the Root CA Private Key is used for signing OCSP responses, then the digitalSignature bit MUST be set.”)

In order to comply, last year, GTS re-signed the R2 Root with the digitalSignature key usage bit. See https://crt.sh/?id=3448659678  It did the same with five other root CA certificates. The only changes made were for that one keyUsage bit and the certificate serial number.

We are currently processing an inclusion request for these GTS replacement CA certificates (https://ccadb-public.secure.force.com/mozilla/PrintViewForCase?CaseNumber=00000666). However, of note, in re-signing the R2 root, GTS used the SHA1WithRSA algorithm because that is the signature algorithm that was used for the original CA certificate. The other new CA certificates were re-signed with their respective signature algorithms (which happened not to be SHA1WithRSA). Whether this re-signing is a violation of section 5.1.3 of the Mozilla Root Store Policy (MRSP) has been alluded to. https://bugzilla.mozilla.org/show_bug.cgi?id=1652581#c22. As most of you know, policies applicable to signing with SHA1 have changed overtime.

GTS would like a decision that section 5.1.3. allows this type of root CA certificate modification because if it is not allowed, then there is no other practical solution to address new requirements for older roots that support legacy clients that might have trouble with SHA2.

Here is the relevant language from MRSP section 5.1.3:

CAs MAY sign SHA-1 hashes over intermediate certificates which chain up to roots in Mozilla's program only if the certificate to be signed is a duplicate of an existing SHA-1 intermediate certificate with the only changes being all of:

·        a new key (of the same size);

·        a new serial number (of the same length);

·        the addition of an EKU and/or a pathlen constraint to meet the requirements outlined above.    …

CAs MUST NOT sign SHA-1 hashes over other data, including CT pre-certificates.

Note:  Section 7.1.3 of the Baseline Requirements previously stated, “This Section 7.1.3 does not apply to Root CA or CA cross certificates. CAs MAY continue to use their existing SHA-1 Root Certificates.”  Section 7.1.3.2.1 with similar language to MRSP section 5.1.3 above was added last year after GTS had re-signed the R2 Root, but at the time it complied with the Baseline Requirements.

While MRSP section 5.1.3 addresses the use of SHA1 with end entity certificates and intermediate CA certificates, it does not expressly address certificate modifications for roots. (The re-signed root also happens to be an intermediate CA certificate under the MRSP because it “chains up to a root in Mozilla’s program”.  However, the third bullet of the exception above is only for an EKU or pathlen constraint and not for a change in the key usage bit.) It has been raised that the phrase “CAs MUST NOT sign SHA-1 hashes over other data” precludes the use of SHA1 when signing anything beyond what is allowed by section 5.1.3, including the modification of a root certificate. This is an issue of interpreting the meaning of "other data" in the context of MRSP section 5.1.3.

Still, with the foregoing in mind, I am currently inclined to continue processing the GTS inclusion request with this SHA1-signed certificate for the R2 Root because (1) it would only replace the existing SHA1-signed certificate; (2) GTS was attempting to resolve an issue with one of the requirements using a “minimally invasive” approach (i.e. only changing the key usage bit and the serial number); (3) the new CA certificate is still valid only until December 15, 2021; (4) the public key has not changed and browser root stores rely mainly on the public key and not the signature algorithm, which is one of the factors for why SHA1 roots have been allowed to remain in root stores; and (5) applying the MRSP in this manner would be consistent with the then-existing Baseline Requirements that did not apply to roots.

What are your thoughts?

Thanks in advance,

Ben

[Andrew Ayer]

On Wed, 7 Apr 2021 21:37:09 -0600
Ben Wilson <bwi...@mozilla.com> wrote:

> It has been raised that the phrase "CAs MUST
> NOT sign SHA-1 hashes over other data" precludes the use of SHA1 when
> signing anything beyond what is allowed by section 5.1.3, including
> the modification of a root certificate.

This is indeed what it means, and it's critical for security that
the list in 5.1.3 be treated as an exhaustive list of the types
of data which CAs can sign with SHA-1. This is because *any* type
of data could potentially be collided to forge a certificate when
the data is signed with an insecure hash algorithm. Therefore, the only
safe thing to do is enumerate the types of data that are known to be safe
to sign with SHA-1, and forbid everything else.

See here for a proof-of-concept which collided OCSP responses
with certificates, which motivated the wording in 5.1.3:
https://www.mail-archive.com/dev-secur...@lists.mozilla.org/msg02999.html
(that proof-of-concept used MD5 because SHA-1 collision attacks were
not yet practical at the time, but they are now: https://shattered.io/)

See also the discussion from the last time this question was raised:
https://www.mail-archive.com/dev-secur...@lists.mozilla.org/msg11767.html

Although *this particular case* (resigning an existing root and changing
only the key usage bits) does not pose a security risk, and I would
support a policy change to permit it, it's not OK for CAs to ignore rules
when they think it's safe to do so. We've seen too many examples of CAs
making very bad risk assessments, including thinking that using SHA-1
is safe when it really isn't (e.g. signing precertificates with SHA-1).

For the above reasons, signing this root was a non-compliant use of the
root's key and should be treated as an incident involving GTS'
*existing* root. GTS should provide an incident report so that the
community can be assured that GTS has appropriate safeguards to ensure
compliance with Mozilla's policies. This doesn't necessarily preclude
the inclusion of the replacement root, as other roots have been
included despite compliance incidents with the CA, but the inclusion
should be put on hold pending a satisfactory outcome of the incident
report.

Regards,
Andrew

[Nick Lamb]

On Wed, 7 Apr 2021 21:37:09 -0600
Ben Wilson <bwi...@mozilla.com> wrote:

> GTS would like a decision that section 5.1.3. allows this type of
> root CA certificate modification because if it is not allowed, then
> there is no other practical solution to address new requirements for
> older roots that support legacy clients that might have trouble with
> SHA2.

Like Andrew, I don't agree that this makes sense. I agree that in fact
what GTS did here isn't a big problem and needn't result in rejecting
this modification or other measures, but I don't think 5.1.3 allows
this as it stands and we should encourage CAs to *ask first* rather
than relying on the principle that it's easier to seek forgiveness.

What particularly jumped out at me was: Why do we care about "legacy
clients that might have trouble with SHA2" ?

Is the implication here that since Firefox now doesn't implement SHA1
there are GTS systems merrily issuing new SHA1 certificate under these
roots to "support legacy clients" ? That seems like a terrible idea.

If not, what is the concern? Maybe I'm slow here and need somebody to
spell out how a "legacy client that might have trouble with SHA2"
benefits from us allowing this.

Nick.

[Ben Wilson]

All,

I am gathering from Andrew's comment that GTS should file an incident report, but that it would be acceptable to include the new R2 Root in an inclusion request. From Nick's comment, maybe the reference to legacy clients was my own overstatement of the rationale. In any event, I am getting ready to announce public discussion of the six roots, so I suppose further discussion on the R2 root can occur during the public discussion phase.  Am I right?

Thanks,

Ben

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/20210411041352.63b3826a%40totoro.tlrmx.org.

[Andy Warner]

All, we appreciate the comments and want to address some issues raised thus far.

Before taking this action, GTS consulted with root program contacts for all the root programs we participate in, notifying them of our intent to modify the affected roots in this way. In these communications, we stressed the fact that GS R2 is very tenured and important to keep active for the many devices that do not update their roots, as there were questions about why we were not simply retiring the root. We, perhaps mistakenly, believed the implications of modifying a SHA1 root were clear, but perhaps they were not. Only after the fact were these issues raised. After extensive discussion, Mozilla and GTS agreed that getting community input made sense, thus Ben's post. 

We are glad to see that at least some community members see a need for policy modification. There is no such thing as an exception in this space, but we're hoping that there is agreement that it makes sense to modify existing policy to allow this narrow use case. In retrospect, it seems clear that wider community input would have been beneficial here, but we want to be clear that GTS did not take unilateral action and had consulted a number of key community stakeholders before conducting the root modification ceremony.

Finally, it looks like there may be a bit of confusion about the extent to which SHA1 signing is being discussed here. GTS does not issue SHA1 leaf certificates and never will. A single modification event for a tenured root was the only use of SHA1. Sadly, there are many embedded devices that do not update their root stores and will eventually break when the last of the roots they support hit their end of life. We cannot avoid these devices eventually breaking, but keeping tenured roots valid until their natural expiration delays older devices breaking.

The issues covered here are detailed in the existing incident report (https://bugzilla.mozilla.org/show_bug.cgi?id=1652581) and the replacement tracking bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1675821) if the community feels another report focused specifically on the modification is warranted, we will happily provide one.

[Ryan Sleevi]

On Mon, Apr 19, 2021 at 7:34 PM 'Andy Warner' via dev-secur...@mozilla.org <dev-secur...@mozilla.org> wrote:

All, we appreciate the comments and want to address some issues raised thus far.

Before taking this action, GTS consulted with root program contacts for all the root programs we participate in, notifying them of our intent to modify the affected roots in this way.

Hi Andy,

(In a Google capacity) I don't see any such e-mail from GTS to our chrome-root-au...@google.com alias. To help make sure there's no configuration issues on our end, could you clarify when you sent it and what the subject was? I'm only finding emails related to the new GlobalSign cross-signs, which is what led to https://github.com/cabforum/servercert/issues/179 being filed and fixed.

In these communications, we stressed the fact that GS R2 is very tenured and important to keep active for the many devices that do not update their roots, as there were questions about why we were not simply retiring the root. We, perhaps mistakenly, believed the implications of modifying a SHA1 root were clear, but perhaps they were not.

(In a personal capacity) It sounds like you're saying "We asked Mozilla for an exception, even if we didn't call it out as an exception, and because they didn't say no, we assumed they meant yes". There's a lot to unpack there, so perhaps if that's not what you meant, you could clarify.

As called out in https://wiki.mozilla.org/CA/Responding_To_An_Incident , while this is not an official policy, failure to follow it without good reason can absolutely affect the general opinion of the CA, which I think is what we're seeing here: consequences of a decision. Why didn't GTS file an incident report, before it had performed the action, to solicit feedback from Mozilla and the community, if it was aware that the plan would violate Mozilla policies? GTS is expected to already have familiarity with this process, given the discussion of COVID-19 audit delays leading to incident reports being filed (which GTS pre-emptively did), and this has come up in the past, regarding discussions like underscores. Did GTS believe it, as a CA, was somehow exceptional, or this circumstance was somehow exceptional? If so, why?

As the bug discussion captures, there seems to be no ambiguity this was forbidden, and these changes had already been incorporated in the Baseline Requirements, so GTS should also have been freshly familiar with them, in the process of reviewing the changes to the BRs as part of SC31, which happened less than a month before GTS went and violated the BRs (and Mozilla's requirements). Further, if the argument is that there is no option, that also seems demonstrably false, because that same change made OCSP optional for sub-CAs, which would have obviated the need to add the digitalSignature bit to the root. Even still, assuming these changes are ignored, GTS could have issued an OCSP responder certificate from that root (which lacked digitalSignature).

So if the position is that GTS was in a rock and a hard place, that's also false. Had GTS publicly communicated its plans and concerns, and detailed their own assessment, this might have been addressed. As it stands, we're now forced to depend on GTS' word that they considered all options, and didn't simply take unilateral action, as all (public) signs indicate.

It appears that GTS' position is "We sent an email" (which cannot be verified by the public) and "No one pointed out our problems" (which tries to shift the burden to whoever GTS mailed, rather than GTS itself), which it acknowledges that it itself didn't really call out that it was aware of this being problematic.

Finally, it looks like there may be a bit of confusion about the extent to which SHA1 signing is being discussed here. GTS does not issue SHA1 leaf certificates and never will. A single modification event for a tenured root was the only use of SHA1. Sadly, there are many embedded devices that do not update their root stores and will eventually break when the last of the roots they support hit their end of life. We cannot avoid these devices eventually breaking, but keeping tenured roots valid until their natural expiration delays older devices breaking.

I haven't seen anything to support your claim of confusion here, which is troubling, because it appears this is an attempt to dismiss the legitimate concerns being raised. This also hints at something "could go wrong", but factually, GTS has to date failed to provide any evidence to support this claim.

More specifically: Can you give a precise technical explanation why SHA-1 was necessary for this new signature, as opposed to being able to use SHA-256? I'm not aware of any technical reasons, across any client, and so it'd be very useful and educational to know more here.

The issues covered here are detailed in the existing incident report (https://bugzilla.mozilla.org/show_bug.cgi?id=1652581) and the replacement tracking bug (https://bugzilla.mozilla.org/show_bug.cgi?id=1675821) if the community feels another report focused specifically on the modification is warranted, we will happily provide one.

I'm not sure if, at this point, we'll find much useful information, but I do think it's worthwhile to record this as an incident report, because I do find this a troubling incident from GTS. It's unfortunate that this needs to be publicly stated, but in the effort of transparency, it's essential to highlight that these incidents unfortunately demonstrate a clear lack of technical precision and candor in GTS' operation, a theme that seems to be carried through on this message. https://bugzilla.mozilla.org/show_bug.cgi?id=1652581#c1 and https://bugzilla.mozilla.org/show_bug.cgi?id=1652581#c18 are both examples of GTS having troubling replies that are only corrected on follow-up, which I worry that most any reply to this message will, by definition, be more of the same.

Or, for greater concern, consider https://bugzilla.mozilla.org/show_bug.cgi?id=1652581#c23 was set, and there has still been zero public follow-up by GTS, unless we consider this thread (started by Ben, not GTS) to be follow-up, 7 months later, a marked departure from the Incident Reporting guidelines "and in no circumstances should a question linger without a response for more than one week". That's... not a positive sign for any CA.

If you feel these are unfair characteristics that unfairly raise concerns about GTS, an Incident Report is a useful way to provide a precise timeline with the relevant technical details. I fully acknowledge, I may have overlooked things here, and so it's a useful way to "correct the permanent record", which is useful when considering a set of CA incidents to do so as contemporaneously as possible.

You might feel this is making a mountain out of a molehill, or, as your message seems to indicate, that this incident itself "is not a big deal, and we did a reasonable effort". I think both of those conclusions would be false. I think the concern here is GTS' seeming lack of awareness of the existing requirements (both of Root Programs and the BRs), GTS' lack of transparency in handling and responding to incidents, GTS' lack of candor, transparency, and technical thoroughness in their replies, and GTS' lack of follow-up. These are all worrying patterns associated with CAs that the community has lost trust in, and so it's very reasonable to be deeply concerned about the behaviours and patterns demonstrated here, even if the actual technical issue may (luckily, and exceptionally in this case), turned out to probably not be that risky.

However, we don't trust nor allow CAs to unilaterally or non-transparently make those decisions, precisely because the majority of the time, the CA is wrong in their risk and solution analysis. And, unfortunately, this seems to be another example of the same: GTS being wrong in its solution analysis, at least based on the information provided.

To the extent this has bearing on the new root inclusion, it seems GTS is in significant need of an overhaul of its incident reporting process and in improving transparency. It's not clear that GTS is keeping aware of changes in both policies and CA/B Forum, it's not clear that GTS is prioritizing transparent incident reporting, and it's very clear that GTS' more recent incident reports have significantly devolved in quality and thoroughness, both compared to other CAs (including CAs operated by other Root Program vendors), and relative to their own past incidents. There are plenty of examples of "good" incident reports, but I would struggle to think of GTS even approaching that, so perhaps that's a concrete goal and measurable community commitment that GTS could make here, with a practical near-term demonstration being a report to the community about how its historically handled incidents and what it's doing to improve that.

[Andrew Ayer]

On Tue, 20 Apr 2021 11:33:41 -0400
Ryan Sleevi <ry...@sleevi.com> wrote:

> I'm not sure if, at this point, we'll find much useful information,
> but I do think it's worthwhile to record this as an incident report,
> because I do find this a troubling incident from GTS. It's
> unfortunate that this needs to be publicly stated, but in the effort
> of transparency, it's essential to highlight that these incidents
> unfortunately demonstrate a clear lack of technical precision and
> candor in GTS' operation, a theme that seems to be carried through on
> this message. https://bugzilla.mozilla.org/show_bug.cgi?id=1652581#c1
> and https://bugzilla.mozilla.org/show_bug.cgi?id=1652581#c18 are both
> examples of GTS having troubling replies that are only corrected on
> follow-up, which I worry that most any reply to this message will, by
> definition, be more of the same.

I agree that this has been a theme with GTS. GTS' communications often
contain dismissals, misrepresentations, or misdirections. I often get
the vibe that GTS doesn't believe that their policy violations are a
problem.
https://www.mail-archive.com/dev-secur...@lists.mozilla.org/msg12613.html
and
https://www.mail-archive.com/dev-secur...@lists.mozilla.org/msg12633.html
are more examples of appalling responses from GTS.

> To the extent this has bearing on the new root inclusion, it seems
> GTS is in significant need of an overhaul of its incident reporting
> process and in improving transparency. It's not clear that GTS is
> keeping aware of changes in both policies and CA/B Forum, it's not
> clear that GTS is prioritizing transparent incident reporting, and
> it's very clear that GTS' more recent incident reports have
> significantly devolved in quality and thoroughness, both compared to
> other CAs (including CAs operated by other Root Program vendors), and
> relative to their own past incidents. There are plenty of examples of
> "good" incident reports, but I would struggle to think of GTS even
> approaching that, so perhaps that's a concrete goal and measurable
> community commitment that GTS could make here, with a practical
> near-term demonstration being a report to the community about how its
> historically handled incidents and what it's doing to improve that.

+1, and I believe that all GTS root inclusion requests should be put
on hold until this is resolved in a satisfactory manner.

Regards,
Andrew

[Andrew Ayer]

On Mon, 19 Apr 2021 16:34:53 -0700 (PDT)

"'Andy Warner' via dev-secur...@mozilla.org"
<dev-secur...@mozilla.org> wrote:

> We are glad to see that at least some community members see a need
> for policy modification.

This appears to be a gross misrepresentation, as I do not see any
emails in this thread that address policy modification besides my own,
and my email very clearly did not address necessity, but rather the
security risk.

My support for a policy modification was based on an assumption, made
in the interest of charity, that GTS had accurately determined that
there was a compelling technical need to use SHA-1 instead of SHA-256.
In hindsight, I should have waited to see public evidence from GTS. I
regret my oversight, and I withdraw my support for amending policy to
permit this.

Regards,
Andrew

[Ryan Hurst]

Let me start with an apology for the delayed response to this thread. I took some time off in April to decompress from the constant work that we have all experienced due to Covid-19, and since back, I have been trying to address a set of urgent matters that came up while I was out including this one.

Given the tone of the thread, I had asked the team to let me respond but overestimated my ability to get to it with the urgency the thread deserved. The fact that I am a critical failure point in the communication process is a problem in itself, and something I will talk more about below.

This is not an excuse, and I want to make it clear that we understand we need to do better in communicating with this community and you have our commitment that we will.

Since there are several topics in play right now, my thinking is that it is best to move the discussions of each into the respective Bugzilla issues. This should enable information related to each to be more easily tracked. For others, not already following in Bugzilla, those issues are:

- digitalSignature KeyUsage not set (https://bugzilla.mozilla.org/show_bug.cgi?id=1652581)

- Failure to provide regular and timely incident updates (https://bugzilla.mozilla.org/show_bug.cgi?id=1708516)

- Forbidden Domain Validation Method 3.2.2.4.10 ( https://bugzilla.mozilla.org/show_bug.cgi?id=1706967)

- Signing SHA-1 Hash for existing CA certificate with changes in Key Usage (https://bugzilla.mozilla.org/show_bug.cgi?id=1709223)

You will notice I have created a third incident to cover the SHA-1 usage in the re-issuance of ‘GlobalSign Root CA - R2’, which was the thread's original topic. I have tried to include the complete picture of what transpired and how the decision was made.

Hopefully, this will allow us to make progress as a community on how to tackle each of these issues thoughtfully and correctly. 

As I look at this thread and the incidents in question, I see several areas for Google Trust Services to improve, and I thought I would talk a little about those here.

First, broadly at Google, we see automation as a force multiplier, a way to improve the quality of our services and make things more reproducible. This is why we focus heavily on automating as much as possible, including in non-engineering activities like auditing. For example, we have built extensive automation around streamlining our audits with the hope of moving to a world of being able to continually demonstrate our compliance with ease rather than the current state where the industry relies on point-in-time assessments. 

Where Google Trust Services, and me in particular, have had a blind spot, is that we have failed to utilize automation to augment how we interact with the root programs and the communities supporting the WebPKI. The monitoring of these communities is something we believe we can automate, for example by polling Bugzilla and m.d.s.p for conversations relating to our services we can back manual participation with tracking in our internal systems. This will ensure a quicker response and that issues do not fall between the cracks in a manual process. I believe this will help address the latency in communication issues called out by Andrew.

Second, I believe that communications should be public and transparent, however, I do use individual conversations by phone, video, email, or text with many here to form my opinions on topics but have failed to reflect the outcomes of these communications on the list. 

To that end, I want to stress to Mozilla and the members of this community, moving forward we will make a concerted effort to come to this list with issues much earlier and not just rely on the list as a communication channel for conversations related to incidents.

Third, just like we have an on-call rotation for security and operational issues, we also have what can be thought of as an on-call rotation for communication with Root Programs and this list on behalf of Google Trust Services. As people's life situations change and their scope of responsibilities evolve, we have failed to grow this on-call rotation, which has contributed to the rushed responses and delays in response.

To remedy this, we will be increasing the number of people involved in these communications, so we are no longer so dependent on a small number of individuals. We hope that this will improve the quality of our communication as well as the speed at which you hear from us. 

Thanks for your patience,

Ryan Hurst

Product Management

Google Trust Services