Public Discussion of GoDaddy cross-signing two Certainly Intermediate Certificates

[Brittany Randall]

This is to announce and begin public discussion of GoDaddy’s intent to use its publicly trusted Starfield Root Certificate Authority - G2 (https://crt.sh/?caid=796) to create two new external subordinate CA certificates to be operated and maintained by Certainly, LLC.  These will be cross-certificates sharing their respective key pairs with subordinate CA certificates signed by two Certainly Root CAs that are pending inclusion (https://bugzilla.mozilla.org/show_bug.cgi?id=1727941).

In accordance with Mozilla Root Store Policy, Section 8 - CA Operational Changes for new program participants and at the instruction of Process for Review and Approval of Externally Operated Subordinate CAs we have created Bugzilla Bug 1755851 and are initiating this formal discussion period.

Certainly is a wholly owned subsidiary of Fastly, Inc., a cloud service provider headquartered in the USA. Certainly plans to issue certificates to existing Fastly customers. The two Certainly subordinate CAs will issue publicly-trusted DV TLS server certificates. More details may be found in Certainly’s root inclusion case in CCADB. Certainly has performed a CA Compliance Self-Assessment and has committed to adhere to all Mozilla requirements, Baseline Requirements of the CA/Browser Forum, and the GoDaddy (Starfield Technologies) CP/CPS.

All the operational services related to Certainly’s Subscribers will be performed by Certainly, including processing of certificate applications, certificate issuance, certificate publishing, certificate status services, and certificate management. Certainly has implemented the open-source Boulder CA and interacts with Applicants and Subscribers via an ACME API endpoint.  Certainly has applied for inclusion as a root CA to Mozilla and a number of other root store programs, requesting inclusion of two root certificates. Both will be used exclusively to issue DV TLS certificates, with the distinction that one root will anchor an RSA hierarchy and the other will anchor an ECDSA hierarchy. These roots, as well as the two corresponding subordinate CAs that are constrained to TLS usages, have been disclosed in CCADB.

Certainly has received the following unqualified audit reports (see Bug 1755851 for full reports) from the WebTrust Practitioner, Schellman, LLC:

• WebTrust for CAs point-in-time dated June 30, 2021
• WebTrust SSL Baseline with NCSSRs point-in-time dated June 30, 2021
• WebTrust for CAs Key Lifecycle Management report (covering the period between key generation and type-1 audits)

Certainly will undergo WebTrust for CAs and WebTrust SSL Baseline with NCSSRs period-of-time audits no later than June 30, 2022, covering a period beginning July 1, 2021. Certainly has further committed to ongoing WebTrust audits for the 10-year lifetime of the cross-signed certificates.

As operator of a Mozilla-trusted root CA (and a trusted root in other browser root store programs), we recognize that through this cross-sign event, we are ultimately accountable for any actions taken by the Certainly intermediates which will inherit our trust and have worked closely with Certainly to perform due diligence activities including the review of the Certainly CP/CPS, Subscriber Agreement, and Relying Party Agreement against CA/B forum requirements, GoDaddy Policies, and Mozilla policies. We have also reviewed Certainly’s CA Compliance Self-Assessment and operational practices, interviewed Certainly personnel, and reviewed the external audit opinions to verify appropriate scope of coverage and conformance with requirements as expected. Currently and following the proposed cross-sign event, we will continue working closely with Certainly to oversee ongoing compliance efforts.

Of note, Certainly has filed two Mozilla incident reports to date (listed below) which we have followed and reviewed with Certainly. It is our expectation that the second bug be resolved prior to any cross-sign event.

• Root CRL validity period exceeds maximum by one second  (27-September 2021)
• TLS Using ALPN TLS Version and OID (27-January 2022)

This email begins a 3-week comment period, after which Mozilla is expected to consider approval of GoDaddy’s request.

Best,

Brittany Randall

[Ryan Sleevi]

On Thu, Feb 17, 2022 at 1:10 AM 'Brittany Randall' via dev-secur...@mozilla.org <dev-secur...@mozilla.org> wrote:

This email begins a 3-week comment period, after which Mozilla is expected to consider approval of GoDaddy’s request.

My individual position would be that GoDaddy does not perform this signing unless, and until, https://bugzilla.mozilla.org/show_bug.cgi?id=1727941 is resolved with a positive disposition.

I think there's wide agreement and understanding that the security risks to the community, and to end users, remains, at minimum, the same whether introducing a root or an intermediate, as they both have the same capability. Cross-signing an intermediate versus adding a root largely only functionally differs in that rather than Mozilla and the community reviewing the CA-to-be-signed, the cross-signer performs that review. I don't think we have reason to be confident in the general case that CAs are effective at upholding the same principles or thoroughness as the current Mozilla process was designed to engage and promote. I don't mean this as a slight against GoDaddy, but rather the general principle.

The public review policy for intermediates was designed to address this gap, by ensuring intermediates and roots undergo the same level of public review and consideration. It applies for two cases:

1) Where the CA-to-be-signed does not plan to operate as a root

2) Where the CA-to-be-signed does plan to operate as a root (as is the case here)

Strangely, 1) is a bit academic, because there's very little compelling reason to pursue that path given the above considerations, so it's not surprising that we find ourselves in 2).

However, this means that the cross-sign review period is, in effect, a way to "jump the queue", as it were, bypassing the priorities applied to CA review in order to get to a state where certificates work with Mozilla products (and broadly, the ecosystem). It's difficult to imagine a scenario where a positive dispensation is given to a cross-signing, but then later rescinded for the root; and, in that case, one would expect (as has happened in the past), that the cross-sign itself would then be prohibited as well.

On the upside, the suggestion here is that GoDaddy has reviewed Certainly to a level that they are satisfied, and given the substantial risk that GoDaddy would be embracing, it might be reasonable to allow the queue jump on the principle that GoDaddy is vouching for Certainly. On the downside, there's likely a form of remuneration involved here, so it may simply be a "pay for play", except that GoDaddy is the beneficiary for such scheme.

It's possible for both of these things to be true, and I don't mean to suggest that they are morally correct or incorrect, but it's worth careful evaluation when considering the next steps. Certainly is not yet to the public discussion phase, indicating that there are still a number of factors to occur, including the detailed CP/CPS review. To try to "do this right" would suggest that this process be compressed, and where normally there would be three weeks of discussion *after* a lengthy discussion with the CA in question (Certainly), that this all happen within three weeks. Alternatively, it's to suggest that these things are not actually essential for CA Root Inclusion, since it would be granting Certainly the same privileges without these having occurred.

I say this as someone who suspects that Certainly is in good hands, especially given the stakeholders involved, which notably includes former Mozilla CA Certificate Policy Owner Wayne Thayer. I realize that, in the absence of a cross-sign, the widespread interoperability is not yet achievable. However, it seems to be a feature, not a bug, to ensure a process of multi-stakeholder review and comment occurs, and that's why I think coupling the cross-sign to the dispensation of https://bugzilla.mozilla.org/show_bug.cgi?id=1727941 is an ideal outcome. It does mean that the benefit of a cross-sign is reduced from "skips the root inclusion phase" into one of "updates quicker than a new Firefox release, and works with those that haven't yet updated", but that's still a hugely valuable outcome, and hopefully not too unreasonable a burden.

From a prioritization perspective, it might be reasonable to consider "Another CA has expressed a willingness to cross-sign" as a reason to prioritize a given CA higher, precisely because it's a CA willing to risk their reputation for the to-be-signed entity, and thus we can expect some degree of self-interested due diligence. I have no objections to doing that in this case, and further discussion later for any gotchas about making that a general policy principle. But that would still suggest some period longer than three weeks, to allow for the detailed information gathering and CP/CPS reviews, and to treat Certainly as any other root from a policy/process standpoint.

Knowing Wayne's familiarity with policy and practices, I suspect this would be a hopefully very streamlined process, as there have been for other CAs (e.g. Amazon Trust Services) in which long-standing mdsp participants were involved in the design and establishment. That said, we also have counter-examples, in which CAs with long-standing participants repeatedly demonstrate failures to adhere to the minimum expectation, so it's not a guarantee, just a suspicion :)

[Wayne Thayer]

Ryan,

I understand your concern in principle. However, what you are proposing is at the very least a major new requirement, and could be interpreted as being in direct conflict with the existing Process for Review and Approval of Externally Operated Subordinate CAs that are Not Technically Constrained [1], which (as you described) states:

When a root CA operator intends to sign a subCA certificate for use by an external, third-party operator that has not previously undergone this review process for the type of certificate that it will be authorized to issue, the root CA operator is responsible for collecting information and performing due diligence similar to that performed by Mozilla on root inclusion requests (i.e. Quantifying Value and other steps not outlined in this wiki page would not be required), and for initiating the public discussion on the Mozilla dev-security-policy mailing list

You will recall participating in the discussion of this policy in 2021 [2], at which time some similar ideas were suggested by others. I acknowledge that Mozilla has the right to perform whatever due diligence they deem appropriate to protect users; however, the decision to do so should not be arbitrary. What you are proposing amounts to a retroactive policy change.

I suspect that Mozilla would consider your proposal as part of the related policy 2.8 discussions [3] [4]. That would be an appropriate forum to work out the details and then update the policy for externally operated subCAs.

Wayne

[1] https://wiki.mozilla.org/CA/External_Sub_CAs_not_Technically_Constrained

[2] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/AA5G1bzOwZQ/m/mZBO_IIcBQAJ

[3] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/r9ZeggoH9KA/m/NZwE3TuYAwAJ

[4] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/zBGXSngpwWw/m/aIm4sKXyAAAJ

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.
To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAErg%3DHG0ZYEGzVj40XA7c%3DVjjztfj-_qGzOA8BpNFK3n0aQbmw%40mail.gmail.com.

[Ryan Sleevi]

On Thu, Feb 17, 2022 at 2:52 PM Wayne Thayer <wth...@gmail.com> wrote:

You will recall participating in the discussion of this policy in 2021 [2], at which time some similar ideas were suggested by others. I acknowledge that Mozilla has the right to perform whatever due diligence they deem appropriate to protect users; however, the decision to do so should not be arbitrary. What you are proposing amounts to a retroactive policy change.

I'm not sure that it would amount to a retroactive policy change? Doesn't that conclusion assume that a positive dispensation should be automatically granted?

From that same policy document:

Following public discussion, the Mozilla CA Program Manager will determine whether the subCA operator will be accepted, and update the corresponding CCADB record to indicate the result.

and

After a minimum of 3 weeks have passed, a Mozilla representative will announce a one-week “last call” for objections. Mozilla may determine to extend public discussion, or approve or reject the subCA operator.

I'm suggesting that it would be prudent to either reject, or, alternatively, extend, public discussion until the two processes are in sync.

I'm not trying to suggest GoDaddy has done anything wrong in starting this process; as you highlight, it's following the established policy. But I'm suggesting that given the lack of broader context at this time (e.g. the information gathering and review, the detailed CP/CPS assessment) may be sufficient reason to consider holding off accepting. And I did try to capture that if the decision is to accept the sub-CA, at this time, then it's functionally no different than accepting the root CA without those processes being completed.

[Ben Wilson]

All,

Here are some of my thoughts.

The current policy allows cross-signing of externally operated CAs. What GoDaddy proposes to do with Certainly is the current practice.

In this case, Certainly happens to have an application to have its root trusted directly in the root store, which is actually a better scenario than one in which the cross-signed external operator does not intend to seek direct inclusion in the root store.

The wiki page (https://wiki.mozilla.org/CA/External_Sub_CAs_not_Technically_Constrained) clarifies how public discussion is to occur. It can be improved as needed, but it does not contemplate that the regular application for root inclusion should delay the process.

There are two different trust decisions and processes presented here. Certainly will still have to meet all of the steps that other CA operators are required to meet for root inclusion, although some of these tasks will be closer to completion due to the fact that Certainly is going through this process with GoDaddy. I suppose we can review the prioritization criteria and see whether anything needs amendment.

The announced three-week period should be sufficient for people to more closely examine Certainly, to ask questions, and to bring comments forward regarding whether they believe Certainly is an appropriate entity to operate a subCA under GoDaddy's root.

Thanks,

Ben

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAErg%3DHHXiw7ienF%2BJ%3D0wnka5nusYryZ7ZqqNkco8o8Sev1MQ5g%40mail.gmail.com.

[Dimitris Zacharopoulos]

On 17/2/2022 5:50 μ.μ., Ryan Sleevi wrote:

On Thu, Feb 17, 2022 at 1:10 AM 'Brittany Randall' via dev-secur...@mozilla.org <dev-secur...@mozilla.org> wrote:

This email begins a 3-week comment period, after which Mozilla is expected to consider approval of GoDaddy’s request.

My individual position would be that GoDaddy does not perform this signing unless, and until, https://bugzilla.mozilla.org/show_bug.cgi?id=1727941 is resolved with a positive disposition.

I think there's wide agreement and understanding that the security risks to the community, and to end users, remains, at minimum, the same whether introducing a root or an intermediate, as they both have the same capability. Cross-signing an intermediate versus adding a root largely only functionally differs in that rather than Mozilla and the community reviewing the CA-to-be-signed, the cross-signer performs that review. I don't think we have reason to be confident in the general case that CAs are effective at upholding the same principles or thoroughness as the current Mozilla process was designed to engage and promote. I don't mean this as a slight against GoDaddy, but rather the general principle.

I believe this is not correct. The security risks to the community are different because in the cross-signing/externally-operated subCA model, the Subordinate is under the close supervision of the signing (already publicly-trusted) CA, compared to the individual Root which stands on its own.

The public review policy for intermediates was designed to address this gap, by ensuring intermediates and roots undergo the same level of public review and consideration. It applies for two cases:

1) Where the CA-to-be-signed does not plan to operate as a root

2) Where the CA-to-be-signed does plan to operate as a root (as is the case here)

Strangely, 1) is a bit academic, because there's very little compelling reason to pursue that path given the above considerations, so it's not surprising that we find ourselves in 2).

There are cases where a CA wants to operate independently, without any third-party supervision. This is the most common scenario that's why option 2 is what we usually see. However, there are cases where a company wants to operate a subCA under the guidance and supervision of another CA, which uses option 1. CCADB includes cases that prove that both options exist today.

This is not to say that the trust impact of an externally-operated Sub CA vs a Root CA are not similar, but it is reasonable to assume that when an already-trusted CA member of this community (in good standing)

-  accepts the liability and reputational risks,
-  places sufficient controls to monitor the operations/compliance program of an externally-operated SubCA,

it is sufficient for the community to accept. Taking it one step further, I would say that this scenario (operating under the guidance/supervision of an already-trusted Root CA) is actually MORE trustworthy than an independent Root being accepted for which the community has no visibility about internal operations/compliance program apart from an audit report and self-evaluation.

I assume that GoDaddy and any CA that plans to cross-sign a CA that is new to this community, performs a similar analysis/due diligence as Ben and the community does in Root inclusion requests. Three weeks should be sufficient for additional review/scrutiny by any other member but if more time is needed, people can just ask.

For this specific request, besides your good comments about Wayne and his familiarity with existing policies, I will add that Fastly is a very active Interested Party member in the CA/Browser Forum Server Certificate WG with lots of contributions and not just from Wayne.

I believe it would be unfair to delay this application more than the current Mozilla Policy dictates on the grounds that there is a pending Root inclusion request by Certainly. They could just as well not have applied for a Root inclusion.

Dimitris.

[Ryan Sleevi]

On Fri, Feb 18, 2022 at 12:40 AM Ben Wilson <bwi...@mozilla.com> wrote:

There are two different trust decisions and processes presented here.

Can you clarify what you see these two different trust decisions as being? In particular, can you clarify what differences there are for Mozilla users?

The announced three-week period should be sufficient for people to more closely examine Certainly, to ask questions, and to bring comments forward regarding whether they believe Certainly is an appropriate entity to operate a subCA under GoDaddy's root.

Is three weeks sufficient to review a root request, from end to end? This ties back to the above remark about understanding what difference there is for end users.

If anything, the cross-signing action suggests greater risk, because the potential of greater “blast radius,” as it were, should issues arise, as they would implicate GoDaddy, and require addressing those risks. This is, in effect, the “too big to fail” problem, and thus makes meaningful remediation more complex, rather than providing greater assurance.

It’s useful to understand if I’m overlooking some dimension here in terms of risks to end users between a root vs a sub-CA. As best I can tell, the argument is that the rather than the normal community evaluation of roots, the community should defer and trust the issuing CA to have done that. I think it’s fairly easy to understand why I have reservations about such a claim, much in the same way we do not allow CAs to delegate domain validation.

[Ryan Sleevi]

On Fri, Feb 18, 2022 at 1:36 AM Dimitris Zacharopoulos <ji...@it.auth.gr> wrote:

I believe this is not correct. The security risks to the community are different because in the cross-signing/externally-operated subCA model, the Subordinate is under the close supervision of the signing (already publicly-trusted) CA, compared to the individual Root which stands on its own.

In the event it was not clear, I very much disagree with this position, which is why I do not think it advisable to proceed at present.

1. Root CAs have significantly less experience reviewing the inclusion of CAs, including CP/CPS reviews.
2. Root CAs are primarily focused on their perceived risks, which are indirect proxies for user risk, as opposed to being directly focused on user risks
3. Root CAs do not generate the same degree of transparency, artifacts and records as the public review process.
1. For example, we have no such record of a detailed CP/CPS review (although, to the prior two points, it's questionable to what extent it would be aligned with community needs)
2. We have no discussion and/or transparency as to Root CAs' processes for conducting such reviews
3. We have no transparency into any commitments and/or representations made
4. In practice, the risk is greater, not less, because any issues now present two CAs to remediate, rather than one.

If we accept the conclusion that the risks are different, then similarly, it would seem that any restrictions upon Super CAs, or any concerns with Trust Service Lists, are unfounded, because they cannot be reasonably held positions when delegation of subordinate signing is seen as less risky. In all three cases, it works out the same: the delegation of security functions to external parties. While in theory, differences can be suggested ("Super CAs aren't as experienced", "the TSL isn't accountable"), those arguments don't stand up under scrutiny.

At fundamental question is whether the goal of a Root Program is to ensure a thorough supervision and vetting of all CAs in the program, in order to protect from user risk. If that is the argument, then it stands to reason that subordinate CAs from new entities should not be introduced unless and until that vetting has occurred, to the same level of detail, and with the same processes in place to ensure the right result. To do otherwise is to effectively secure a loan of "trust" based on collateral that is exceedingly difficult (technically and policy) to repossess if mistakes are made, and relying on some undocumented, opaque assurance that due dilligence has occurred. If we were trying to develop real estate in New York, that might be the norm, but this isn't the Big Apple.

 

There are cases where a CA wants to operate independently, without any third-party supervision. This is the most common scenario that's why option 2 is what we usually see. However, there are cases where a company wants to operate a subCA under the guidance and supervision of another CA, which uses option 1. CCADB includes cases that prove that both options exist today.

While this is true, exceeding from the incident reports has been an active discouragement for option 1, because it does not, in practice, work. The past decade has been very much towards Option 2 as the only option, which is exactly why unaudited sub-CAs were sunset, why technically constrained CAs are still required to be audited by some root programs (independent of/despite the BRs carve outs), and overall, why cross-signing is being limited in favor of direct integration of the root.

 

This is not to say that the trust impact of an externally-operated Sub CA vs a Root CA are not similar, but it is reasonable to assume that when an already-trusted CA member of this community (in good standing)

-  accepts the liability and reputational risks,
-  places sufficient controls to monitor the operations/compliance program of an externally-operated SubCA,

It's reasonable to state it's reasonable, but it's not actually reasonable. The suggestion here of reputational risks is somewhat suspect, given that certificates are undifferentiated fungible commodities, in which reputation plays little overall effect in the selection. The extent reputational concern is an issue is developing a "bad reputation" that might lead to distrust, and that's precisely why the risk is greater, because it's an activity that carries with it profoundly high risk. For example, do we believe that it would at all be a labor-free endeavour to contemplate removing trust in GoDaddy if Certainly routinely violated requirements? I think the significance of GoDaddy's extant operations in the broader Internet at large would, if anything, suggest that Certainly would have far more opportunities to make mistakes that are overlooked, precisely because of the desire to avoid distrusting GoDaddy.

The second point here is that we should assume controls are sufficient, when we have shown repeatedly, through a variety of CA incidents, the lack of controls. To that end, GoDaddy's discussion has failed to even detail the controls, such as those learned from other CAs, and based on GoDaddy's past incidents regarding being unaware of other CAs' remediations, it seems unlikely to just assume those controls exist. However, even if controls existed, the degree of transparency around those controls and their routine functioning doesn't exist.

Would it be better if GoDaddy regularly engaged in Detailed Control Reports, required those of Certainly, and made both of those publicly available? Sure, that might be a path towards a sufficient degree of transparency. But we won't see that within the next few weeks, and that still fits in the broader question about what it should look like for all CAs.

Neither of these points holds up under scrutiny, and equally, they don't hold up with made generic across all CAs. To the extent we cannot assume that the small CA with only 5,000 certificates is capable and knowledgeable to fully audit new entrants, we shouldn't assume the CA with 50 million certificates is any more, because they are different skills than traditional issuance.

 

it is sufficient for the community to accept. Taking it one step further, I would say that this scenario (operating under the guidance/supervision of an already-trusted Root CA) is actually MORE trustworthy than an independent Root being accepted for which the community has no visibility about internal operations/compliance program apart from an audit report and self-evaluation.

I hope I have put to bed how flawed this conclusion is, but since sometimes, the horse must be beaten to glue for the argument to stick: by this logic, it would suggest that the ecosystem would be better without any root review, and instead delegate to a set of Super CAs, with each Super CA accountable to the browser. I think most folks would recognize that as the antithesis of the direction we should go in, but that's logically where a conclusion of MORE trustworthy takes things.

 

I assume that GoDaddy and any CA that plans to cross-sign a CA that is new to this community, performs a similar analysis/due diligence as Ben and the community does in Root inclusion requests. Three weeks should be sufficient for additional review/scrutiny by any other member but if more time is needed, people can just ask.

There is no basis for that assumption, nor data to support it.

 

I believe it would be unfair to delay this application more than the current Mozilla Policy dictates on the grounds that there is a pending Root inclusion request by Certainly. They could just as well not have applied for a Root inclusion.

The goal is to protect users, not "fairness". Transparent, consistent processes, aligned with the policy objectives, but that does not mean a dispensation towards assuming inclusion, or that "If you did the work, and got an audit, you should be trusted". This process is one of risk management, and that cannot and should not be abdicated.

Had Certainly not applied for a Root Inclusion, I would be actively encouraging that the request be rejected entirely, precisely for the same reasons here: the risk of a subordinate is indistinguishable to end users from that of a root, while the risk to the process and policies is even greater. The march has been towards containing that risk, and, consistent with concerns raised about delegating trust to third-parties such as governments, not one that should be lightly abdicated, least of all because "of a misguided goal to fairness over security. 

[Dimitris Zacharopoulos]

I can't see significant differences in terms of transparency between a new Root and an externally-operated subCA.

At the very least, in the new Root CA case, you know there is no external oversight by another CA. In the externally-operated subCA we assume there is some external oversight by the signing CA, which is "better than nothing"? I agree that more details of this oversight process would be useful -if not necessary- to be at least shared with the community.

My understanding is that users are protected equally by both routes:
- new Root CA, where at least a Root Store Manager performs the initial due diligence,
- externally-operated subCA, where the issuing CA performs the due diligence.

In both cases the community has the same amount of time to review the applications (at least 3 weeks) and dig into the CP/CPS and audit reports.

Dimitris.

[Peter Bowen]

On Sun, Feb 20, 2022 at 9:07 AM Ryan Sleevi <ry...@sleevi.com> wrote:
>
> The goal is to protect users, not "fairness". Transparent, consistent processes, aligned with the policy objectives, but that does not mean a dispensation towards assuming inclusion, or that "If you did the work, and got an audit, you should be trusted". This process is one of risk management, and that cannot and should not be abdicated.
>
> Had Certainly not applied for a Root Inclusion, I would be actively encouraging that the request be rejected entirely, precisely for the same reasons here: the risk of a subordinate is indistinguishable to end users from that of a root, while the risk to the process and policies is even greater. The march has been towards containing that risk, and, consistent with concerns raised about delegating trust to third-parties such as governments, not one that should be lightly abdicated, least of all because "of a misguided goal to fairness over security.

I fully agree that we should not prioritize fairness over security,
but I have to disagree that this should be rejected if there was not
already a root inclusion request. If the security risk was so great,
then why does Mozilla policy not simply prohibit new Externally
Operated Subordinate CAs that are Not Technically Constrained? Eight
months ago, in July 2021, the last time this topic came up, two
different people suggested simply prohibiting the practice[1][2].
However Mozilla decided to not do so and instead opened a Policy 2.8
topic on the process to approve issuance [3] and updated the Mozilla
wiki in December 2021.

While I hope we can all agree that revisiting policy is necessary as
we learn new things and technology changes, I do not think that
anything notable has changed in the last two to eight months that
suggests this request represents a new security risk compared to what
could be reasonably envisioned when this was last discussed.

I do not think this request, or other requests for a new Externally
Operated Subordinate CA, should be rejected or accepted based on
whether the CA operator is applying for inclusion of a root CA they
operate.

Thanks,
Peter

[1] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/AA5G1bzOwZQ/m/-L7Q_bdABQAJ
[2] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/AA5G1bzOwZQ/m/mS5rard2BQAJ
[3] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/zBGXSngpwWw/m/aIm4sKXyAAAJ

[Matt Palmer]

On Sun, Feb 20, 2022 at 10:04:50PM +0200, Dimitris Zacharopoulos wrote:
> My understanding is that users are protected equally by both routes:
> - new Root CA, where at least a Root Store Manager performs the initial due
> diligence,
> - externally-operated subCA, where the issuing CA performs the due
> diligence.

That assumes that the quality of the initial due diligence is the same in
both cases. Can you explain why you believe that to be the case?

> In both cases the community has the same amount of time to review the
> applications (at least 3 weeks) and dig into the CP/CPS and audit reports.

For a root inclusion request, the module owner (IIRC) does a deep-dive on
the CP/CPS before the discussion period.

- Matt

[Ryan Sleevi]

On Sun, Feb 20, 2022 at 3:06 PM Peter Bowen <pzb...@gmail.com> wrote:

I do not think this request, or other requests for a new Externally
Operated Subordinate CA, should be rejected or accepted based on
whether the CA operator is applying for inclusion of a root CA they
operate.

This conclusion may, unintentionally, seem to suggest something I was not proposing. I'm hoping this was simply trying to be carefully worded to limit the scope of what you agree with, although it may be misinterpreted as being a rephrasing of my position, which it isn't.

The past discussions you link to are with respect to a different scenario than the one we find ourselves in now, and so too were the proposals different. Note that I'm not suggesting that subordinate CAs be prohibited, as the previous posters suggested, but rather that such subordination be placed on hold unless, and until, the CA has undergone adequate review.

This speaks to Dimitris' point, or perhaps misunderstanding, about the root inclusion process. The suggestion of there being simply a three week review process overlooks the significant, and transparent, vetting that occurs on the CCADB Case and Bugzilla issue prior to acceptance, including, as has been previously mentioned, the detailed CP/CPS review by someone who regularly performs CP/CPS reviews, and with a vested interested towards protecting users. The incentives, process, and outcomes are all radically different with respect to subordination, and yet the risks are, at best, the same, or as previously highlighted, even greater than those risks of a root (due to shared fate).

I'm not sure if you share Dimitris' position that the quality, competenancy, and outcomes of an externally-delegated review are commiserate with those of one performed by the root inclusion process. If they are distinct, and have differences in competencies and outcomes, then I'm not sure why the externalized subordinate CA review would be seen as acceptable, except for "tradition". Yet that tradition is both inconsistent with the goals, and inconsistent with the other practices, and so I do believe it's worth suggesting here to break from that misguided tradition, and to defer the acceptance or rejection of the subordinate until the root has been completed. Unless we're assuming that the policy aims are to guarantee inclusion, this seems entirely consistent with the existing policies and practices.

[Peter Bowen]

On Sun, Feb 20, 2022 at 5:29 PM Ryan Sleevi <ry...@sleevi.com> wrote:
>
>
>
> On Sun, Feb 20, 2022 at 3:06 PM Peter Bowen <pzb...@gmail.com> wrote:
>>
>> I do not think this request, or other requests for a new Externally
>> Operated Subordinate CA, should be rejected or accepted based on
>> whether the CA operator is applying for inclusion of a root CA they
>> operate.
>
>
> This conclusion may, unintentionally, seem to suggest something I was not proposing. I'm hoping this was simply trying to be carefully worded to limit the scope of what you agree with, although it may be misinterpreted as being a rephrasing of my position, which it isn't.
>
> The past discussions you link to are with respect to a different scenario than the one we find ourselves in now, and so too were the proposals different. Note that I'm not suggesting that subordinate CAs be prohibited, as the previous posters suggested, but rather that such subordination be placed on hold unless, and until, the CA has undergone adequate review.

I was responding to your statement:

"Had Certainly not applied for a Root Inclusion, I would be actively
encouraging that the request be rejected entirely, precisely for the
same reasons here: the risk of a subordinate is indistinguishable to
end users from that of a root, while the risk to the process and
policies is even greater. The march has been towards containing that
risk, and, consistent with concerns raised about delegating trust to
third-parties such as governments, not one that should be lightly
abdicated, least of all because "of a misguided goal to fairness over
security."

This seemed to indicate that you were recommending that requests for a
new externally operated subordinate CAs only be approved if the
operator of the new subordinate has already applied for root
inclusion. Is this not accurate?

Thanks,
Peter

[Filippo Valsorda]

2022-02-17 01:10 GMT-05:00 'Brittany Randall' via dev-secur...@mozilla.org <dev-secur...@mozilla.org>:

This is to announce and begin public discussion of GoDaddy’s intent to use its publicly trusted Starfield Root Certificate Authority - G2 (https://crt.sh/?caid=796) to create two new external subordinate CA certificates to be operated and maintained by Certainly, LLC.  These will be cross-certificates sharing their respective key pairs with subordinate CA certificates signed by two Certainly Root CAs that are pending inclusion (https://bugzilla.mozilla.org/show_bug.cgi?id=1727941).

At a high level, I can't personally see why the process for approving a new externally-operated unconstrained cross-sign should be less onerous and thorough than the process to include a new root. Allowing externally-operated cross-signs is beneficial in that it allows new CAs to bootstrap without crippling ubiquity issues, but there is no value in it being a shortcut in the inclusion process. In both cases the first-order risk to users is the same (if we discount the issuing CA's oversight according to the track record of relying on CAs for self-oversight), and if we consider the ecosystem complexity of remedial actions the cross-sign is in fact riskier, as Ryan points out.

If this cross-sign is approved before the root inclusion is, it should follow that the natural path for a new CA is to 1) file an inclusion request, 2) obtain a cross-sign, and 3) eventually address the concerns raised by the inclusion process but not by the cross-sign discussion. If we believe the inclusion process is not capable of raising additional concerns on top of those of the cross-sign discussion, then the former should be made more lightweight to match the latter. If we believe the inclusion process is capable of raising additional concerns, then the cross-sign bypasses that important part of the process, and renders it moot.

[Scott Helme]

Hi everyone, 

Quick question! Will Certainly be offering a public ACME CA for use by everyone (à la Let's Encrypt), or will this be for Fastly customer use only?

Cheers, 

Scott. 

[Andrew Ayer]

I took a look at parts of Certainly's CP/CPS. I'm happy to see
Certainly use a combined CP/CPS. However, section 3.2.2 says:

"Certainly validates domain control primarily in an automated fashion
using the ACME protocol. In exceptional cases control may be validated
using methods similar to those employed by ACME, but performed
manually."

The BRs permit the ACME methods. The BRs do not permit CA-invented
methods that are "similar to" ACME methods. This is the same as the
forbidden "any other method of confirmation", and relies on trusting
the CA's judgment that the methods "similar to" the allowed methods
are as secure as the allowed methods themselves. It should be
very clear that the BRs no longer permit CAs to make this judgment call.

I agree with Sleevi: Mozilla should not extend trust to a new
organization until it completes the normal, full review process. As
shown by this security-critical omission by GoDaddy, Mozilla cannot
rely on the "due diligence" of another CA to substitute for Mozilla's
own process.

Regards,
Andrew

[Dimitris Zacharopoulos]

On 21/2/2022 3:28 π.μ., Ryan Sleevi wrote:
> This speaks to Dimitris' point, or perhaps misunderstanding, about the
> root inclusion process. The suggestion of there being simply a three
> week review process overlooks the significant, and transparent,
> vetting that occurs on the CCADB Case and Bugzilla issue prior to
> acceptance, including, as has been previously mentioned, the detailed
> CP/CPS review by someone who regularly performs CP/CPS reviews, and
> with a vested interested towards protecting users. The incentives,
> process, and outcomes are all radically different with respect to
> subordination, and yet the risks are, at best, the same, or as
> previously highlighted, even greater than those risks of a root (due
> to shared fate).

I would like to remind people that before Mozilla adopted the great
practice for detailed CP/CPS reviews by its own staff (with the
unquestionable incentives, experience that Ryan mentioned), the Mozilla
community contributed to these CP/CPS reviews. Members of the community,
including people associated with CAs and Browsers, were performing
reviews (perhaps not as detailed as the ones performed during the last 2
years) and technical checks (for example CRLs, OCSP and other "publicly
visible" technical elements).

My point is that we should not outright consider CA reviews as
non-trusted. In fact, any review is useful especially if it is publicly
disclosed. This is also supported in
https://wiki.mozilla.org/CA/Application_Verification#Public_discussion.

If GoDaddy has performed such an analysis in Certainly's CP/CPS, I would
recommend its disclosure to this request so that members can
independently assess. It would also help Ben with his review during the
Root inclusion request process.

[Brittany Randall]

We can provide some of our review documentation. I'll shoot to have something early next week. I'll plan to add any attachments to the bug, but will reply in this discussion to let folks know items are there.

Best,

Brittany

[Wayne Thayer]

Hi Scott,

Certainly touched on this point in our root inclusion CCADB case, which states “Certainly will initially issue certificates to existing Fastly customers. Fastly.serves a broad range of organizations around the world, and also offers free services to open source projects. Certainly only issues DV TLS certificates.” We would like to broaden the reach of our services, but want to take it one step at a time.

- Wayne

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/94f405b3-b189-4e3e-a1f9-67db6233d28an%40mozilla.org.

[Brittany Randall]

Regarding the GoDaddy CP/CPS review of Certainly, we have attached the following review artifacts to Bug 1755851:

• Attachment Compendium.pdf
• CPCPSReviewTracker.xlsx
• CSAReview.zip (contains three files)
• FastlyWebTrustAuditReportReview.zip (contains seven files)

The first document, “Attachment Compendium.pdf” provides details and additional context for the remaining three attachments uploaded. Also, for reference, Certainly has published version 1.3 of the Certainly CP/CPS to https://certainly.com/repository/

Best,

Brittany Randall

[Ben Wilson]

All,

Today I read through the Certainly CP/CPS and reviewed the Compliance Self-Assessment and GoDaddy's review documents. I did not see anything in the CP/CPS that did not conform to the Mozilla Root Store Policy or the CA/B Forum's Baseline Requirements.

I also looked at the GoDaddy-Fastly cross-certificate profiles and did not see anything that concerned me.

The public comment period will close next Wednesday, 9-Mar-2022.  Please provide any additional comments you may have by then.

Yours sincerely,

Ben

--
You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/d73a51c1-5f68-4626-b4a7-ea3643747a19n%40mozilla.org.

[Ryan Sleevi]

Ben:

Did I miss Andrew’s remarks being addressed? Or did you see them not as concerning as we did?

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CA%2B1gtaYTK4SA2h6f3ej8hGifT-7-EyWVaJd-z0nbwE3s%2BFoUCg%40mail.gmail.com.

[Ben Wilson]

Ryan,

Let me compare what I reviewed (CP/CPS dated March 1, 2022) with what Andrew reviewed and get back to you.

Ben

[Ben Wilson]

Ryan,

The language I read states, "Certainly validates domain control primarily in an automated fashion using the ACME protocol."

The other language is no longer there.

Ben

[Wayne Thayer]

In response to the following concern raised by Andrew:

On Mon, Feb 21, 2022 at 11:06 AM Andrew Ayer <ag...@andrewayer.name> wrote:

I took a look at parts of Certainly's CP/CPS.  I'm happy to see
Certainly use a combined CP/CPS.  However, section 3.2.2 says:

"Certainly validates domain control primarily in an automated fashion
using the ACME protocol. In exceptional cases control may be validated
using methods similar to those employed by ACME, but performed
manually."

The BRs permit the ACME methods.  The BRs do not permit CA-invented
methods that are "similar to" ACME methods.  This is the same as the
forbidden "any other method of confirmation", and relies on trusting
the CA's judgment that the methods "similar to" the allowed methods
are as secure as the allowed methods themselves.  It should be
very clear that the BRs no longer permit CAs to make this judgment call.

The CP/CPS goes on to describe specific methods that are used by Certainly. The language Andrew is pointing out is referring to the ACME dns-01, http-01, and tls-alpn-01 implementations. It meant that Certainly  might foreseeably perform a manual 3.2.2.4.7 validation rather than using Boulder’s dns-01 implementation. We’ve never done that, and given the complexity of getting it right I’m confident that we never will. I don’t agree that this is a “security-critical omission”, but I do see how this language is concerning, so it has been removed from the latest version of the CP/CPS that Ben reviewed.

 - Wayne

[Ryan Sleevi]

On Fri, Mar 4, 2022 at 5:07 PM Ben Wilson <bwi...@mozilla.com> wrote:

All,

Today I read through the Certainly CP/CPS and reviewed the Compliance Self-Assessment and GoDaddy's review documents. I did not see anything in the CP/CPS that did not conform to the Mozilla Root Store Policy or the CA/B Forum's Baseline Requirements.

I also looked at the GoDaddy-Fastly cross-certificate profiles and did not see anything that concerned me.

The public comment period will close next Wednesday, 9-Mar-2022.  Please provide any additional comments you may have by then.

Ben,

Do you feel this level of review is appropriate for the risk that is posed to the community, both Mozilla users and those that rely on Mozilla's process?

Mozilla highlights the thoroughness of its reviews, and the importance in providing careful analysis, as part of the reason that it disallows super-CAs and does not blindly accept third-party trust lists. However, it's difficult even for me to discern what functional difference there is between, say, an EUTL, and what this is: a GoDaddy Trust List. If Mozilla is providing unique value above and beyond a cursory review, then it suggests that value needs to be applied equally, because as widely understood, every new CA introduces risk to the ecosystem.

Having worked with Wayne for years, I am sure that, in the long run, this may be a net positive. However, I'm personally uncomfortable with the premise here that this is either lower risk or an appropriate level of review. From Andrew's remarks going unaddressed (until the reping), from a lack of feedback from GoDaddy about their own processes for review, it doesn't seem like things were prepared for total transparency in mind. As a simple example of one of the many ways in which the risk is the same, but this parallel process bypasses those checks, is that there has not been, AFAICT, any quantification of value [1] yet. Is the view that somehow intermediates don't need this?

When we look at risk, I cannot help but look at recent issues with CAs operated by cloud providers and CDNs, and wonder if perhaps they introduce more risk, rather than less. When I look at [2], which is from a different CA, we see a large cloud provider prioritizing "zero impact", by wholly ignoring their compliance obligations and proposing to, effectively, do nothing to remediate. In the past, this sort of approach has lead to CAs being distrusted, but when it's a CA this large, the risk calculus makes having such a frank conversation more difficult, both because the size of the organization and the size of the userbase impacted. It equally demonstrates the challenge of having a CA tightly integrated with a hosting provider, as it may limit the ability to quickly replace certificates, either from the same CA or from a new CA, a situation we also saw with WoSign/StartCom (and Azure China) and which complicated the removal of trust in that CA.While it's good to see Certainly is based on ACME, there's a real risk that the integration, by virtue of only targeting a single user base, may end up a "bespoke" ACME (tied to a particular implementation), and limit and impair agility at the parent organization (Fastly).

Do not get me wrong: I think that the ecosystem has the potential to benefit here. However, I do think that this push for inclusion seems inconsistent with past and recent statements about the value provided by Mozilla through the Mozilla Root Program, and see it difficult to square in consideration for super-CAs and trust lists, in which each individual CA must go through the Root Inclusion Process for consideration. Why would this not apply to new entities and new CP/CPSes, given they are functionally identical?

I have not done a detailed CP/CPS review as I would normally do, in part, because there are not the supporting materials that normally exist by the point that I would do such a thing. I am concerned that the "Self-Assessments" may have moved from a "thing to benefit the CA" and into a "thing the Root Program relies on," and I do worry about that. I do want to make it clear: Despite the thoroughness of my comments here, I did not do my normal deep review here, and I would not rely on this as a "Hey, this is proof that if this is all he found, we're fine" - it's simply a quick scan of concerns:

• 1.3.2 notes that RA functions are not delegated, but the section is titled "Registration Authorities and other delegated third parties". In Section 1.3.5, it notes that "Certainly vendors and service providers that have access to confidential information or privileged systems ...", suggesting that there may be at least some functionality that is delegated to third parties. 1.3.2 of the BRs allows for delegating the RA functions (Section 3.2) to DTPs, but the definition within 1.6.1 of the BRs makes it clearer that a DTP applies to any delegation of function. That is, the BRs allow for either 3.2 functionality (e.g. as supported by 4.2.1 of the BRs) or any other function other than 3.2.2.4/3.2.2.5 (supported by 5.3.7 and 5.4.1 - related to ISMS and not just RA functions)
• 1.4.1 limits the certificates to use with the TLS protocol, and 1.4.2 prohibits any other use. What does this mean for QUIC, which Fastly supports [3]? I realize the extent that QUIC repurposes the TLS handshake, but it is a different protocol [4].
• A brief scan of 1.6.1 shows that there are subtle differences in the definitions between Certainly's CP/CPS and the BRs. That's _not_ problematic in and of itself, but requires careful scrutiny to review that the meaning and obligations are preserved throughout the entire document and are consistent under these modified definitions. 
• 1.6.3's references are unversioned. There are various semantic and syntactic differences in X.509, for example, up to and including the recent changes to the underlying model [5], that these differences may matter, even if there is a "common sense" understanding one might take.
• 3.1.4 gives conflicting advice in terms of understanding the construction of names. With respect to DNs, this is the opportunity to be unambiguous about the name forms employed (presumably, C and CN) and their interpretation - and limits.
• 3.1.6 states "Certainly reserves the right to refuse to include a domain name within a Certificate". This right seems to go beyond those otherwise enumerated by the BRs (e.g. scenarios contemplated within 4.9.1), and so it's unclear when, why, and how this right might be exercised. I anticipate that Certainly wouldn't want to give up this right, even if it's not strictly necessary, but it perhaps bears elaborating why this reservation is seen as necessary and how it might be exercised.
• 3.1.6 seems to suggest that, independent of the UDRP, Certainly may make its own determinations about the right or authorization to use a domain name. That seems suboptimal.
• 3.2.5 appears that Certainly have read the BRs 3.2.5 as [If A then (B, C, and D)], rather than [(If A then {B, C}), and D]. That is, whether the "In addition" in the terminal paragraph joins to the first condition (if the Applicant ... is an organization), or whether it states an procedure that is applicable for all Applicants, and not just those that are organizations. While understandably the BRs can benefit from clarity here, it bears calling out, because as a consequence, they provide no such method to limit accounts that can request certificates, made explicit in 4.2.1
• 7.1 does not list all of the RFC 5280-required fields and extensions within the relevant profiles. This raises the question of whether they are "permitted, but unspecified", "prohibited, but not omitted, violating the CP/CPS", or "prohibited, but omitted, violating 5280". Taking the more generous first interpretation, on the basis of the first sentence in this section, it seems clearer to explicitly document the fields that will be included within the profile, and ensure no fields are included without a corresponding CP/CPS update.
• 7.1 uses a rather long OCSP signing certificate (5 years), without documentation (AFAICT) of the corresponding key protection. Given the description of the tradeoffs of OCSP no-check, it seems like a shorter lifetime reduces risks/concerns.

Yes, there are positives to Certainly, and I do want to acknowledge that:

• 1.4.1 makes it clear that they only issue DV certificates
• 2.1 gives a very clear SLO for Repository services
• 3.1.4 makes it clear that homographs are not a PKI-layer problem
• 7.2 provides a CRL profile that is meaningful

Further, I think there are elements of operational transparency that would be addressed by Certainly. For example, as far as I know, Boulder does not have any "release" process, but rather works through an iterative CI/CD workflow with weekly tags. It's unclear how Fastly tracks changes within Boulder, and how frequently and quickly they are deployed. Obviously, this concern exists in equal measure for non-Boulder CAs (e.g. EJBCA), and we've seen a number of CAs have very poor processes around such management, so it's reasonable to understand how Certainly is doing things different.

I realize this sounds like I'm making a purely process argument, but I do sincerely believe that it would be a net-negative for Mozilla users to continue this intermediate inclusion without the corresponding broader root discussion, and in particular, the discussion of both value and risk, to understand how Certainly is addressing this. The current policies around intermediates were developed both on the perceived risk to the ecosystem, and the actual risk demonstrated by a CA who failed to meet the criteria for inclusion of its root, but which had already obtained and was operating an intermediate. Unless we're willing to say that we're predisposing the conversation and biased in Certainly's favor, then it seems the only way to mitigate that risk is to treat the intermediate signing with the same degree of care and caution as the root, and to accept that any decision about the intermediate is functionally, technically, and procedurally identical to a decision about the root. If we're not willing to say "OK, we can skip all root discussion" based on this thread, then I don't believe it should proceed here. And I don't believe we should skip the root discussion.

[1] https://wiki.mozilla.org/CA/Quantifying_Value 

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1743943

[3] https://www.fastly.com/blog/state-of-quic-and-http3-2020

[4] https://www.fastly.com/blog/quic-is-now-rfc-9000

[5] https://www.itu.int/rec/T-REC-X.501-202110-I!Amd1/en

[Ryan Sleevi]

On Sun, Mar 6, 2022 at 10:23 AM Ryan Sleevi <ryan....@gmail.com> wrote:

<snip> From Andrew's remarks going unaddressed (until the reping), from a lack of feedback from GoDaddy about their own processes for review, it doesn't seem like things were prepared for total transparency in mind.

Apologies Brittany/GoDaddy folks, it looks like Brittany's messages [1][2] got incorrectly flagged as spam by my filters, which only showed up after getting a "You're replying to a thread flagged as spam" notification after sending. Thanks for sharing that documentation.

[1] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/bEnn98Dajzc/m/mZlMimQHBAAJ

[2] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/bEnn98Dajzc/m/rRwhAZpxBQAJ

[Wayne Thayer]

Ryan,

I want to acknowledge that Certainly and GoDaddy are aware of your feedback on the CP/CPS and we plan to use it to make improvements. Thank you.

Regarding the question of staying current with Boulder releases: Certainly automatically tracks new tags and has established a process in which we review the changes and deploy updates on a weekly cadence. In practice this means that Certainly is typically one Boulder release behind Let’s Encrypt. GoDaddy worked with us to understand this process as part of their due diligence.

- Wayne

--

You received this message because you are subscribed to the Google Groups "dev-secur...@mozilla.org" group.
To unsubscribe from this group and stop receiving emails from it, send an email to dev-security-po...@mozilla.org.

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAErg%3DHEJ1g1GA6QnwJ2G5a3cXRL7Jcvu%2Bs4JZbAeZYipQkdXxQ%40mail.gmail.com.

[Ben Wilson]

Pursuant to the procedure outlined in the wiki for externally operated CAs[1], three weeks ago GoDaddy initiated public discussion[2] announcing its intent to issue two externally operated intermediate CAs to Certainly, LLC.[3] 

Background

In evaluating roots for inclusion, we are bound by the Principles of the Mozilla Manifesto[4] and the Mozilla Root Store Policy (MRSP) [5]. We use an open, public, and transparent procedure, and we evaluate the risks to users’ security and privacy.

Under section 8 of the MRSP (v.2.7.1), “All CAs whose certificates are included in Mozilla's root program MUST notify Mozilla if: … an organization other than the CA obtains control of an unconstrained intermediate certificate (as defined in section 5.3.2 of this policy) that directly or transitively chains to the CA's included certificate(s).”  The procedure for performing the review of externally operated subordinate CAs was published in the wiki[1] relatively recently. Previously, there had only been a requirement to notify Mozilla.  

The above-referenced procedure makes it clear that the “Quantifying Value” step and other steps not outlined in the procedure are not required. Such additional steps are reserved for when we process a root inclusion request.

Summary of Discussion

Ryan Sleevi argued that Mozilla's approval of the two intermediate CAs should be withheld until Certainly had been approved as a root CA operator pursuant to its pending root inclusion application.[6] His primary argument has been that the security risks to the community are equal under either case and that the approval of Certainly allows it to "jump the queue" and avoid some of the due diligence that would otherwise be required (e.g. CP/CPS review and a risk-vs-value justification).  

Wayne Thayer on behalf of Certainly responded that Ryan was asking for a process that would be new and different from the existing process.

Ryan responded that accepting the sub CAs would be "functionally no different than accepting the root CA without those processes being completed."

I responded that two different trust decisions were involved – one was trusting the root(s) and the other being to allow GoDaddy to issue the sub CAs to Certainly.  

Dimitris Zacharopoulos argued that the situation at hand involves an existing CA operator (i.e. GoDaddy), which had performed due diligence and was taking on liability and reputational risks and was placing controls to monitor the compliance.  He also argued that procedurally, it would be unfair to delay the application more than the current Mozilla Policy dictates.

Ryan responded that the risks are the same for Mozilla users, and that the level of vetting and review should be the same under either scenario. He argued that security of users should outweigh the "fairness" considerations. He also said, "Transparent, consistent processes, aligned with the policy objectives, but that does not mean a dispensation towards assuming inclusion, or that 'If you did the work, and got an audit, you should be trusted'. This process is one of risk management, and that cannot and should not be abdicated."[7]  In that list posting, Ryan also raised the following four points:

1.       Root CAs have significantly less experience reviewing the inclusion of CAs, including CP/CPS reviews.

2.       Root CAs are primarily focused on their perceived risks, which are indirect proxies for user risk, as opposed to being directly focused on user risks

3.       Root CAs do not generate the same degree of transparency, artifacts and records as the public review process.

1.       For example, we have no such record of a detailed CP/CPS review (although, to the prior two points, it's questionable to what extent it would be aligned with community needs)

2.       We have no discussion and/or transparency as to Root CAs' processes for conducting such reviews

3.       We have no transparency into any commitments and/or representations made

4.  In practice, the risk is greater, not less, because any issues now present two CAs to remediate, rather than one.

Matt Palmer argued that initial due diligence by a Root CA operator would not be as thorough as one done by the module owner.

In support of the argument that Mozilla cannot rely on the due diligence of the root CA operator, Andrew Ayer pointed to a statement in the Certainly CP/CPS concerning domain control validation ("In exceptional cases control may be validated using methods similar to those employed by ACME, but performed manually.")  That sentence was subsequently deleted from the Certainly CP/CPS.

Brittany Randall, on behalf of GoDaddy, provided documentation of the due diligence performed by GoDaddy. (See Comment 9 in Bug 1755851.)

An updated CP/CPS was posted Certainly, and I reviewed that CP/CPS, the Compliance Self-Assessment, and GoDaddy's review documents and found that they conformed to MRSP and BR requirements.

Ryan asked whether the level of review conducted was appropriate for the risk posed to the community, both Mozilla users and those that rely on Mozilla's process. He argued that GoDaddy's ability to issue external CAs was the equivalent of a "trust list" or "Super CA". He asked whether a value quantification needed to be performed. He also noted problems with the Certainly CP/CPS (which Certainly and GoDaddy have since committed to address). In conclusion, Ryan stated, "Unless we're willing to say that we're predisposing the conversation and biased in Certainly's favor, then it seems the only way to mitigate that risk is to treat the intermediate signing with the same degree of care and caution as the root, and to accept that any decision about the intermediate is functionally, technically, and procedurally identical to a decision about the root. If we're not willing to say 'OK, we can skip all root discussion' based on this thread, then I don't believe it should proceed here. And I don't believe we should skip the root discussion."[8]

Close of Public Discussion and Intent to Approve Request

Mozilla appreciates the many who have reviewed the documentation and made comments on this matter. Mozilla is open to suggestions that will improve the process going forward. However, we currently see two distinct processes – the root inclusion process and this new process for approving externally operated unconstrained subordinate CAs. While the risks presented by a root and an intermediate may be very similar, they are not identical. For example, a root certificate can issue other CA certificates, which introduces the risk that a CA certificate will be mis-issued. While the intermediate CA certificates proposed here cannot be used to issue CA certificates, and can only be used to issue DV server certificates. They can also be easily distrusted by adding them to OneCRL, unlike a root, which must be removed from the trust store. Therefore, we are not inclined to require that Certainly complete the root inclusion process before moving forward as a sub CA.

This is notice that I am closing public discussion and that it is Mozilla’s intent to approve GoDaddy's request for the reasons explained below. (Certainly will still need to complete the root inclusion process and provide a risk-vs-value justification for inclusion of its roots.)

Certainly has been audited under the WebTrust for CAs principles and criteria[9]. GoDaddy has conducted thorough policy, compliance, and risk management due diligence related to the cross-signing of the Certainly sub CAs.[10] Because GoDaddy is putting its own root at risk, such reviews have gone beyond the minimum requirements of Mozilla and the CA/Browser Forum's Baseline Requirements. GoDaddy's well-documented onboarding of Certainly has included multiple reviews of its audit reports, risk assessments, CP/CPS, training materials, and other documentation. GoDaddy has conducted a CP/CPS alignment review to compare the Certainly CP/CPS with its own CPS and has tracked changes to be made in both parties' CP/CPSes to not only ensure consistency, but to also address the comments that have been received during this public discussion process. GoDaddy's risk management department will conduct ongoing oversight of Certainly's compliance with requirements and hold quarterly touch-point meetings with Certainly. GoDaddy will also be responsible for ensuring that the CCADB is kept up to date with current audits and CP/CPSes for the subCAs, and its Governance and Policy Committee (GPC) will also review updates to the Certainly CP/CPS to ensure ongoing compliance.

As noted in the above-referenced procedure[1], this begins a 7-day “last call” period for any final objections.

Thanks,

Ben

[1] https://wiki.mozilla.org/CA/External_Sub_CAs_not_Technically_Constrained

[2]  https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/bEnn98Dajzc/m/32NwZHWSAAAJ

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1755851

[4] https://www.mozilla.org/en-US/about/manifesto/details/

[5] https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/policy/

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1727941

[7] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/bEnn98Dajzc/m/61wX1hOiAQAJ

[8] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/bEnn98Dajzc/m/58uqrxV7AgAJ

[9] https://bugzilla.mozilla.org/attachment.cgi?id=9264323

[10] https://bugzilla.mozilla.org/attachment.cgi?id=9266002

To view this discussion on the web visit https://groups.google.com/a/mozilla.org/d/msgid/dev-security-policy/CAPh8bk_nUp_F9TB35AtwNZ8%2Bi%3Df66g1y3jvGNS%2BQfUy_j9b7_Q%40mail.gmail.com.

[Ryan Sleevi]

Hi Ben,

I appreciate the clarity on the decision, but I do believe that the rationale and justification of it are inconsistent - internally to the message itself, with the public discussion, and with Mozilla's stated values, goals, and past public statements. I hope to highlight these, in the event that these were oversights in the evaluation process and worthy of reconsideration.

# Inconsistent internally

Within the message, you note that the primary purpose is

> We use an open, public, and transparent procedure, and we evaluate the risks to users’ security and privacy

and later state:

> While the risks presented by a root and an intermediate may be very similar, they are not identical. For example, a root certificate can issue other CA certificates, which introduces the risk that a CA certificate will be mis-issued. While the intermediate CA certificates proposed here cannot be used to issue CA certificates, and can only be used to issue DV server certificates.

Unfortunately, this suggestion of "less risk" is both technically flawed, and inconsistent with the explanation of the rationale. To focus exclusively on the latter, the message later states:

> Because GoDaddy is putting its own root at risk, such reviews have gone beyond the minimum requirements of Mozilla and the CA/Browser Forum's Baseline Requirements

This suggests that the conclusion is based on the assumption that, should Certainly have issues, GoDaddy would be appropriate to distrust. As discussed in previous messages, this clearly demonstrates the significantly greater risk posed by the existence of an intermediate, and shows that the ability to issue sub-CAs is entirely orthogonal to the risk remediation process, namely, sanction against GoDaddy. GoDaddy represents one of the largest CAs in Mozilla's program, and it seems unlikely that Mozilla is willing to take action except in the most egregious of scenarios, due to the end-user impact. If this is the statement, then it seems Certainly poses high risk to users, because it can "take out" GoDaddy. Alternatively, if this is not the statement, then it shows the inconsistency in the conclusion in that GoDaddy is under very little functional risk, and thus has less incentive to perform adequate review - as highlighted by Andrew Ayer's review, my review, and Certainly's own disclosed incidents thus far.

Notably, while Certainly has noted it's using Boulder, it's worth highlighting that both Certainly and GoDaddy failed to discover issues with that implementation until reported by another CA [1]. This should raise questions about the degree of thoroughness and correctness of the review performed - of code, practice, and policy - and highlight the questionable nature of relying on GoDaddy's assertions.

# Inconsistent with the public discussion

Without repeating your excellent summary of the discussion thus far, it shows that there is limited support in the community for continuing forward. While understandably, the role of Mozilla is to weigh many different viewpoints, and not just count by number and volume (length), it should raise some degree of concern that the emphasis of risk that exists here seems unaddressed. Much of this seems to be dismissed under the basis that GoDaddy has performed an analysis sufficient for Mozilla to ignore its own extant policies and procedures for managing CA risk, which also seems inconsistent with the degree of public discussion and transparency.

For example, although GoDaddy has shared their process, there's been no substantive response to the concerns raised by Andrew Ayer that discusses procedurally how this was overlooked. Although we have the response from Wayne Thayer regarding the intent, it highlights the possibility that both GoDaddy and Certainly are operating on certain undocumented assumptions about the interpretation of language and/or requirements that would normally be part of a public discussion during review. This, equally, should raise questions about the degree of thoroughness and correctness of the review performed.

# Inconsistent with Mozilla's stated values, goals, and past statements

The nature of risk presented here certainly raises questions with respect to Principles 4 and 8 [2]. I think it's questionable to suggest that GoDaddy's internal review satisfies those. From examining the spreadsheet [3], it's also clear that GoDaddy's primary review was conducted not with the same Mozilla principles in mind, but rather, whether there was consistency and/or conflict with GoDaddy's (Starfield's) CP/CPS. This an important step for cross-signing, but it does not address the same goals, nor ensure the same perspectives. The additional documents [4] supplied are very clearly forms of self-review, and lack both transparency and accountability that demonstrate a commitment to these principles.

With respect to past statements, it's difficult to reconcile the glowing remarks about the value of audits, and the diligence provided by third-parties (e.g. GoDaddy), with Mozilla's statement of concerns with respect to proposals such as the eIDAS Regulation [5], or its justification for its root store [6]. For instance, in the concerns regarding the EU Trust List, it's highlighted that there are concerns with transparency, audits, and risk management. There is no transparency into the ongoing GoDaddy supervision, only in the conclusion, which denies users appropriate transparency into the discussions and interpretations, as highlighted above. While Mozilla highlighted concerns with the irregularity of the audits in [5], past concerns raised by Mozilla [7] have highlighted there is far more to supervising CAs and minimizing risk than audits alone. However, your statement in favor of positive dispensation for Certainly seems to rest largely, if not entirely, on an assumption that audits are alone sufficient. And in terms of risk management, this thread has shown a number of ways in which Mozilla is failing to fully consider its risk management. These risk management concerns are echoed in [6] about the value of the root store, but this reply seems to separate out the selection of roots from those of intermediates, as if intermediates aren't an intrinsic part of both the value of, and risk to, Mozilla's policies and practices highlighted in that post.

Equally, when one considers the super-CA policy, [8], it suggests that if one wants to start a super-CA, the process is rather to add the super-CA as a root first, before signing subordinates, such that newly-added subordinates can use the expedited process and bypass the review currently required if there is an existing parent/subordinate relationship.

# Conclusion

Ultimately, Mozilla's decisions for its programs are Mozilla's, and there is no question that there is incredible value in the transparency of this discussion, and value in the opportunity to participate in it. However, the conclusion seems to argue that audits and basicConstraints are sufficient to mitigate risk to end users, such that it's not necessary for Mozilla to perform a careful, detailed, and diligent review. It also seems to discount the numerous concerns raised by participants about the process.

I appreciate the discussion with respect to the wiki, but note that it's been even longer that concerns have been raised around such cross-signing events (e.g. [9], [10]), to which Mozilla did not substantively respond. Originally proposed as part of Policy 2.8, it now seems to have been adopted in full independent of that, and it leaves it ambiguous whether current proposals [11] are advocating for a Mozilla-led review, or whether they're instead weakening the intent of both policy and language to being led by CAs.

My biggest concern, however, remains that it's difficult to see the distinction Mozilla is making here, with respect to delegating and deferring to GoDaddy, and how that squares with concerns it has raised delegating and deferring to other organizations, whether super-CA operators ([8]) or the European Trust Lists ([5], [7]). Mozilla is, functionally, outsourcing the oversight and compliance, and it's unclear the value that Mozilla, or users, derive from this process, other than expediting the inclusion of CAs. Can Mozilla articulate why it cannot substitute "EU Supervisory Bodies" in for "GoDaddy" in the rationale to continue, and thus automatically adopt the European Trust Lists, provided it also has the ability to remove? It's difficult to see, based on the stated rationale, why it could not, and that's concerning.

As I said, I have no objections if the view is that such subordinate CAs should be moved to the front of the queue for prioritization, provided that the same processes occur and without any predisposition towards including the CA. The rigorous vetting [12] that Mozilla has endorsed [13] does not appear to be performed here, or appears to be performative rather than substantive, and that remains a concern.

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1752452

[2] https://www.mozilla.org/en-US/about/manifesto/

[3] https://bugzilla.mozilla.org/attachment.cgi?id=9266000

[4] https://bugzilla.mozilla.org/attachment.cgi?id=9266001

[5] https://blog.mozilla.org/netpolicy/files/2020/09/Mozilla-Attachment-to-the-European-Commission-Review-of-eIDAS.pdf

[6] https://blog.mozilla.org/security/2019/02/14/why-does-mozilla-maintain-our-own-root-certificate-store/

[7] https://www.ccadb.org/documents/TSP_Technical_Best_Practices_eIDAS.pdf

[8] https://wiki.mozilla.org/CA/Subordinate_CA_Checklist#Super-CAs

[9] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/AA5G1bzOwZQ/m/-L7Q_bdABQAJ

[10] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/AA5G1bzOwZQ/m/mS5rard2BQAJ

[11] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/r9ZeggoH9KA/m/NZwE3TuYAwAJ

[12] https://www.eff.org/document/eidas-letter-2022

[13] https://blog.mozilla.org/en/security/mozilla-eff-cybersecurity-experts-publish-letter-on-dangers-of-article-452-eidas-regulation/

[Wayne Thayer]

In a prior message I stated that Certainly would review Ryan's specific comments on our CP/CPS and make improvements. We have drafted the changes described below and will include them in the next version of our CP/CPS.

On Sun, Mar 6, 2022 at 8:24 AM Ryan Sleevi <ryan....@gmail.com> wrote:

I have not done a detailed CP/CPS review as I would normally do, in part, because there are not the supporting materials that normally exist by the point that I would do such a thing. I am concerned that the "Self-Assessments" may have moved from a "thing to benefit the CA" and into a "thing the Root Program relies on," and I do worry about that. I do want to make it clear: Despite the thoroughness of my comments here, I did not do my normal deep review here, and I would not rely on this as a "Hey, this is proof that if this is all he found, we're fine" - it's simply a quick scan of concerns:

• 1.3.2 notes that RA functions are not delegated, but the section is titled "Registration Authorities and other delegated third parties". In Section 1.3.5, it notes that "Certainly vendors and service providers that have access to confidential information or privileged systems ...", suggesting that there may be at least some functionality that is delegated to third parties. 1.3.2 of the BRs allows for delegating the RA functions (Section 3.2) to DTPs, but the definition within 1.6.1 of the BRs makes it clearer that a DTP applies to any delegation of function. That is, the BRs allow for either 3.2 functionality (e.g. as supported by 4.2.1 of the BRs) or any other function other than 3.2.2.4/3.2.2.5 (supported by 5.3.7 and 5.4.1 - related to ISMS and not just RA functions)

There is no reason for the section title to deviate from RFC 3647 by adding “and other delegated third parties”, so that will be corrected in the next CP/CPS update. Certainly does rely on delegated third parties to meet a few specific requirements. An example would be storing encrypted backups offsite. A 3rd party penetration tester would be an example of a vendor with access to confidential information.

• 1.4.1 limits the certificates to use with the TLS protocol, and 1.4.2 prohibits any other use. What does this mean for QUIC, which Fastly supports [3]? I realize the extent that QUIC repurposes the TLS handshake, but it is a different protocol [4].

Certainly certificates are not currently deployed to the Fastly CDN, or any service that supports QUIC. This section of the CP/CPS will be updated to eliminate any future conflict with QUIC.

• A brief scan of 1.6.1 shows that there are subtle differences in the definitions between Certainly's CP/CPS and the BRs. That's _not_ problematic in and of itself, but requires careful scrutiny to review that the meaning and obligations are preserved throughout the entire document and are consistent under these modified definitions.

Certainly’s CP/CPS will be updated to more closely mirror the BR definitions.

• 1.6.3's references are unversioned. There are various semantic and syntactic differences in X.509, for example, up to and including the recent changes to the underlying model [5], that these differences may matter, even if there is a "common sense" understanding one might take.

We’ll remove this section from our CP/CPS.

• 3.1.4 gives conflicting advice in terms of understanding the construction of names. With respect to DNs, this is the opportunity to be unambiguous about the name forms employed (presumably, C and CN) and their interpretation - and limits.

We’ll remove the RFC references from this section.

• 3.1.6 states "Certainly reserves the right to refuse to include a domain name within a Certificate". This right seems to go beyond those otherwise enumerated by the BRs (e.g. scenarios contemplated within 4.9.1), and so it's unclear when, why, and how this right might be exercised. I anticipate that Certainly wouldn't want to give up this right, even if it's not strictly necessary, but it perhaps bears elaborating why this reservation is seen as necessary and how it might be exercised.

While we do not currently envision a clear scenario in which this right would be used, consistent with other CAs, we feel that it is important to reserve this right in the event that we need it to address unforeseen circumstances.

• 3.1.6 seems to suggest that, independent of the UDRP, Certainly may make its own determinations about the right or authorization to use a domain name. That seems suboptimal.

We’ll remove the final sentence referencing domain name disputes. 

• 3.2.5 appears that Certainly have read the BRs 3.2.5 as [If A then (B, C, and D)], rather than [(If A then {B, C}), and D]. That is, whether the "In addition" in the terminal paragraph joins to the first condition (if the Applicant ... is an organization), or whether it states an procedure that is applicable for all Applicants, and not just those that are organizations. While understandably the BRs can benefit from clarity here, it bears calling out, because as a consequence, they provide no such method to limit accounts that can request certificates, made explicit in 4.2.1

Correct. 

• 7.1 does not list all of the RFC 5280-required fields and extensions within the relevant profiles. This raises the question of whether they are "permitted, but unspecified", "prohibited, but not omitted, violating the CP/CPS", or "prohibited, but omitted, violating 5280". Taking the more generous first interpretation, on the basis of the first sentence in this section, it seems clearer to explicitly document the fields that will be included within the profile, and ensure no fields are included without a corresponding CP/CPS update.

We’ll add the omitted fields.

• 7.1 uses a rather long OCSP signing certificate (5 years), without documentation (AFAICT) of the corresponding key protection. Given the description of the tradeoffs of OCSP no-check, it seems like a shorter lifetime reduces risks/concerns.

This section will be removed as Certainly does not currently perform delegated OCSP signing, and thus has not signed and does not intend to sign any OCSP signing certificates.

Thanks,

Wayne

[Ben Wilson]

All,

Public discussion and the 7-day last-call period recently ended[1], and Certainly's request to include its R1 and E1 root CAs in the root store has been approved[2]. The purpose of this email is to clarify that GoDaddy's request to cross-sign Certainly's issuing CAs[3] is similarly approved. GoDaddy and Certainly have successfully completed the Process for non-Technically Constrained Subordinate CAs[4].

Thanks,

Ben

[1] https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/EhXhiHfWGC8/m/3PcJHizqAAAJ

[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1727941

[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1755851

[4] https://wiki.mozilla.org/CA/External_Sub_CAs