issuance to subscribers who cannot tolerate prompt revocation

[Mike Shaver]

Hi; I hope this is the right list for these questions and discussion. Please feel free to redirect me to a more suitable venue if there should be one.

TL;DR

Is it permissible for a CA to issue a certificate to a Subscriber if they know that the Subscriber cannot fulfill the warranties and acknowledgements from section 9.6.3 of the BRs, such as Subscribers who are not able to “accept” or tolerate revocation along the BR’s timelines?

CONTEXT

Section 9.6.3 of the TLS BRs (version 2.0.4) deals with Subscriber warranties and acknowledgments. I reproduce the preamble below, which is followed in the document by a list of such warranties and acknowledgements, with apologies for the length:

The CA SHALL require, as part of the Subscriber Agreement or Terms of Use, that the Applicant make the commitments and warranties in this section for the benefit of the CA and the Certificate Beneficiaries. Prior to the issuance of a Certificate, the CA SHALL obtain, for the express benefit of the CA and the Certificate Beneficiaries, either:

1. The Applicant’s agreement to the Subscriber Agreement with the CA, or

2. The Applicant’s acknowledgement of the Terms of Use.

The CA SHALL implement a process to ensure that each Subscriber Agreement or Terms of Use is legally enforceable against the Applicant. In either case, the Agreement MUST apply to the Certificate to be issued pursuant to the certificate request. The CA MAY use an electronic or “click‐through” Agreement provided that the CA has determined that such agreements are legally enforceable. A separate Agreement MAY be used for each certificate request, or a single Agreement MAY be used to cover multiple future certificate requests and the resulting Certificates, so long as each Certificate that the CA issues to the Applicant is clearly covered by that Subscriber Agreement or Terms of Use. The Subscriber Agreement or Terms of Use MUST contain provisions imposing on the Applicant itself (or made by the Applicant on behalf of its principal or agent under a subcontractor or hosting service relationship) the following obligations and warranties:

Item 8 in the list is as follows:

8. Acknowledgment and Acceptance: An acknowledgment and acceptance that the CA is entitled to revoke the certificate immediately if the Applicant were to violate the terms of the Subscriber Agreement or Terms of Use or if revocation is required by the CA’s CP, CPS, or these Baseline Requirements.

CONCERN

I draw attention to this requirement, which must be legally binding upon the Subscriber, because there have been *many* incident reports in Bugzilla describing delayed revocation caused by a Subscriber’s inability (or unwillingness to invest sufficiently) to replace a misissued certificate within the 24-hour window, or even the recently-extended 5-day window. Furthermore, the services to which these certificates are deployed are described as “critical infrastructure”, or the CA claims that revocation would be “disruptive to the web ecosystem”. From the descriptions provided by the CAs, it seems that due to the context in which the subscribers operate (government, health, banking, travel) the CAs in question *expected* that these Subscribers would not be willing to replace certificates within the well-known and agreed-to BR deadlines.

By my understanding of the BRs, no CAs should be issuing certificates to Subscribers if they know that the Subscriber’s acknowledgement and acceptance of point 8 is untrue. Many CA’s CPSs also prohibit use of their certificates in contexts that may lead to loss of life, human injury, or substantial financial loss. (Surprisingly, some don’t, which I understand to mean that they warrant the certificate’s suitability for life-critical applications?)

QUESTIONS

What is the understanding of this group? Is it permissible for a CA to issue a certificate to a Subscriber when they know that this Subscriber does not, in fact, acknowledge and accept item 8? If CAs are to work with Subscribers to help those Subscribers tolerate immediate revocation as per the BRs, I believe that this work should happen *before* a certificate is issued, not in the wake of a misissuance. (What would happen if there were a key compromise? WebPKI is used for many important things, but I don’t think it is appropriate for contexts where loss of CA or Subscriber key material could result in intolerable consequences for society and the web.)

As a minor follow-up question, would it be permissible for a CA to provide a guarantee to a Subscriber that may require the CA to violate the BRs? For example, could a CA contractually warrant to a Subscriber that the Subscriber would have 30 days to replace before revocation?

Thank you for your time and consideration, and all the work you do to keep the WebPKI healthy for its primary constituents: users of the web.

Mike

[Aaron Gable]

(Everything below is my personal opinion, not the opinion of my employer.)

On Fri, May 24, 2024 at 1:58 PM Mike Shaver <mike....@gmail.com> wrote:

Is it permissible for a CA to issue a certificate to a Subscriber when they know that this Subscriber does not, in fact, acknowledge and accept item 8? 

I think there is a subtlety here: just because a Subscriber "acknowledges and accepts that the CA is entitled to revoke the certificate immediately" doesn't mean they're actually ready to deal with the fallout of the CA actually doing so. Just because I acknowledge and accept that those close to me will one day pass away, I know I'm certainly not ready to actually manage their wills!

On the whole, I certainly agree with your sentiment here: Subscribers who cannot replace a certificate in less than 5 days are not protecting the privacy and the security of their users, and either need to improve their response time or investigate non-WebPKI solutions. And CAs which knowingly issue to such Subscribers (perhaps because that Subscriber has failed to replace a certificate in a prior incident) are asking for trouble.

But I also think that issuing to such a Subscriber is not necessarily itself a misissuance. The Subscriber has agreed to a legally binding Subscriber Agreement which includes the necessary warranties. It is not necessarily the CA's fault -- though it is their problem -- that the Subscriber has failed to understand the full ramifications of that warranty.

Aaron

[Mike Shaver]

On Fri, May 24, 2024 at 5:22 PM Aaron Gable <aa...@letsencrypt.org> wrote:

On the whole, I certainly agree with your sentiment here: Subscribers who cannot replace a certificate in less than 5 days are not protecting the privacy and the security of their users, and either need to improve their response time or investigate non-WebPKI solutions. And CAs which knowingly issue to such Subscribers (perhaps because that Subscriber has failed to replace a certificate in a prior incident) are asking for trouble.

The problem is this: the “trouble” is being pushed onto the users of WebPKI, because the Subscriber’s unwillingness to accept prompt revocation is used along with what I think is bad faith abuse of the latitude provided to CAs to unilaterally delay revocation and therefore keep misissued certificates in use for an extended period of time (more than a month in some cases!).

This represents real risk for the web, not only because in the event of a key compromise, if the pronouncements of the CAs in question are true, we would have to choose between an insecure web and harm to human health or critical infrastructure. Having misissued certificates (of any kind) in circulation harms the ability of WebPKI users to rely on the BRs’ guarantees, and risks interoperability issues for new entrants.

Section 1.4.2 of the CPS does not have any requirement that WebPKI certificates not be used in critical contexts, but if we are told that it is disastrous to revoke these certificates in a timely manner, then perhaps there should be some minimum exclusions. (Perhaps there is a CA who genuinely believes that their certificates are reliable enough for such uses, though? A scan of some CPSes sort of indicates that, but I expect it is more omission than deliberate choice.)

But I also think that issuing to such a Subscriber is not necessarily itself a misissuance. The Subscriber has agreed to a legally binding Subscriber Agreement which includes the necessary warranties. It is not necessarily the CA's fault -- though it is their problem -- that the Subscriber has failed to understand the full ramifications of that warranty.

I honestly don’t care about whose fault it is, just how to avoid this pattern repeating. Some of these Subscribers have been using a given CA’s certs for longer than the BRs have existed, and—barring force majeure use of something like OneCRL—the CAs are the only ones who can keep the risk and harm from being externalized to the users who rely on the WebPKI. CAs are knowingly entering into these arrangements, and if you’re correct that it’s not a misissuance then I don’t understand what point 8 being in the baseline requirements is.

Relatedly, what is the purpose of requiring the acknowledgements of that section to be legally binding, if the CAs don’t have a responsibility to use that legal tool and revoke misused certificates? Any CA could themselves add such a clause to their terms if they wanted it themselves, but unfortunately I do not recall the motivation for making such a requirement part of the BRs. Maybe it’s in mailing list archives?

Mike

[Tim Callan]

Today we see many CAs who don’t want to go through a forced revocation event who use their Subscribers’ ostensible incompetence as an excuse for their own failed CA operations.  This can only be either deliberate misdirection or extreme ignorance and misunderstanding of the responsibilities and nature of a public CA.

 

It’s quite simple.  The CA’s responsibility is to perform the revocation within the allocated timeframe.  That’s all.  The Subscriber’s responsibility is to deploy replacement certificates from the misissuing CA or another, on time to avoid an outage or not.  The CA’s responsibility does not extend to the Subscriber’s certificate agility or lack thereof.  It does not include the Subscriber’s process or involved parties or governing law.  It does not include the nature, severity, duration, or likelihood of a resulting outage.

 

This is all the Subscriber’s responsibility.  The risk/reward analysis of running an operation that cannot weather a forced revocation event is the Subscriber’s to calculate.  The consequential process and resourcing decisions are the Subscriber’s to make.  The public-versus-private trust analysis is something only the Subscriber can conduct.

 

Which is appropriate, since the Subscriber has to live with the consequences.  In the event of a forced revocation, if the Subscriber deploys the new, replacement certs on time, then everything continues to run.  And if not, then not.

 

This is the only conceivable model that can actually work.  There is no mechanism for CAs to reliably determine if “critical infrastructure” uses their certificates and no clear definition of what qualifies as critical.  There is no way for a CA to realistically evaluate the consequences of a revocation event.  And unfortumately, we have a healthy dose of empirical evidence telling us that Subscribers are unreliable reporters of the nature of their own services and the severity of downtime.

 

There is research suggesting that three quarters of CAs with current mandatory revocation events are not getting them done in five days, and that at 30 days in, multiple CAs still have more than 95% of their certificates unrevoked.  If my memory serves me correctly, 100% of these CAs have blamed their revocation failure on Subscribers’ inability to swap out these certs and the terrible consequences of a potential outage.  These CAs would have us believe that more than 95% of their certificates enable critical functionality that cannot undergo an outage without such disastrous consequences that they warrant throwing out the only error mitigation mechanism the WebPKI possesses. It defies credibility that this is true for any CA in the world, let alone three quarters of those with a misissuance event.

 

Our own experience with forced revocation events here at Sectigo has shown us that Subscribers will always ask for more time.  They will always say they can’t get it done and that the effect of not getting it done will be disaster.  That makes sense.  An unscheduled certificate replacement exercise is inconvenient for the Subscriber.  It takes time that could be spent on other tasks.  It adds hassle and worry.  It may cost overtime or incremental service budget.  Of course the Subscribers don’t want to do it, so when they are allowed to choose, they choose not to.

 

But when the CA simply offers that this revocation will occur in a specified timeframe, the Subscribers miraculously get the new certificates deployed.  Every time.  We see this miracle occur over and over again.

 

It could be that Sectigo has the luckiest and most competent set of Subscribers on the planet.  Or maybe, just maybe, the fact is that nearly all Subscribers can handle a forced revocation event when it occurs.  They simply don’t want to.  If CAs had sufficient emotional fortitude and commitment to the principles they had signed up for, they would complete all these revocations on time, and the Subscribers would manage to deploy their new certificates, and we wouldn’t be having this conversation.  But the evidence overwhelmingly tells us that CAs do not.

 

So what we need is for this decision to stay out of the CA’s hands.  In fact, by the BRs and root program rules as written, with the exception of Mozilla, CAs do not have the authority to make this decision today.  It’s just that the CAs are nonchalantly violating these rules and for the most part there have not yet been visible consequences.  (Though we should note the likelihood that mismanagement of revocations was a contributing factor in the recent distrust of ecommerce monitoring GMBH.)

 

It is within the power of major root programs to affect policies and/or software functionality that will change CAs’ risk/reward decisions.  This can be threat of distrust but also could include less severe consequences.  Community participants have suggested ideas including capping the maximum term for offending CAs at 90 days, distrusting the EV roots of offending CAs, and enforcing a browser-side ban on misissued certificates after the revocation deadline.  These are all technically possible, and each root program will evaluate if one of these ideas, or another, makes sense.

 

I personally believe that any of these outcomes would have been motivational for most or all of the current offending CAs to complete their revocations on time.  I further believe that, in the absence of a completed revocation, they would have meaningful mitigating effects on these and future incidents from the same CAs.

 

-tlc

 

 

From: pub...@ccadb.org <pub...@ccadb.org> On Behalf Of Mike Shaver
Sent: Friday, May 24, 2024 5:41 PM
To: Aaron Gable <aa...@letsencrypt.org>
Cc: pub...@ccadb.org
Subject: Re: issuance to subscribers who cannot tolerate prompt revocation

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

 

--
You received this message because you are subscribed to the Google Groups "CCADB Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@ccadb.org.
To view this discussion on the web visit https://groups.google.com/a/ccadb.org/d/msgid/public/CADQzZqtoOnz5TXr9vh1tKrka%3DU3SC0vMLPoUhR7ySV4uqZViuw%40mail.gmail.com.

[Wayne]

I wholeheartedly agree that revocation is a must within the required timeframes. We've had automation being used as an excuse for delayed revocation, but I don't think that's telling us the full story. There is the elephant in the room that the CA industry is a commercial operation, and automation can be seen as another revenue point. Repeated revocations in a short timeframe also impact subscriber relations and provide perverse incentives to a CA in how they handle incident response.

To that end I want CAs to consider if they, or competitors, are charging for automation and what benefits that brings to the WebPKI ecosystem. Of course, there is a difference between charging for support personnel in configuring the environment vs responding to ACME requests. I doubt this will be a public topic, but please consider it in your private discussions however.

[Jesper Kristensen]

While I like the spirit in what you write, I think having a vague requirement about a CA knowing a subscriber's intentions will just result in an endless stream of incidents with arguments of subjective matters. I think that would be a waste of the web PKI community's time and resources, just like I think that the current requirement to not issue for weak keys has taken more resources historically than the benefit provided by the requirement.

I can also very much relate to Tim's message. From my experience with critical infrastructure, a change is not allowed with less than X weeks of notice, unless we have done everything we could to avoid it. I would interpret that policy to mean that we would only be allowed to replace a certificate within 5 days or 24 hours if we had told the CA that we would not be able to do it, and the CA had then told us that they would revoke on time anyway.

I think it is a problem that we only discuss these matters with CAs after they had an incident that required revocation.

People are bad at judging risks, just like how people rush to buy flood insurance just after a major flood has happened, many subscribers will not understand their obligations until their CA tells them that their certificate is about to be revoked.

I think the only way to improve is for these CAs to start revoking. Maybe apply the principles of grease / chaos engineering. Maybe CAs should be required to revoke a random set of certificates at random times with less than 24 hours notice just to test the replacement process, or maybe CAs should be required to hold a drill every year where they replace their entire certificate population within 24 hours.

--
You received this message because you are subscribed to the Google Groups "CCADB Public" group.
To unsubscribe from this group and stop receiving emails from it, send an email to public+un...@ccadb.org.

To view this discussion on the web visit https://groups.google.com/a/ccadb.org/d/msgid/public/CADQzZqv-6M-t0FYr0cHkzEeo-mWzVj-Db9wmEaoTU2Esy6RrWw%40mail.gmail.com.

[Wayne]

I agree that one of the possible approaches that Root Programs can employ would be a random set of certificates being revoked. 24 hours is a bit extreme for that though, I'd put forward 5 days as its the maximum time for a real issue to be handled.

Other potential levers I've considered that could be used but have a variety of different problems, just for consideration:

1. Add in extra limits for certificate lifespan, this will reduce the impact and add in pressure to improve processes. If we're moving to 90 days eventually then having non-complying CAs take the first step would be helpful for all
2. The random revoke/reissuance mentioned above, there's already a 3% audit so I don't see what the major barrier is for increasing this to revocation too - up the days to 15 for these and have that be an audited metric for successes in specific timeframes
3. If a CA is misbehaving then reducing what types of certificates they can issue would apply some pressure (i.e. no EVs will be trusted after x date until 3/6 months later), particularly if their issues are primarily in mishandling these certificate types
4. Policy-wise only allowing certificates that have been issued via automated means to subscribers' servers. We've had the excuse of automation peddled out enough, add in some pressure to force change there
5. Increase the self-reporting required to include which subscribers have received certificates, what their revocation means are, and whether they are using automation currently
6. If we're getting the critical infrastructure excuse time and time again, restrict a CA from handling these subscribers until they can consistently deal with less vital subscribers
7. At the extreme limiting what TLDs the CA can issue under, this is more a stepping stone to reducing root exposure across the board and would be policy until the technical measures are implemented
8. If problems are happening and the audits haven't noticed it then enforce additional an audit outside of the 'annual' interval by a company and personnel the CA hasn't used before
9. To help the industry more add in some requirements on implementing lints in zlint, etc. The CA should already be putting in resources into this area but this would be beneficial to the ecosystem as a penalty
10. Lastly a measure that would only be possible in a real regulated industry: bringing in personnel to oversee the CA's operations for 3/6 months directly that are employed by a regulatory body

I'm aware that these are not all possible but inspiration is being brought in from what an actual regulated industry would do. Having a full distrust as the main mechanism for a CA to expect is only causing them to push the limits of what is acceptable practice while not seeming bad enough to earn all of their roots revoked. What the industry needs to do in response is apply pressure to ensure that less-severe measures are used regularly for non-compliance to ensure that resources are actually put into compliance and not sales.

For revocation incidents I don't see what the barrier is for a final report to be generated by the CA of a list of their certificates that were due and when they actually happened. Care must be taken in ensuring that CAs are generating this data to show compliance, and not to generate excuses to push for slower and slower revocation deadlines however. This is part of why I included 30 days in my research and didn't only list 1/5/15 days.