CA Incident Response and Delayed Revocation Correspondence

[Wayne]

Given all of the discussion on delayed revocation the past few months I was thinking it would be worthwhile for CAs to have a public discussion on how they approach incidents, especially in communicating to subscribers.

There are a lot of expectations set forth by CA/BF and Root Programs, and from that I suspect there's a lot of re-inventing the wheel across CAs. From tone, content, and to what is critical to encourage immediate action and response.

I trust all parties are trying their best, so let's start with a generic guide - plenty of room for improvement:

---
Subject: Certificate Services Message - URGENT ACTION REQUIRED: Revocation May be Required
- This has obvious steps for improvement, consider it an easy step for your CA to show how much better they are

Email Content:
We are writing to inform you about a recent issue that affected [some of the EV digital certificates] issued by $CA. We apologize for any inconvenience this may have caused you and we are committed to ensuring the security and integrity of your online transactions.

Summary:
- Does your CA just ask for revocation, or are they detailing what went wrong in the certificate? Note that while being transparent in this front can be helpful, you may be asking subscribers to argue over the details when a revocation is MUST.
- Be very careful in giving options here, as they'll be the first thing a rushed subscriber will latch onto as a reason. Priority should be given on telling the subscribers the certificates will be revoked, it's just a matter of when they'll be grabbing their new certificate.
- Do you state who discovered the issue? Is this relevant to the subscriber, and are you being up front on initial discovery vs confirmation internally?
- Do you give a specific timeframe for when re-issuance started after a solution was put in place?
- Do you give a final deadline for revocation? If so do you start the 24h/5-day clock on corresponding with subscribers, when you internally confirmed the issue, or when you read the initial notification (if third-party)?

ACTION REQUIRED:
- A call to action is good, but are you telling subscribers what will happen to their certificates if they ignore the email?
- Do you give a broad criteria of mis-issued certificates, or can you give a cert-by-cert breakdown to each subscriber? How well does this scale for the 1-certificate subscriber to the 10000-certificate subscriber?
- Do you have updated translations for all of your potential subscribers? This is where having a generic template is key and delays can happen for trying to explain complex BR issues.

Steps:
- A step-by-step process for what the subscriber should do to receive a fresh certificate, whether it is by ACME, or a manual process through a web portal.
- Be very sure that your portal is telling subscribers the correct revocation period. Giving them an arbitrary 30-day delay if they re-issue a certificate can make trouble for yourself too.
- Be very specific on when revocation will occur if the re-issued certificate is put in place but no follow-up on the 'revoked' certificate happens.
- If you decide to do a arbitrary delay, enforce the deadlines. If your subscriber has received a certificate then you've done everything in your power, the issuance should give terms enforcing this.
- This is the ideal time to do a gentle push for automation, make sure to highlight this and remind subscribers they wouldn't be doing this manually if they had automated

Finally:
- Reinforce that revocation will be happening.
- Remind subscribers to update all emergency contact information
- Give another push for automation, remind them that handling this all manually is additional work
---

Beyond correspondence I want to highlight a few potential rooms for improvement in creating Bugzilla incidents:
Make sure to give per-subscriber revocation reasons, and a timeframe for when these certificate would naturally expire if not revoked. It sounds simple, but most CAs aren't going to this depth which is required.

Don't be afraid to talk about delays which occurred internally, or mistakes that have occurred. That's a natural part of a complex issue like this, the important part is everyone understands why it happened. Be sure to document your action items to show that substantial effort is happening to make sure it doesn't re-occur. If it does re-occur, have a serious discussion internally and ask your fellow CAs what they are doing in the field.

You must remember to provide updates at least once a week. Even if progress has been small it helps everyone understand the work you are doing. It helps keep the issue fresh and shows that they haven't been forgotten by your CA.

If a question is asked, try and consider why the person asked and how your response best informs them on your priorities. Most critically: always pay attention to other issues for common themes, that could easily be your CA instead.

To be clear I'm not putting forth a single set template for every CA to have mandated. I hope everyone would agree that having generally-agreed best practices can do an industry a lot of good in setting expectations and standards. While these discussions have happened in private with other CAs, ultimately any correspondence reaches thousands of people.

I hope everyone realizes none of this should be breaking new ground, and is instead an important place for the industry to reconsider their approach from the fundamentals.

- Wayne