Support for quick certificate replacement in subscriber tooling

[Jesper Kristensen]

Hi,

Given the recent discussions on how to avoid delayed revocations, I wanted to know how feasible it is for certificate subscribers to have automation in place that can handle rapid certificate replacement. While cloud providers and big tech companies can build their own custom certificate automation, what options do the rest of subscribers have using standard off the shelf tools? Can the tools handle mass revocations, or does it have to be done manually, which is likely to be impossible within 24 hours or 5 days for a subscriber with several thousands of certificates?

I could not find any statistics on the popularity of certificate automation tools, so I picked some of the tools listed on acmeclients.com plus a few others. Sorry if I didn't include your favorite tool, you are welcome to add to the list. I did not try to install or run the tools. I just looked at their publically available documentation. I tried to answer these questions:

- ARI support: Does the tool use ARI, OCSP or another technology to check if its certificates need early replacement?
- Fallback CA support: Can the tool automatically try with another CA after it has failed to renew a certificate from the primary configured CA?
- Multi CA support: Can the tool get a backup certificate from another CA automatically before it fails to renew a certificate from the primary configured CA, so the backup is ready in advance in case it is needed?
- For each question above: If the tool has a default configuration, is it part of that configuration? If the tool does not have a default configuration, is it part of the most prominent getting started guide in the documentation?

My conclusion is that the ecosystem isn't where it needs to be to support trouble-free revocation within the TBR timelines. What can we do about that?

Here is a list of my findings:

Caddy
ARI support: no
ARI default: no
Fallback CA support: yes https://caddyserver.com/docs/automatic-https#issuer-fallback
Fallback CA default: yes, 2 CAs. https://caddyserver.com/docs/automatic-https#issuer-fallback
Multi CA support: no
Multi CA default: no

Traefik
ARI support: no https://doc.traefik.io/traefik/https/acme/#automatic-renewals
ARI default: no
Fallback CA support: no https://doc.traefik.io/traefik/https/acme/#certificate-resolvers
Fallback CA default: no
Multi CA support: no https://doc.traefik.io/traefik/https/acme/#certificate-resolvers
Multi CA default: no

acme.sh
ARI support: no https://github.com/acmesh-official/acme.sh/issues/4944
ARI default: no
Fallback CA support: no https://github.com/acmesh-official/acme.sh/wiki/Server
Fallback CA default: no
Multi CA support: no
Multi CA default: no

Certbot
ARI support: no
ARI default: no
Fallback CA support: no https://eff-certbot.readthedocs.io/en/latest/using.html#changing-the-acme-server
Fallback CA default: no
Multi CA support: no
Multi CA default: no

cert-manager
ARI support: no https://cert-manager.io/docs/usage/certificate/#issuance-triggers
ARI default: no
Fallback CA support: no https://cert-manager.io/docs/usage/ingress/#inner-workings-diagram-for-developers
Fallback CA default: no
Multi CA support: no
Multi CA default: no

Lego
ARI support: yes https://go-acme.github.io/lego/
ARI default: no https://go-acme.github.io/lego/usage/cli/options/
Fallback CA support: no https://go-acme.github.io/lego/usage/cli/options/
Fallback CA default: no
Multi CA support: no
Multi CA default: no

Certify The Web
ARI support: no https://docs.certifytheweb.com/docs/renewals
ARI default: no
Fallback CA support: yes https://docs.certifytheweb.com/docs/guides/certificate-authorities
Fallback CA default: yes, 3 CAs without EAB requirements configured by default
Multi CA support: no
Multi CA default: no

cPanel
ARI support: no
ARI default: no
Fallback CA support: no https://docs.cpanel.net/whm/ssl-tls/manage-autossl/#providers
Fallback CA default: no
Multi CA support: no https://docs.cpanel.net/whm/ssl-tls/manage-autossl/#providers
Multi CA default: no

Plesk
ARI support: no https://docs.plesk.com/en-US/obsidian/administrator-guide/website-management/websites-and-domains/advanced-website-security/securing-connections-with-ssltls-certificates/securing-connections-with-the-ssl-it!-extension.80001/#renewing-installed-ssl-tls-certificates
ARI default: no
Fallback CA support: partially, 2 CAs supported, one with automatic renewal, fallback to that one
Fallback CA default: no
Multi CA support: no
Multi CA default: no

NGINX njs-acme
ARI support: no
ARI default: no
Fallback CA support: no https://github.com/nginx/njs-acme?tab=readme-ov-file#staging-by-default
Fallback CA default: no
Multi CA support: no
Multi CA default: no

Apache mod_md
ARI support: no https://httpd.apache.org/docs/current/mod/mod_md.html#mdrenewwindow
ARI default: no
Fallback CA support: yes, but only CAs without EAB https://httpd.apache.org/docs/current/mod/mod_md.html#mdcertificateauthority
Fallback CA default: no, 1 CA configured by default https://httpd.apache.org/docs/current/mod/mod_md.html#mdcertificateauthority
Multi CA support: no
Multi CA default: no

[Walt]

Caddy absolutely does support ARI as of 2.8.0.

I'd argue that it also doesn't need to try to renew ahead of time and get a backup certificate. With how aggressive Caddy is for checking if a cert needs to be renewed, I'm having trouble imagining a situation where ZeroSSL fails, and then so does LE (or vice versa) before the certificate needs to be expired (including ARI).