federated certificates?

[Eric Peters]

Hi ~

I'm curious if anyone has come up with a solution for multi-region kubernetes clusters.  

The primary issues I'm trying to wrap my head around is how HTTP ACME Challenges would work in a multi-cluster environment.  If eu-central's cert-manager is requesting an HTTP ACME certificate, and a Route53 Latency Based Routing for LetsEncrypt connects to us-west, then the cert-manager in us-west will respond with the wrong challenge response.

A couple ideas I'm exploring:

"master" cert-manager cluster:

• Deploy cert-manager in only a single "master" cluster
  
• Deploy routes in over slave clusters to proxy ACME-challenge requests to the master cluster
• Have (does it currently support?) cert-manager write to federation secrets? (https://kubernetes.io/docs/tasks/administer-federation/secret/)

pros: 

• avoids duplicate certificates for the same domain/resource across the clusters
• 
  

cons: 

• additional configuration overhead for provisioning "master" cluster configuration and different deployment configurations
• no idea if cert-manager would work with federated secrets
  
• not sure how this would work across namespaces, or how it would integrate with other kuberntes applications like ambassador/envoy ingresses that generally read certificates from the current namespace
• pollute a shared secret with ad-hoc environment certificates, or if the federated secrets could (by convention) be created with namespace prefixes - handling acme staging vs production certificates are an open question

"cluster proxy/polling" endpoint:

• Each cluster has a normal cert-manager installed and will respond to HTTP challenges, and setup DNS validation
• Have an endpoint for the HTTP challenges that can make parallel requests to each of the clusters in the federation looking for a cert-manager that has a valid response for the challenge, and return back the appropriate response.

pros:

• more traditional setup, certificates get created in the normal cluster's namespace-specific secrets, no integration issues with other k8s apps
• each cluster has an identical deployment profile (lower overhead)

cons: 

• might run into rate-limiting issues if too many clusters all request certificates for the same domain at the same time? (the 50 certificates per domain, per week probably would be enough for 4-5 clusters though)
• ??? are there any CONS or edge cases if each cluster has a different certificate for a domain, or the renewals are staggered?
• not 100% sure how the cluster polling/proxying would work

"manual certificate management":

This is effectively what we are doing now (in a pre-kubernetes architecture), using certbot to manage certificates and package them with haproxy/ingres.

pros:

• easy audit trail for deployments

cons:

• manual intervention required
• more complicated (maybe not with wildcard certs...) to create an adhoc environment with new host/certificates

"hybrid" cert-manager:

Use cert-manager for DNS-based certs, manually manage any HTTP-based certificates and package them with the ingress.

pros:

• 100% vanilla implementation
• Ad Hoc environments that depend on wildcard certs can still use DNS-validation 

cons: 

• requires manual maintenance and setup for any HTTP-based certs that are needed.

So, has this been tackled by anyone already?  What else am I not thinking about?  Any ideas/advice would be greatly welcomed!

Thanks,

Eric