Webhook fails after caBundle in MutatingWebhookConfiguration updated

[Yu Peng]

Hi team, I am using cert manager to inject certificates for a kubernetes webhook service. My webhook uses http.ListenAndServeTLS() from standard lib.

It seems to have a problem because cert-manager can update the caBundle in MutatingWebhookConfiguration. Once caBundle is updated, certificates read by http.ListenAndServeTLS is no longer valid, and it causes webhook request errors like:

x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "serial:xxx")

After restarting webhook pods(loading new certificates), webhook comes back to life.
My questions are
- Do I get the right understanding on how things work?
- And what's the proper way to handle updation of MutatingWebhookConfiguration by cert-manager?

Thanks : )

[Maël Valais]

Hi,

Thank you for your message. As I understand it, your webhook pod loads a Secret resource that contains the private key and certificate that were created by cert-manager. On the other side, cert-manager’s cainjector puts the contents of the ca.crt in that Secret resource to fill the caBundle in the MutatingWebhookConfiguration object.

The function http.ListenAndServeTLS loads the private key and certificate from disk (where the Secret above is mounted) when the pod starts. When cert-manager rotates the certificate and private key, the tls.crt and tls.key files on disk are changed, but http.ListenAndServeTLS doesn’t know about that change.

You have two solutions to solve this limitation: (1) have something that will restart the webhook process when the secret changes (there is a controller for that, I don’t remember the name).

(2) or you can do the same as what controller-runtime does in [1], which is to use fsnotify to watch for changes in disk and gracefully reload the ListenAndServe function when that happens.

[1]: 

https://github.com/kubernetes-sigs/controller-runtime/pull/421/files

Have a good day,

Maël 

[Erik Godding Boye]

We use Reloader to make "dumb" apps restart when configmaps/secrets are updated without the app being able to detect that. I'll guess that's the operator that Maël refers to? You should be able to mend your issue with that, but I would recommend to use controller-runtime to expose your webhook - if possible. We have webhooks using certs issued by cert-manager without any issues, and we use controller-runtime.

Erik

[Yu Peng]

Thank you both, controller-runtime would be the best solution.

I tried not to use kubebuilder because it introduces some unnecessary scaffold files. Looks like it wasn't a very wise decision.
I will use controller-runtime to fix my code. Thank you both again.