A question about updating kubebuilder generated deployment

[drag...@gmail.com]

Hi,

We have a controller that is built using kubebuilder. I am wondering what happens if we update the deployment using "Recreate" strategy. Obviously, "RollingUpdate" doesn't make sense as at any given time, only one pod can be active any way. With Recreate, the old pod will be terminated before a new one is created.

When the current pod is terminated, would controller runtime stop processing new changes? Also, would it wait until any current reconciliation to be completed before existing?  I saw that the manager has "SetupSignalHandler()" but its usage is not very clear to me. Are we supposed to call this function and check ctx.Done() in our reconciler?

Any help is greatly appreciated.

Thanks,

Raghu

[David Sharp]

When K8s kills a pod, it first sends SIGTERM (generally), and gives a grace period (Pod.spec.terminationGracePeriodSeconds, default 30) before sending SIGKILL. As you noticed SetupSignalHandler() will capture SIGTERM and tell controller-runtime to shut down by cancelling it Context. This stops the caches from updating. This Context (or rather a child Context) is the one eventually passed to Reconcile().

Ideally, your Reconcile() methods should be fast and finish well within the grace period. If so, I would say there's no need to check ctx.Done(). If you've written long-running blocking operations into your Reconcile(), or access external resources that may take a long time to return when degraded, then checking ctx.Done() is a good idea.

Of course, no changes will be processed by your controller while it is not running. If your operator installs an admission webhook (Validating/MutatingWebhookConfiguration) with failurePolicy = Fail (the default), then clients will get failures (and the guarantees of your validations/mutations will be preserved). Otherwise, the api-server will continue to accept changes to your custom resources. When your controller starts again, it will List all objects of your resource and Reconcile them.

---

From: kubeb...@googlegroups.com <kubeb...@googlegroups.com> on behalf of drag...@gmail.com <drag...@gmail.com>
Sent: Thursday, January 21, 2021 3:55 PM
To: kubebuilder <kubeb...@googlegroups.com>
Subject: A question about updating kubebuilder generated deployment

 

--
You received this message because you are subscribed to the Google Groups "kubebuilder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubebuilder...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubebuilder/b9083d9f-543d-4ae4-9753-76a6e156aec2n%40googlegroups.com.

[drag...@gmail.com]

Thanks for the response. It is much clearer now.

If controller runtime stops updating caches on receiving SIGTERM, that effectively means no reconciles would happen even if there are changes to resources. Right? Just wanted to make sure that I understood your reply correctly.

Thanks,

Raghu

[David Sharp]

Right, no new Reconciles will start. But any currently running Reconciles would be allowed to finish.

---

From: kubeb...@googlegroups.com <kubeb...@googlegroups.com> on behalf of drag...@gmail.com <drag...@gmail.com>

Sent: Friday, January 22, 2021 10:45 AM
To: kubebuilder <kubeb...@googlegroups.com>
Subject: Re: A question about updating kubebuilder generated deployment

 

--
You received this message because you are subscribed to the Google Groups "kubebuilder" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubebuilder...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kubebuilder/ec3a6d63-b250-4a3f-bc31-2e533483a829n%40googlegroups.com.

[drag...@gmail.com]

Thanks.