Pod termination order on drained nodes

[Pol Llovet]

Two questions:

1. When a node is drained, how is the order of pod termination determined?

2. Is it possible to prevent daemonsets from being terminated until after all other pods are completely terminated.

The situation is that on scale-in, we have active pods still terminating (wrapping up current requests), but the daemonsets are terminated immediately on the drained node. This can cause two issues: 1) if the daemonset is providing critical dependency service for the other terminating pods, those pods start to fail.  2) the failures are invisible because observability daemonsets are also terminated.

Any advice here?  Also I am curious to try to craft an "anatomy of scale-in and scale-out events", to get very deep on what the process is when nodes are scaled in and out. Any advice on where to start with that would be nice (any prior art especially). 

-pol

[Pol Llovet]

I think I found the answer to the second question: https://kubernetes.io/docs/concepts/cluster-administration/node-shutdown/

>  Note that by default, both configuration options described below, shutdownGracePeriod and shutdownGracePeriodCriticalPods, are set to zero, thus not activating the graceful node shutdown functionality.

So, we need to see if we are able to enable this functionality, and that should ensure that our `system-node-critical` services are terminated after the other services.  

I am not clear what happens if the gracefulGracePeriod is reached but there are still running pods (or if the configured shutdown grace period for the individual pods is greater than the shutdownGracePeriod).  Does kubelet force kill the pods, or does it proceed to critical pod termination?

thanks,

pol