proposal: Node eviction controller

[yqu...@redhat.com]

I would like to present a node eviction controller (for maintenance purposes) project now started under Kubevirt:

https://github.com/kubevirt/node-maintenance-operator

Our motivation for the controller:

Since drain/eviction process should not depend on the sysadmin's laptop. The idea is to write a controller that would manage the maintenance mode from server side.

We also want the ability to fire node maintenance from UI - which couldn't be achieved by invoking kubectl drain command (not possible).

The purpose of the controller (wrapped as an operator in the current project):

The purpose of the node-maintenance-operator is to watch for new or deleted custom resources called NodeMaintenance which indicate that a node in the cluster should either:

• NodeMaintenance CR created: move node into maintenance, cordon the node - set it as unschedulable and evict the pods (which can be evicted) from that node.
• NodeMaintenance CR deleted: remove pod from maintenance and uncordon the node - set it as schedulable

Via UI we will be able to create the CRs mentioned above and start a procedure that resembles kubectl drain (with fine tuning).

Some notes:

• Some parts in the current implementation utilize the drain pkg (moved to a library in https://github.com/kubernetes/kubernetes/pull/72827)
  
• Rest of the drain flow was taken from cmd pkg and adjusted
  
• There is still work to be done on parallel runs and other aspects.

Questions:

• Are there any plans in the future on drain functionality other then the one exits ?
  
• Would such a controller be a interest for the broader public  ?

Finally, i would be more than happy to get some opinions and points on related work, and present this solution in one of the coming SIG's meetings.

Thanks,

Yanir Quinn

[David Oppenheimer]

https://github.com/kubernetes/kubernetes/issues/25625

may be relevant

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-node" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-...@googlegroups.com.
To post to this group, send email to kubernete...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kubernetes-sig-node/0da1cb27-ab10-47dc-aef9-8038254ad69a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lei Zhang]

Hi Yanir

I'm looking forward to it!

We are proposing a Node Eviction Policy to kubelet. In this KEP, we expect users can set kubelet to only make eviction suggestion as a CRD instead doing hard eviction immediately, and then, let an outside controller (or, a build-in controller with configurable policy) to make final decision and do eviction. 

The reason we want this feature is: currently eviction are kinda "dangerous" in large prod clusters, we hope there can be double check and control for the eviction process since it is designed to delete prod users' applications.

As you can see here, the node-maintenance-operator seems to be a good outside controller candidate.

---

Harry Zhang

[fdeu...@redhat.com]

Hey Lei,

in which KEP actually? :)

- fabian