OS patching in Kubernetes cluster

[ijunaid...@gmail.com]

Im about to perform an OS upgrade on a K8s cluster and was hoping to know the best practices for doing so.

I heard from others that a few bad experiences happened when one host node was upgraded ( OS patch applied ) and then the Master was unable to see it ( some JSON incompatibility issue ). This ended up causing an outage.

So what is the best way to upgrade an entire K8s setup.

I have 1 K8s Master and 5 hosts on which my pods are deployed. I would prefer a rolling update. If there is anyother solution, id be glad to hear about it.

[Rodrigo Campos]

You need to be careful between which releases you update, it should always work
from one release to the next.

I think upgrading the master, then the nodes one by one, is what tools like kops
do. It works fine :)

[Junaid Subhani]

I see what you say and understand it. But my requirement here is not to upgrade the Kubernetes version. 

It is simply to apply OS patches on nodes of an an already running cluster with minimal downtime for the application.

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/oZHghV-k_WU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Rodrigo Campos]

On Mon, Mar 20, 2017 at 02:06:38PM -0400, Junaid Subhani wrote:
> I see what you say and understand it. But my requirement here is not to
> upgrade the Kubernetes version.
>
> It is simply to apply OS patches on nodes of an an already running cluster
> with minimal downtime for the application.

Oh. Then just use kubectl drain to move pods away from the node, upgrade and
reboot (if kernel or something needing reboot changes), then add the node again
with kubectl uncordon.

It depends on what you need to upgrade, you can also not do a drain on the node
at all

[Adieu]

To rolling update the nodes is easy. Just adding more nodes with
updated os to the cluster and delete old nodes after you do a drain on
them.

If you want to do rolling update on the master, you have to solve the
following problems:

- How to migrate etcd data to the new master
- How to keep the apiserver address static for node servers

There are many solutions to the problems. For example, you could setup
a dedicated etcd cluster and put a proxy in front of the master
server.

I would recommend to use the [bootkube] tool to setup a Kubernetes
cluster with self-hosted-etcd. And create a loadbalancer for the
apiserver.

We had put up an [example] in our cluster management tool for an aws
setup. Hope it's useful.

[bootkube]: https://github.com/kubernetes-incubator/bootkube
[example]: https://github.com/kubeup/archon/tree/master/example/k8s-bootkube