Kubernetes master availability and CoreOS updates

[Jimmy Cuadra]

Hello,

CoreOS has been recommending Kubernetes as a tool to use for container orchestration. Kubernetes has a master/slave architecture with only one master node, and according to their roadmap, high availability masters will not be a focus of development until after 1.0. While this seems like a strange choice to me, my real question is how this lack of high availability will be affected by CoreOS's update system. It's a philosophy of CoreOS that you shouldn't assume that any single node will ever be healthy and should plan your services accordingly. How, then, does a Kubernetes-based system respond when the Kubernetes master node goes down for a CoreOS update?

Thanks,

Jimmy

[Oliver Soell]

We schedule the kube master components with fleet, and then register the kube master with an ELB in AWS so there's always a stable dns name for it. The kubelet and kube-proxy components can point to the ELB for the apiserver. If you don't have an api-driven load-balancer at your disposal you could do some tricks to bind a roaming IP to your kube master or something similar.

Another related issue: kubernetes doesn't yet have the concept of taking a machine out for maintenance (gracefully moving all the deployed pods to another node) which some might consider a pre-requisite for using locksmithd to update your CoreOS nodes. It'll happen eventually :)

-o

[Jimmy Cuadra]

Unless I misunderstood, adding an ELB in front of a single Kubernetes master doesn't provide failover/increased availability. What I'm wondering is what happens to the entire Kubernetes cluster when the master goes down, since Kubernetes doesn't seem to support a multi-master setup. On a more traditional Linux distribution you could naively just hope that your master node doesn't crash, but in CoreOS it's guaranteed that the node will be down sometimes for a system update, so you really do need to plan for it.

[Oliver Soell]

Fleet will move the master units over to a new host when the master node does down, and since it's behind an ELB, the kubelet and kube-proxy processes can find the new master.

When the master is down, k8s will coast. No pods get stopped, but the management that the master processes provide will be absent:

https://groups.google.com/forum/#!starred/google-containers/GgGDnHjMSo4

cheers,

-o

[Vaidas Jablonskis]

Hi Jimmy,

I had the same issue. Some environments where I deploy Kubernetes don't have ELBs or other fancy services. I wrote a little service which basically follows kubernetes API server and creates a proxy on each CoreOS node, so your API is always available on:

localhost:8081

Have a look here: https://github.com/vaijab/kube-apiproxy

Thanks,

Vaidas