Latency between Master and Nodes

[Kevin Heatwole]

Is it okay to run the Kubernetes Master in a VM in some cloud provider and host the Nodes on bare metal servers in a potentially distant datacenter. How much latency in the network between the Master and the cluster Nodes is tolerable (for production deploy)?

Or, should all of Kubernetes be hosted in servers within the same dc?

I'm not clear on how fast the connection between Master and the cluster nodes needs to be. Can Kubernetes work well with 1 second latencies? Or, should pings be sub-millisecond?

[Brendan Burns]

Yes, I think this should work mostly.  We run watch to do hanging gets (instead of polling) from kubelet to apiserver, but I bet it would work fine even with 1 second latency (as long as the connections were reliable)

Please let us know what you try and how it works out!

--brendan

--
You received this message because you are subscribed to the Google Groups "Containers at Google" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-contain...@googlegroups.com.
To post to this group, send email to google-c...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-containers.
For more options, visit https://groups.google.com/d/optout.

[Kevin Heatwole]

What happens to the cluster if the Master goes offline for an hour or two? That is, the Nodes can no longer contact the Master over the network. Would this cause any issues in production? I would think the Pods and Services would continue operating, except that no reconfiguration/monitoring would go on during the downtime. 

And, do the cluster nodes themselves need to contact each other if they are located in multiple data centers to maintain the cluster's operation?

I'm just trying to learn about Kubernetes in production and what the issues would be if I did decide to distribute the cluster geographically (over a WAN that I have no real control for High Availabiltiy).

[Brendan Burns]

On Mon, Aug 17, 2015 at 2:32 PM, Kevin Heatwole <ktwa...@gmail.com> wrote:

What happens to the cluster if the Master goes offline for an hour or two? That is, the Nodes can no longer contact the Master over the network. Would this cause any issues in production? I would think the Pods and Services would continue operating, except that no reconfiguration/monitoring would go on during the downtime. 

Yes, the cluster should coast if it can't talk to the master.

 

And, do the cluster nodes themselves need to contact each other if they are located in multiple data centers to maintain the cluster's operation?

The nodes themselves don't really contact each other.  The pods on those nodes may, but they too, use the datacenter network without talking to the master.

[Kevin Heatwole]

One last question... Can the Master be both the Master and one of the Nodes in the cluster? That is, can the scheduler consider the Master server when launching a Pod as a possible server for the new instance?

[David Oppenheimer]

On Mon, Aug 17, 2015 at 2:56 PM, Kevin Heatwole <ktwa...@gmail.com> wrote:

One last question... Can the Master be both the Master and one of the Nodes in the cluster? That is, can the scheduler consider the Master server when launching a Pod as a possible server for the new instance?

No, currently the master is a dedicated node.

 

On Monday, August 17, 2015 at 3:29:11 PM UTC-4, Kevin Heatwole wrote:

Is it okay to run the Kubernetes Master in a VM in some cloud provider and host the Nodes on bare metal servers in a potentially distant datacenter. How much latency in the network between the Master and the cluster Nodes is tolerable (for production deploy)?

Or, should all of Kubernetes be hosted in servers within the same dc?

I'm not clear on how fast the connection between Master and the cluster nodes needs to be. Can Kubernetes work well with 1 second latencies? Or, should pings be sub-millisecond?

--

[Brendan Burns]

That's not strictly true.  It is possible, its just not the default setup, and if you do it, its possible that work scheduled onto the master will starve resources (or even oom) master components, so its worth being very careful and understand what you are doing.

Brendan

[Marek Grabowski]

If master disconnects from the cluster for a prolonged period of time everything will be running until the master reconnects. If the master were running all this time, and noticed that it cannot connect to Nodes it will start evicting Pods in it's local etcd store. After the master reconnects Kubelets will notice that all Pods that are running on them are gone, and will proceed with killing everything that moves. As long as down time is shorter than 5 minutes (default value for pod-eviction-timeout flag) you'll be OK, if it's longer you'll end up in trouble.