CoreOS VxLAN, OVS / Flannel / Kubernates / Docker Container network questions

[Jonathan Weeks]

As we begin to look at migrating services to a CoreOS/Docker world, we will have clients outside the Docker world accessing services via a typical load balancer (e.g. F5 BigIP, NetScaler, Zeus), and are trying to understand the networking implications for high throughput, low latency, robust production services.

Good resources to date have been:

https://github.com/coreos/flannel/issues/18

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/design/networking.md

https://github.com/GoogleCloudPlatform/kubernetes/issues/1107

https://github.com/GoogleCloudPlatform/kubernetes/issues/1307

http://weaveblog.com/2014/11/13/life-and-docker-networking/

http://cloudn1n3.blogspot.com/2014/10/networking-options-for-docker.html

https://github.com/docker/docker/issues/8951

https://sreeninet.wordpress.com/2015/01/01/docker-networking-part2/

http://fbevmware.blogspot.com/2013/12/coupling-docker-and-open-vswitch.html

http://containerops.org/2014/11/11/future-of-docker-networking/

While the current Kubernates Services model with distributed internal IPTables NAT, etc. is great for internal clients, the solution is a bit lacking for incoming external traffic in a few areas:

• Latency impact of multiple hops and multiple veth traversals
• Load balancing is primitive/immature/early (e.g. round-robin only)
• Still need to to coordinate the external IPs in any case

We would like to have as few "hops" and as little user-space packet handling as possible, to minimize latency and maximize robustness, and are willing to coordinate/synchronize pod provisioning/lifecycle and networking details as necessary, assuming the below is viable/possible.

So my questions are:

• Does flannel support injection of two veths into a Docker container? (The idea would be one would support native Kubernetes traffic, and one would be for inbound external traffic from a LB outside the Kubernetes deployment)
• If so, is an OpenVswitch port/topology possible? e.g. the last diagram at the bottom of: http://www.opencloudblog.com/?p=66, or at least bare VxLAN support for this dual veth topology?

Thanks in advance,

-Jonathan

[eugene.y...@coreos.com]

On Tuesday, February 10, 2015 at 9:40:15 AM UTC-8, Jonathan Weeks wrote:

As we begin to look at migrating services to a CoreOS/Docker world, we will have clients outside the Docker world accessing services via a typical load balancer (e.g. F5 BigIP, NetScaler, Zeus), and are trying to understand the networking implications for high throughput, low latency, robust production services.

<...snip...>

 

So my questions are:

• Does flannel support injection of two veths into a Docker container? (The idea would be one would support native Kubernetes traffic, and one would be for inbound external traffic from a LB outside the Kubernetes deployment)

flannel performs two functions: 

1. Allocates a subnet to a host

2. Routes packets on these addresses between hosts. The packets to/from containers pass through the host's routing layer and either through an overlay interface: e.g. VXLAN) or routed natively by the host to their destination (host-gw and alloc backends).

The allocated subnet is handed over to Docker which creates the linux bridge and veth pairs. flannel itself does not manage veths/ports of the containers. Of course you can take Docker out of networking business and setup veths yourself (https://docs.docker.com/articles/networking/#building-a-point-to-point-connection). You can then use this in conjunction with flannel by handing out individual IPs out of the subnet that flannel provides. If you go this route, use "docker create" instead of "docker run" as described in that article to avoid a race.

 

• If so, is an OpenVswitch port/topology possible? e.g. the last diagram at the bottom of: http://www.opencloudblog.com/?p=66, or at least bare VxLAN support for this dual veth topology?

Not out of the box. If you setup container networking yourself (see prev question), you could plug an internal ovs port into the container. If you assign OVS an IP so it acts as a gateway, you should be able to use flannel's vxlan backend. However flannel was not made for L2 -- you can't plug VXLAN interface that flannel creates into OVS.

Best,

Eugene