Flannel Networking

[Scott Crisenbery]

I've been looking into using an overlay network for use with Kubernetes, and I am still trying to wrap my head around a few pieces of the networking.

More specifically, I am wondering how things within a Flannel network (containers, pods, etc.) interact with things outside of the network. For example, if I wanted my application to be exposed on a public IP address, how do packets get from there to the flannel IP of the destination pod?

Also, where do services come into play with all of that?

Thanks!

-Scott

[Brandon Philips]

Hello Scott-

Kubernetes's solution to the "getting traffic into the cluster" problem with the concept of a NodePort. Basically, a service with type of NodePort will expose a known port on every "worker node" in the cluster. This can then be used to route traffic to the cluster from an F5, ELB, etc.

Docs here: http://kubernetes.io/v1.0/docs/user-guide/services.html#type-nodeport

Cheers,

Brandon

--
You received this message because you are subscribed to the Google Groups "CoreOS User" group.
To unsubscribe from this group and stop receiving emails from it, send an email to coreos-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Griessen]

On 11/02/2015 04:40 PM, Brandon Philips wrote:
> Kubernetes's solution to the "getting traffic into the cluster" problem with the concept of a NodePort. Basically, a service with
> type of NodePort will expose a known port on every "worker node" in the cluster. This can then be used to route traffic to the
> cluster from an F5, ELB, etc.
>

How do you stop the load balancer from becoming a single point of failure? I will be putting some worker nodes
at separate physical providers. No private network connects them since they are with different companies as well as
physically separate.

Is it necessary to do all of this with connecting VPN tunnels? If so, how do you stop the VPN tunnel setup from becoming a single
point of failure?

I'd like to see a secure way of doing clusters' etcd2 communication relying on redundant DNS of some kind, even if it meant DNS
publishing the IP of an etcd2 node. Workers naturally have a public facing IP, but they should have the same secure and redundant
way of communicating between the etcd2 nodes.

Can this all be done now without a single point of failure?

[Scott Crisenbery]

Maybe a bit more to the point: Is there a potential for conflict between the network at large and the subnet specified in Flannel? If I have some VMs on another area of the network that use a particular range of IPs, would it matter if Flannel was also using the same IP range, or is it segmented?

[eugene.y...@coreos.com]

It's easiest if flannel network range does not overlap with any other IPs on the network. But the requirement is that

- the nodes that run the flanneld agent are not assigned a flannel address

- nor need to communicate with other nodes that have those addresses

- nor have a route entry that is more specific than flannel network.

So for example, say your flannel network has been configured with 10.5.0.0/16. Suppose you have a VM that runs a flanneld agent and has its host interface with an IP of 10.1.2.3. This IP is not in the flannel network range so it's OK. However you also have to make sure that it does not have some route like 10.5.128.0/17 since it would be narrower than flannel network. As such, packets would take that route and not the 10.5.0.0/16 that flannel has set up.

[Scott Crisenbery]

Ok, so let me set up a scenario. Let's say I have two host in a Kubernetes cluster. The host IPs are 10.0.0.2 and 10.0.03 in a /24 subnet. In the Kubernetes cluster, there are two services running, a service for a website at 10.0.1.2 and a service for some type of database that the website references at 10.0.1.3 in a /24 subnet. Flannel is configured to use the 10.0.2.0/24 network. There are two Kubernetes pods, one for the website and one for the database, and they need to be able to talk to each other. The website should be accessible publicly on 1.1.1.1 which routes to our internal network.

I want the website to display a page that requires data from the database. So I go to the webpage at 1.1.1.1. From there, how does my request get fulfilled? What does that look like in terms of all of these things communicating with each other?

[eugene.y...@coreos.com]

Using a /24 for flannel network is a bit small since it carves it up to give each host a slice. But let's use it for this example anyway. So you have 10.0.2.0/24 as flannel network and configure flannel to give out /28 to each node.

10.0.0.2 host gets assigned 10.0.2.32/28

10.0.0.3 host gets assigned 10.0.2.80/28

- On a host with 1.1.1.1 (or host that it's NAT'ed to), you want to set up something like HAProxy and add 10.0.0.2 and 10.0.0.3 as upstream servers for port 80. (you can add as many nodes running k8s as you want).

- You then set up website service at 10.0.1.2 and type NodePort. Since Kubernetes is running on both 10.0.0.2 and 10.0.0.3 hosts, any traffic coming on port 80 to those nodes will be redirected to a website pod(s) that is currently running the website. When a user puts in 1.1.1.1:80 into the browser, the request goes to HAProxy which may forward it to 10.0.0.3:80. 10.0.0.3 will then forward it to 10.0.0.2 that happens to be running the website pod now.

- Your database is running inside a pod on 10.0.0.3 host. The pod IP happens to be 10.0.2.82 but is exposed as a Kubernetes service as 10.0.1.3. When your website pod contacts the database, it uses the service IP -- 10.0.1.3 but gets redirected to 10.0.2.82. It then traverses the flannel network to get to the database pod.

I hope this all makes sense.

[Scott Crisenbery]

It's making sense, there's just one or two things missing. So it sounds like abstractions like services and pods aren't actually routable across the network as a whole, just between themselves?

Also, in your the second point, the traffic is forwarded to 10.0.0.2 which has the pod running the website. Where does the traffic go from there to actually reach the pod? Am I right in thinking that the 10.0.0.2 host then routes it to the service IP address? How is that happening?

Thanks for the answers, though. It is starting to come together for me.

[eugene.y...@coreos.com]

On Tuesday, November 3, 2015 at 3:09:48 PM UTC-8, Scott Crisenbery wrote:

It's making sense, there's just one or two things missing. So it sounds like abstractions like services and pods aren't actually routable across the network as a whole, just between themselves?

Sort of. It depends on the concrete environment. For example, if k8s is used in a "cloud provider" mode with AWS (it also supports GCE), it will give Pods IPs from AWS VPC (programming VPC router as necessary). It will also expose service IPs via ELB. This way the pods and services are routeable across the whole AWS VPC. But in the fallback scenario, Pod IPs are handled by things like flannel and are only reachable between k8s nodes. It also means that service IPs are only available on k8s nodes and then the outside world needs to "get in" via NodePort.

Also, in your the second point, the traffic is forwarded to 10.0.0.2 which has the pod running the website. Where does the traffic go from there to actually reach the pod? Am I right in thinking that the 10.0.0.2 host then routes it to the service IP address? How is that happening?

Correct. The NodePort (10.0.0.2:80 although by default it won't be 80) forwards it to the service IP (10.0.1.2) via iptables DNAT rule. The service IP will match another iptables rule that will either REDIRECT it user space proxy that will forward it to the pod (based on label query) or do the same in the kernel using more iptables rules (I'm actually not sure if kernel based load balancing is used by default now). See http://kubernetes.io/v1.0/docs/user-guide/services.html for more info about NodePort.

[Scott Crisenbery]

I'm thinking specifically of a bare-metal environment in this case. To your second point, can you explain that same situation if there was another section of the network with the same subnet that flannel is using?

On another note, does Docker 1.9's networking capabilities that just hit a stable release work the same way and would it suffer from the same problems?

[eugene.y...@coreos.com]

On Wednesday, November 4, 2015 at 12:11:00 PM UTC-8, Scott Crisenbery wrote:

I'm thinking specifically of a bare-metal environment in this case. To your second point, can you explain that same situation if there was another section of the network with the same subnet that flannel is using?

As long as all the Kubernetes nodes don't have a more specific route for that other section of the network, the packets will flow through flannel. From HAProxy to k8s node, the flannel IPs don't matter as it's just using normal host IP. From k8s node to the pod, it's over the flannel network so the usual rules apply. I'm not sure I fully understand your question though. 

[Scott Crisenbery]

I think what I am missing is: why does that more specific route take precedence over what flannel is doing?

[eugene.y...@coreos.com]

Suppose you're on a node that has flanneld running. You want to send a packet from a Docker container (10.1.5.3/24) running on that host to another container (10.1.8.6/24) on a different host. The packets leaves docker0 bridge and consults the routing table on the host. There will be a route for "10.1.0.0/16 -> flannel.1" and the packet will be directed to flannel.1 interface, get encapsulated and sent to the proper host. But if there's some route that looks like "10.1.0.0/17 -> eth5", the packet will take that route b/c it's more specific.