Making a pod network routable within AWS

[tris...@cloudera.com]

Hi All - 

Has anybody managed to find a configuration within AWS that makes the pod network fully routable to corporate networks that are connected via a AWS virtual private gateway?  My understanding is there are two obvious approaches:

1. Carve out part of the subnet CIDR and create a routing rule that maps the pod network cidr to a kubernetes gateway node running Flannel/Calico/Weave.  As far as I can tell this doesn't work because AWS doesn't allow routing rules that are more specific that than the subnet local rule.

2. Use a cidr that is external to the subnet and create a routing rule between that cidr and a kubernetes node or alternatively use something like the flannel AWS backend to do this automatically.  My understanding is that it is still impossible to make this cidr fully routable externally because AWS will not allow traffic into the subnet targeting an external cidr.

Any other ideas that don't involve running a custom VPN gateway within the subnet?

Thanks,

Tristan

[Vicente De Luca]

What I am doing here, since AWS limit the routing table up to 100 entries (can't run flannel on aws-vpc mode), was to spun up a "routing instance".

We run flannel in host-gw mode. The routing instance is basically a linux instance part of flannel and with ip_forward = 1  -- don't forget to disable src/dst checking on the ENI
since flannel install all the pod prefixes on this linux routing table, it'll know how to properly forward the packets destined to a pod network.

on AWS routing table you have your VPN you can create a single static route for the aggregate pod network (i.e: 10.200.0.0/16) pointing to the routing instance ENI.
the routing instance ENI is also the default route for the k8s nodes subnet. A secondary ENI connected to a public subnet and doing iptables SNAT ensure k8s nodes and pods internet packets are properly translated.
 
we have been staging this setup for some time and its working fine for a single availability zone.
side note:  for own datacenters a similar approach is used, but the routing instance is injecting flannel learned routes to the BGP aggregate router, so its not taking part of forwarding traffic, but just acting as a route reflector.


--vicente

tris...@cloudera.com wrote:

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

[Tristan Zajonc]

Vicente,

Thanks.  I'm not sure I 100% understand this line:

"on AWS routing table you have your VPN you can create a single static route for the aggregate pod network (i.e: 10.200.0.0/16) pointing to the routing instance ENI."

If I add 10.200.0.0/16 -> ENI on my subnet routing table, my understanding is this route will only work within the subnet but not externally (other subnets or my on-premise corporate network that's connected via a AWS customer gateway).  Are you referring do a different routing table?

Thanks,

Tristan

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Vicente De Luca]

Hi Tristan

I see your point.

so within your VPC all of subnets that needs access to pod network will need the route created.
in regards to the VPN, I presume it also uses BGP so Amazon will be advertising the VPC range -- lets say 10.0.0.0/16
that doesn't include the pod network (10.200.0.0/16)..

in theory if you add a route for the pod networks (10.200.0.0/16) at your on-premise gateway using the AWS tunnels endpoint as gateway it should work --  But I didn't test this case
make sure you have the on-premise network route propagated to your k8s nodes subnet where the routing instance lives as well.

in case AWS VPN don't forward packets to pod network in the above scenario then I see no much way around than building a custom vpn instance.
interested in testing this because I might hit this scenario anytime soon.

--vicente

[Bryan Boreham]

If you use Weave Net you can connect up Weave nodes on the corporate network, either on every machine that needs connectivity or on a smaller number and set up routing rules on your corporate network.  Weave will do the multi-hop routing between its own nodes as necessary.

Bryan (I work on Weave Net)

[Casey Davenport]

Hi Tristan,

To throw another option out there (though I haven't done this myself).  You might be able to take advantage of Calico's BGP capabilities to peer with your virtual private gateway device.  This would allow the VGW to route traffic directly to individual pod IP addresses.

- Casey

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.

[Tristan Zajonc]

Thanks Bryan and Casey.

Casey:  This would be ideal, but I don't believe BGP will work within AWS.  BGP routes do get broadcast to the gateway from the corporate network, but AWS does not make it possible to broadcast foreign looking CIDRs like the pod network through the customer gateway to the corporate network, as far as I can tell.   Moreover, AWS will block incoming traffic if you set up static routes.  They appear to do this to avoid customers using the AWS network as a backbone.  I've opened an issue with AWS support but haven't heard of a solution except to run a custom VPN tunnel.

Bryan:  Using something like weave as a tunnel is does sound possible.  I could put one weave node on my corporate network along with a rule to route all pod traffic through that node.  I was hoping to avoid a custom tunnel setup though since I wanted something without major performance overhead and HA.  The ideal solution would be something like Casey proposes, but I don't think I can make that work within AWS.

I very much appreciate the responses. 

Tristan

On Tue, Sep 6, 2016 at 1:59 PM, Casey Davenport <ca...@tigera.io> wrote:

Hi Tristan,

To throw another option out there (though I haven't done this myself).  You might be able to take advantage of Calico's BGP capabilities to peer with your virtual private gateway device.  This would allow the VGW to route traffic directly to individual pod IP addresses.

- Casey

On Wed, Aug 31, 2016, 12:26 PM <tris...@cloudera.com> wrote:

Hi All - 

Has anybody managed to find a configuration within AWS that makes the pod network fully routable to corporate networks that are connected via a AWS virtual private gateway?  My understanding is there are two obvious approaches:

1. Carve out part of the subnet CIDR and create a routing rule that maps the pod network cidr to a kubernetes gateway node running Flannel/Calico/Weave.  As far as I can tell this doesn't work because AWS doesn't allow routing rules that are more specific that than the subnet local rule.

2. Use a cidr that is external to the subnet and create a routing rule between that cidr and a kubernetes node or alternatively use something like the flannel AWS backend to do this automatically.  My understanding is that it is still impossible to make this cidr fully routable externally because AWS will not allow traffic into the subnet targeting an external cidr.

Any other ideas that don't involve running a custom VPN gateway within the subnet?

Thanks,

Tristan

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

[Vicente De Luca]

Tristan,

AWS recently release a "transit VPC". 
Wondering if we could use it for running a k8s cluster and tho can export our POD networks to any VPN endpoint over BGP.

https://aws.amazon.com/blogs/aws/aws-solution-transit-vpc/



Tristan Zajonc wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.