Pod cannot access external services in VPC-peered private network

[Manjiri Gadagkar]

Hello!

My goal is to set up a cluster in Google Container Engine in a private network (nw1:subnet1) that can access services in another private network (nw2:subnet2). The latter (subnet2) is an existing deployment of services running on Google Compute Engine instances (VMs). I have set up VPC Peering between nw1 and nw2.

Master version: 1.5.7
nw1:
        subnet1: 10.224.0.0/11
        Container address range: 10.164.0.0/14
        PodCIDR: 10.164.2.0/24
nw2:
        subnet2: 10.96.0.0/24    (existing)
        subnet3: 192.168.0.0/24     (testing)
        subnet4: 172.16.0.0/24      (testing)

I found that subnet3 and subnet4 are accessible from the pods. My problem is that I am not able to access subnet2 from the pods.

I looked at the iptables on the GKE node, and found this rule:

    -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: SNAT for outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j MASQUERADE

I added a SNAT rule for subnet2 destination as follows. I don't know if it breaks something else, but it appears to work around the issue.

 -A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
 -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT for outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j MASQUERADE
 -A POSTROUTING ! -d 10.0.0.0/8 -m comment --comment "kubenet: SNAT for outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j MASQUERADE
 -A POSTROUTING -s 127.0.0.0/8 -o cbr0 -m comment --comment "SNAT for localhost access to hostports" -j MASQUERADE

Is my use-case supported in GKE? Is there a configuration parameter that I could tweak? Anything else I could try?

Thanks in advance!
-Manjiri

[Tim Hockin]

The VPC peerings are dropping non-VM packets. That's unfortunately
the state of things for now. The solution, as you show here, is to
masquerade. The downside is that you lose the original pod's IP in
the packet.

We've added support for IP aliases, which should be beta in OSS in
v1.7 (and probably alpha in GKE at first), which will remove this
restriction. We're also working on a more configurable way to manage
the masquerade rules, so you can tweak it in small way, instead of the
single kubelet flag we have today.

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

[Manjiri Gadagkar]

Tim,

Thank you for a quick response!

Losing the original pod's IP is not a big concern for me, so this track will be OK. My hope is to be able to use GKE without any customizations/workarounds/hacks. Hence, the v1.7 changes are welcome!

1. Can you confirm that my workaround will not break anything (as far as you know) ?
2. Can you point me to docs on "IP aliases support" that you mention below?
3. The problem I describe here is the pod-to-vpcsvc access. The other side of this coin is vpcsvc-to-k8ssvc. There, I used a Google Internal Load Balancer** whose IP address is in subnet1 and accessible from subnet2 (as expected). This is the complete use-case: "external" service is private and K8S service is exposed on a 10.x.x.x network. I am curious about the thoughts around this use-case in terms of abstraction/configuration options.

** Internal Load Balancer (ILB): I have a NodePort service (portA) whose node:portB are the backend. The ILB only works if the frontend port is portB i.e. same as backend. I suppose this because of the "forwarder, not proxy" architecture of ILB. My goal is to expose this service to VMs in subnet2 on portA. I am able to do that by running HAproxy in subnet1. Is service type=InternalLoadBalancer (or equivalent with annotation or whatever) on the GKE roadmap? Will the TCP+proxy mode be supported by the ILB? I bring this up here because my use-case as a whole needs this piece as well and I would like to understand if the GKE roadmap addresses the masquerade piece separately or this entire use-case is going to be supported.

Thank you!
-Manjiri

[Tim Hockin]

On Sun, May 21, 2017 at 9:28 AM, Manjiri Gadagkar
<manjiri....@ruckuswireless.com> wrote:
> Tim,
>
> Thank you for a quick response!
>
> Losing the original pod's IP is not a big concern for me, so this track will
> be OK. My hope is to be able to use GKE without any
> customizations/workarounds/hacks. Hence, the v1.7 changes are welcome!

You and me both.

> 1. Can you confirm that my workaround will not break anything (as far as you
> know) ?

Should be fine.

> 2. Can you point me to docs on "IP aliases support" that you mention below?

https://github.com/kubernetes/kubernetes/pull/42147

It's gone beta now.

> 3. The problem I describe here is the pod-to-vpcsvc access. The other side
> of this coin is vpcsvc-to-k8ssvc. There, I used a Google Internal Load
> Balancer** whose IP address is in subnet1 and accessible from subnet2 (as
> expected). This is the complete use-case: "external" service is private and
> K8S service is exposed on a 10.x.x.x network. I am curious about the
> thoughts around this use-case in terms of abstraction/configuration options.

ILB should be in 1.7 also.

[Manjiri Gadagkar]

To summarize the resolution:

Solution: available in v1.7
Workaround: iptables rule on k8s nodes:

 -A POSTROUTING -d 10.96.0.0/24 -m comment --comment "manjiri: SNAT for outbound traffic from cluster" -m addrtype ! --dst-type LOCAL -j MASQUERADE

Thank you, Tim!

-Manjiri