Finding a way to get stable public IP for outbound connections

[Romain Vrignaud]

Hello,

I'm running in a GKE cluster (1.4.x) some application that need to connect to a third party API. This third party API has mandatory IP filtering. So in order to get API authorized I need to declare what are the public IP that I'll use to connect to the API.

My problem is that public IPs of GKE nodes are not stable accros upgrade and it would prevent the use of node autoscaling.

Is there any way to have a stable outbound public IP on GKE ?

Thanks

[tonyli...@gmail.com]

There is no native GKE solution for you. You will be looking to implement a outbound NAT in GCE, and set your Route to point outgoing traffic from the GKE cluster to that instance with a Static IP.

You then use that Static IP as the IP to filter.

[Paris, Eric]

With OpenShift we recommend https://docs.openshift.com/container-platform/3.3/admin_guide/managing_pods.html#admin-guide-controlling-egress-traffic to solve this issue. However it does require that the "egress router pod " be run in a place with a reliable egress IP. So that won't work great on GKE if you cannot maintain even a single node's usage of a reliable egress IP.

Basically that pod just forwards traffic exiting the cluster across a node with a stable IP so it is stable no matter the source. Although, like I said, that does mean that across cluster upgrade the pod needs to land somewhere it has access to a stable egress IP.

On Thu, Jan 19, 2017 at 4:46 PM, <tonyli...@gmail.com> wrote:

There is no native GKE solution for you. You will be looking to implement a outbound NAT in GCE, and set your Route to point outgoing traffic from the GKE cluster to that instance with a Static IP.

You then use that Static IP as the IP to filter.

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Tim Hockin]

For now the only way to get a static IP is to use a custom NAT
gateway. https://cloud.google.com/compute/docs/networking#natgateway

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.

[pa...@qwil.co]

Please correct me if I'm wrong, but it looks like using a simple NAT gateway breaks inbound traffic to the cluster; by configuring a default outbound route that goes through the gateway, it's now impossible to make an inbound connection directly to the cluster, as the response from the cluster gets sent via the gateway (and therefore gets dropped at the sender, as the response IP doesn't match). This is not good, as it breaks GLB Services.

The best workaround I can come up with is to set the NAT gateway route to just apply to the specific remote IP addresses that require a fixed source; that is more brittle than I'd like though, as any change in remote IP will result in my route not matching any more.

Am I missing something?

What I really want is for GKE to be able to assign an existing (or allocate a new) static IP to each node in the cluster, and make an effort to move them across node upgrades and cluster resizing. i.e. as long as I have at least one node in my cluster, there should be a static IP "gke-cluster-node-1".

Cheers,
Paul

[Evan Jones]

It turns out I've just run into a requirement to have a stable outbound IP address as well. In looking into this: I think we will likely some kind of proxy server running outside of Kubernetes. This will allow services "opt in" to this special handling, rather than doing it for everything in the cluster. It seems like the simplest way to make this work.

Honestly, this seems like enough of a rare case that I'm not sure Kubernetes should really support anything "natively" to solve this problem (at least not at the moment when there are more common things that still need work).

[EJ Campbell]

Does it require a single stable IP address or a range? You could possibly have a set of dedicated nodes for you outbound proxy; that way you can still use Kubernetes machinery for deployment, pod lifecycles, etc., but still present a stable CIDR to the outside world.

-EJ

On Monday, May 1, 2017, 7:12:40 PM PDT, Evan Jones <evan....@triggermail.io> wrote:

It turns out I've just run into a requirement to have a stable outbound IP address as well. In looking into this: I think we will likely some kind of proxy server running outside of Kubernetes. This will allow services "opt in" to this special handling, rather than doing it for everything in the cluster. It seems like the simplest way to make this work.

Honestly, this seems like enough of a rare case that I'm not sure Kubernetes should really support anything "natively" to solve this problem (at least not at the moment when there are more common things that still need work).

[Evan Jones]

Thank you! I had forgotten about that feature, since we previously have not needed it. That will absolutely solve our problem, and be much better than needing an "exceptional" thing outside of Kubernetes.

You are correct about what we need: We have a small number of services where their outbound requests need to come from known IPs. This will solve the issue for us.

Thanks again.

[Tony Li]

If you have any example repo to show how this works (what kind of proxy/how to opt in), that would be greatly appreciated!

Tony Li

--
You received this message because you are subscribed to a topic in the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/kubernetes-users/C34yKt0qKtY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Paul Tiplady]

Looks like the taint-based approach will only work on bare metal, where your node IPs are fixed -- or am I missing a detail here?

[Rodrigo Campos]

Your nodes IP are in some subnet, usually. Something like 10.0.0.0/16 or something. If you can accept a subnet, also, that should work.

You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.

[Paul Tiplady]

10.0.0.0/8 is a private subnet, it's not routable from the internet.

[Rodrigo Campos]

Oh, sorry. Then use the public IP of the node. With dedicated nodes, as explained, you can get them. Am I missing something?

In AWS, you can have elastic IP. What is the problem? Manually assign on node crash? Not sure if you will need to do set the labels again in those cases too, depends how your provisioning works

[Evan Jones]

As Rodrigo described, we are using Container Engine. I haven't fully tested this yet, but my plan is to assign "dedicated IPs" to a set of nodes, probably in their own Node Pool as part of the cluster. Those are the IPs used by outbound connections from pods running those nodes, if I recalling correctly from a previous experiment. Then I will use Rodrigo's taint suggestion to schedule Pods on those nodes.

If for whatever reason we need to remove those nodes from that pool, or delete and recreate them, we can move the dedicated IP and taints to new nodes, and the jobs should end up in the right place again.

In short: I'm pretty sure this is going to solve our problem.

Thanks!

[Paul Tiplady]

The public IP is not stable in GKE. You can manually assign a static IP to a GKE node, but then if the node goes away (e.g. your cluster was resized) the IP will be detached, and you'll have to manually reassign. I'd guess this is also true on an AWS managed equivalent like CoreOS's CloudFormation scripts.

[Evan Jones]

Correct, but at least at the moment we aren't using auto-resizing, and I've never seen nodes get removed without us manually taking some action (e.g. upgrading Kubernetes releases or similar). Are there automated events that can delete a VM and remove it, without us having done something? Certainly I've observed machines rebooting, but that also preserves dedicated IPs. I can live with having to take some manual configuration action periodically, if we are changing something with our cluster, but I would like to know if there is something I've overlooked. Thanks!

[Paul Tiplady]

Yes, my reply was more directed to Rodrigo. In my use-case I do resize clusters often (as part of the node upgrade process), so I want a solution that's going to handle that case automatically. The NAT Gateway approach appears to be the best (only?) option that handles all cases seamlessly at this point.

I don't know in which cases a VM could be destroyed, I'd also be interested in seeing an enumeration of those cases. I'm taking a conservative stance as the consequences of dropping traffic through changing source-IP is quite severe in my case, and because I want to keep the process for upgrading the cluster as simple as possible.  From https://cloudplatform.googleblog.com/2015/03/Google-Compute-Engine-uses-Live-Migration-technology-to-service-infrastructure-without-application-downtime.html it sounds like VM termination should not be caused by planned maintenance, but I assume it could be caused by unexpected failures in the datacenter. It doesn't seem reckless to manually set the IPs as part of the upgrade process as you're suggesting.

[giorgio...@beinnova.it]

Hello, I've the same problem described there. I have a GKE cluster and I need to connect to an external service. I find the NAT solution is right for my needs, my cluster resizes automatically. @Paul Tiplady have you config the external NAT? Can you share your experiences? I tried following this guide https://cloud.google.com/compute/docs/vpc/special-configurations#natgateway but seems it doesn't work.

Thanks,
Giorgio

[pa...@qwil.co]

Yes, this is the right approach -- here's a detailed walk-through:

https://github.com/johnlabarge/gke-nat-example

[Giorgio Cerruti]

Thank you Paul!

--

Kind Regards,
Giorgio Cerruti

Beinnova di Giorgio Cerruti

WebSites | IT consultant | Web Developer

Phone: +39 340/87.68.326 :: Skype: beinnova

P.IVA: 10755560017

Visit Beinnova Site

• Giorgio Cerruti

[csal...@devsu.com]

The approach of configuring a NAT works but it has 2 major drawbacks:

1. It creates a single point of failure (if the VM that runs the NAT fails)
2. It's too complex!

In my use case I don't need Auto-scaling enabled right now, so I think it's better to just change the IPs of the VMs to be static. Anyways in the future I know I will need this feature.

Does somebody know if there are there any plans to provide this feature in GKE?

[Tim Hockin]

The GKE team has heard the desire for this and is looking at possible
ways to provide it.

> --
> You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.

> To post to this group, send email to kubernet...@googlegroups.com.

[alex....@zalando.de]

Hi everyone, great thread and special thanks to Paul for the gcloud example!
I'm struggling with the same issue but for AWS, would anyone have an idea how that might work?

Since this thread is on the first Google page for the "egress stable Ip traffic kubernetes" I think this might be useful for others.

Thanks!

[csal...@devsu.com]

Hi, are there any updates on this feature? Is it on the roadmap of the GKE team or it hasn't been planned yet?

[Tim Hockin]

AFAIK we need CloudNAT to become available, at which point we can use
it pretty much transparently.

[mi...@percy.io]

Evan,

Did you figure out a way to assign reserved static IP addresses to a few specific nodes in a GKE pool?

We are also fine with doing this manually for a couple of specific nodes for the time being (rather than building a NAT gateway), but I cannot find reliable information about how to assign a reserved static IP to a GKE node.

Cheers,
Mike

[mi...@percy.io]

An update: I was able to do this with the standard add-access-config mechanism here:
https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address

No guarantees around when GKE will rebuild those nodes and lose the node IPs, but it works for now.

[Tim Hockin]

You can build that as a controller that runs in-cluster, picks one of the nodes, and assigns the static IP.  It will still be racy, though, in that it will never be instantaneous.

--

You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-use...@googlegroups.com.

To post to this group, send email to kubernet...@googlegroups.com.