how to expose many pods/deployments on single IP in GKE?

[Wiktor Tomczak]

Hello,

I'd like to run two different containers, each with a different executable, on the same specific external IP address in Google Container Engine (GKE). One executable is a web server and the other is a web client. Requests from the client should originate from the same IP that the server listens on.

So far I've been able to get the client executable to run on an external IP address that I had reserved via Networking > External IP addresses in the cloud console [1]. The address is assigned to my only VM instance, so IIUC all containers running there use this address for their outgoing IP traffic. I can confirm the external IP address eg. by attaching to a running container [2] and fetching checkip.dyndns.org from there [3].

OTOH, I haven't been able to expose the server executable on the same IP address. I tried both .yaml service configuration and kubectl expose but can't get either to work. What should be the value of my service's spec.type? If I set type = LoadBalancer and LoadBalancerIP = <my external IP above>, load balancer creation fails because the IP is reported as already used. If I set type = NodePort, the service gets created, but requests to <my external IP>:8080 aren't forwarded to the server executable (configured to listen on port 8080).

I'd appreciate any help. 

Wiktor

[1] https://console.cloud.google.com/networking/addresses/list

[2] kubectl run -ti <pod name> -- /bin/bash

[3] curl checkip.dyndns.org

[Wiktor Tomczak]

I've had partial success. The solution to expose the server on a reserved external IP address assigned to a VM instance is to 

1. Create a service of type NodePort and with an externalIP set to my reserved external IP.

2. Allow inbound tcp traffic by adding a firewall rule [4]

[4] https://console.cloud.google.com/networking/firewalls/

However, this exposes the server on <external IP>:<nodePort>. How can I expose it on <external IP>:80 ?

[Rodrigo Campos]

Don't know about Google cloud, but if you use the service aa type load balancer, doesn't that create a load balancer with static IP address?

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-users.
For more options, visit https://groups.google.com/d/optout.

[Tim Hockin]

What you're asking for isn't really well supported. The problem is
that the source IP for your client is the VM's IP, and if that pod
should ever get moved, that IP will change. Kubernetes Services are
designed to avoid that, but they can't easily handle client IP.

If you really want this behavior, try this:
- add a hostPort for your server pod. That will make it listen on
the VM's main interface.
- add the VM's IP to the Service's `externalIPs` array. That will
tell the proxy to also accept traffic on that IP.

I think that will do what you want. Unfortunately, the management of
`externalIPs` is not automatic. You can write a small program to sync
that in case your pod gets moved.

Tim

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.

[wiktor....@gmail.com]

Hi Tim,

Thanks for your answer and sorry for not replying before, I didn't realize I should check the group site for answers.

Your solution for exposing the server on given external IP worked, specifically setting hostPort in pod's container spec.

The client job runs in a single-node cluster, so the client pod can't really be moved to a different IP. Thanks for pointing this out though.

For the sake of my understanding, would you mind explaining what you mean by "Kubernetes Services are designed to avoid that" and that they "can't handle client IPs" ?

Possibly on a related note, what would be the right way in GKE to keep running a service at a given external IP address? Let's forget my client job for a moment and limit this discussion to the server job alone. Let's say I have an external IP address in GKE, associated with mycompany.com in a third-party DNS registry, and I'd like to run a web server in GKE visible to the outside world at mycompany.com:80. How should I configure the service/pod/deployment/... in GKE?

[Tim Hockin]

On Dec 11, 2016 12:58 PM, <wiktor....@gmail.com> wrote:

Hi Tim,

Thanks for your answer and sorry for not replying before, I didn't realize I should check the group site for answers.

Your solution for exposing the server on given external IP worked, specifically setting hostPort in pod's container spec.

The client job runs in a single-node cluster, so the client pod can't really be moved to a different IP. Thanks for pointing this out though.

For the sake of my understanding, would you mind explaining what you mean by "Kubernetes Services are designed to avoid that" and that they "can't handle client IPs" ?

Services provided addresses that hide the details of the pods "behind" them.  Pods can be born or die, or move between nodes, and Services keep running on a stable IP.

Possibly on a related note, what would be the right way in GKE to keep running a service at a given external IP address? Let's forget my client job for a moment and limit this discussion to the server job alone. Let's say I have an external IP address in GKE, associated with mycompany.com in a third-party DNS registry, and I'd like to run a web server in GKE visible to the outside world at mycompany.com:80. How should I configure the service/pod/deployment/... in GKE?

A Service with either type=LoadBalancer or an Additional Ingress (HTTP load balancing)

On Monday, November 21, 2016 at 11:35:10 PM UTC+1, Wiktor Tomczak wrote:
> Hello,
>
>
> I'd like to run two different containers, each with a different executable, on the same specific external IP address in Google Container Engine (GKE). One executable is a web server and the other is a web client. Requests from the client should originate from the same IP that the server listens on.
>
>
> So far I've been able to get the client executable to run on an external IP address that I had reserved via Networking > External IP addresses in the cloud console [1]. The address is assigned to my only VM instance, so IIUC all containers running there use this address for their outgoing IP traffic. I can confirm the external IP address eg. by attaching to a running container [2] and fetching checkip.dyndns.org from there [3].
>
>
> OTOH, I haven't been able to expose the server executable on the same IP address. I tried both .yaml service configuration and kubectl expose but can't get either to work. What should be the value of my service's spec.type? If I set type = LoadBalancer and LoadBalancerIP = <my external IP above>, load balancer creation fails because the IP is reported as already used. If I set type = NodePort, the service gets created, but requests to <my external IP>:8080 aren't forwarded to the server executable (configured to listen on port 8080).
>
>
> I'd appreciate any help. 
>
>
> Wiktor
>
>
> [1] https://console.cloud.google.com/networking/addresses/list
>
> [2] kubectl run -ti <pod name> -- /bin/bash
>
> [3] curl checkip.dyndns.org

--
You received this message because you are subscribed to the Google Groups "Kubernetes user discussion and Q&A" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.