Ingress Cascading (example GCE > NGINX)

[Alex Van Boxel]

First SIG post, so I hope this is the right place for this. Here is the situation:

I ran into the GCE Ingress controller limitation of a maximum entries of 50 back-end, we have each service mapped in the ingress (example: /api/auth/ -> service auth). It was horribly slow, so I did some hacks do to route the one GCE load balancer. What I did do is hack the GCE controller that it does cascading to NGINX controlled by the nginx controller running as a daemon set. The team is super happy because ingress update time come down from 30 minutes+ to a few seconds.

Anyway... I want to turn the hack into a proper PR and this is how I would like to do it:

This is the ingress file I would work with:

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  annotations:

    ingress.kubernetes.io/service-mapping: "{'nginx-ingress-lb': { 'namespace':'kube-system', 'service':'nginx-ingress-headless', 'portName':'http-81' }"

  name: iap-lb

  namespace: develop

spec:

  backend:

    serviceName: nginx-ingress-lb

    servicePort: 80

  tls:

  - secretName: tls

1. For each entry in the spec referencing a service that is in the ingress.kubernetes.io/service-mapping it will try to find the the service in the defined namespace ( in my case kube-system).
   
2. If the service is of type NodePort it will point to that node (but for this use-case bad, because it adds a kube-proxy)
3. If the service is headless it will go and search the containers and look at the port with the defined name, and look at hostPort value. This is ideal for nginx controller running as a daemon set.

Does this make sense?! I actually want to use the same approach for Istio.

I'm posting it here because I see sometimes references in documentation about Ingress Cascading. But without hacking like this, I don't see how I could make it happen...

PS. I want to keep the GCE LB for TLS termination, IAP and CDN caching.

[Tim Hockin]

I am not sure I understand the goal of the annotation.

If we formalize ingress classes (as proposed elsewhere on the list)
then you have a single instance of the GCE class pointing to your
nginx, and as many instcnaes of teh nginx class as you need.

ISTR there are some issues with ExternalTrafficPolicy and GCLB
healthchecks - we'd need to make sure that works.

Optimizing for hostPorts seems superfluous - hostPort is literally the
same mechanism as nodePort (once ExternalTrafficPolicy is engaged).
Optimizing for hostNetwork pods might be interesting, but seems
orthogonal to the rest.

Lastly, in this model you're basically opting all of your Ingress
instances in to IAP, CDN, etc, since they will share the same
BackendService instances in GCP. And you have to manage TLS
separately from Ingresses, unless you implement a whole new two-level
controller...

> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-sig-ne...@googlegroups.com.
> To post to this group, send email to
> kubernetes-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

[Alex Van Boxel]

It's sometimes hard to explain in words (I miss a whiteboard in Google Groups). I'll look into the formalize ingress class, but that will be for tomorrow.

The hack that's currently working as follows:

2 GCE LB

iap-lb > GCE LB with CDN + IAP + TLS ( wildcard TLS certificate )  balancing to all nodes port 81

web-lb > GCE LB with CDN + TLS ( wildcard TLS certificate ) balancing to all nodes port 80

nginx listening on port 80 and 81 (maybe I'll find a way on nginx to check of IAP headers), we have then ingress files per hostname (all in the same main domain).

The thing I want to avoid, by directly using hostPort on the container is to avoid the extra hop though kube-proxy (adding nginx uses Endpoints to route traffic to the final destination). 

Anyway... I'll look into the proposals tomorrow, and if it suits my use-case I'll just keep running the hack till the for the formalized ingress class.

Thanks.

> email to kubernetes-sig-network+unsub...@googlegroups.com.

[Sandor Szuecs]

Hi!

On 21 November 2017 at 17:52, 'Tim Hockin' via kubernetes-sig-network <kubernetes-...@googlegroups.com> wrote:

I am not sure I understand the goal of the annotation.

If we formalize ingress classes (as proposed elsewhere on the list)
then you have a single instance of the GCE class pointing to your
nginx, and as many instcnaes of teh nginx class as you need.

ISTR there are some issues with ExternalTrafficPolicy and GCLB
healthchecks - we'd need to make sure that works.

Optimizing for hostPorts seems superfluous - hostPort is literally the
same mechanism as nodePort (once ExternalTrafficPolicy is engaged).
Optimizing for hostNetwork pods might be interesting, but seems
orthogonal to the rest.

Lastly, in this model you're basically opting all of your Ingress
instances in to IAP, CDN, etc, since they will share the same
BackendService instances in GCP.  And you have to manage TLS
separately from Ingresses, unless you implement a whole new two-level
controller...

You can also run 2 controllers, one for GCLB to terminates TLS and move packets to your 2nd tier, that does http routing.

We use in AWS exactly this:

- terminate TLS on ALB using ACM certificates: https://github.com/zalando-incubator/kube-ingress-aws-controller

- http routing based on ingress spec (most of the time Host header) including blue-green traffic switching, shadow traffic, ...: https://github.com/zalando/skipper

- public DNS names with https://github.com/kubernetes-incubator/external-dns

We don't need any additional annotation if you don't want to pin to a certificate or do some more advanced features with skipper filters or predicates.

Best, sandor

> email to kubernetes-sig-network+unsub...@googlegroups.com.

> To post to this group, send email to

> kubernetes-sig-network@googlegroups.com.

> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-network+unsub...@googlegroups.com.
To post to this group, send email to kubernetes-sig-network@googlegroups.com.

Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--

Sandor Szücs | 418 I'm a teapot