[ANNOUNCE] AWS ALB Ingress Controller for Kubernetes

[Brandon Philips]

Hey all,

CoreOS & Ticketmaster have been working together on an AWS ALB ingress controller that is self-contained and easy to deploy. Specifically, one that can be used in place of the default ELB integration and provides more flexibility around its layer 7 routing capabilities and Route 53 DNS records.

- GitHub project 

- AWS Blog Post with examples

- CoreOS Blog Post with context

It has been added to the Ingress Catalog as well.

If you want to discuss checkout the SIG AWS thread.

Cheers,

Brandon

[Tim Hockin]

Great! As a frequent topic, we should discuss where is the best
place for this to live. We have a `kubernetes/ingress` repo, but we
need to stop doing omnibus repos. We could decompose that into
ingress-gcp, ingress-aws, ingress-nginx repos in the kubernetes org,
or we could simply EOL that repo and tell people to host repos in
their own spaces.

There is some shared code there - is it worth jumping through hoops for?

Thoughts?

> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-sig-ne...@googlegroups.com.
> To post to this group, send email to
> kubernetes-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

[Justin Santa Barbara]

We agreed previously that we would put all ingresses into kubernetes/ingress, and then split it out if/when we had a handle on whether a monorepo was a problem (shared vs non-shared code/issues).

kubernetes/ingress is where the official ingress work is happening.

Anyone is free to write their own effort separately, of course.  There are several prototypes of varying degrees of harmony with the kubernetes architecture.  The Zalando effort is in production (AIUI) and would be architecturally more ready to merge IMO.

The reason we haven't done this through sig-aws is because there wasn't a clear use case vs making ingress better; we asked every time it came up but there was never a clear advantage for ALB.  As it seems there is a use case, we should get one of these efforts into the ingress repo.

> email to kubernetes-sig-network+unsub...@googlegroups.com.

> To post to this group, send email to

> kubernetes-sig-network@googlegroups.com.

[Henning Jacobs]

Thanks for the shoutout, to clarify:

We (Zalando) created and open sourced our kube-ingress-aws-controller already in December 2016 which creates HTTPS ALBs on-demand and routes traffic to another layer of HTTP proxy for host/path routing (we use Skipper, but anything [like nginx, Traefik, etc] will work). Our controller does _not_ create DNS records, we rely on External DNS for this, so it's in general pluggable:

• kube-ingress-aws-controller only creates/manages ALBs and automatically discovers the right SSL cert (AWS ACM and IAM), so it's very convenient for users (no human should ever be forced to copy & paste some AWS ARN...)
• the HTTP proxy runs as a DaemonSet (to automatically s-cale by cluster size and for ease of setup) and routes based on host/path to the Kubernetes service: this can be Skipper, nginx, Traefik or anything evaluating Ingress rules
• a DNS controller syncs the hostnames from Ingress to Route53: this can (and should) be External DNS, but could also be some other DNS controller (or none at all)

For more information see https://kubernetes-on-aws.readthedocs.io/en/latest/admin-guide/kubernetes-in-production.html#ingress

If you want to compare both approaches, see this issue: https://github.com/coreos/alb-ingress-controller/issues/85

Major differences:

• CoreOS ALB controller uses purely the ALB for routing (no extra http proxy needed, but AWS limits apply [and customizations are limited]
• CoreOS ALB controller has integrated DNS syncing to Route53 --- this conflicts with kops DNS controller and will be changed as far as I understand (https://github.com/coreos/alb-ingress-controller/issues/9)

The kube-ingress-aws-controller works great for us (running in production for months) and we don't really have any features on the roadmap right now, but we are open for anybody willing to give it a try and any feedback/suggestions :-)

- Henning

> email to kubernetes-sig-network+unsubscri...@googlegroups.com.

[Brandon Philips]

On Wed, May 10, 2017 at 11:00 AM Justin Santa Barbara <jus...@fathomdb.com> wrote:

We agreed previously that we would put all ingresses into kubernetes/ingress, and then split it out if/when we had a handle on whether a monorepo was a problem (shared vs non-shared code/issues).

Hrm, my understanding from another conversation was that we didn't want to add more things into that repo. Happy to move the code in there.

 

kubernetes/ingress is where the official ingress work is happening.

Tim and I had a casual conversation and he had the same understanding not to put stuff into that repo.

 

Anyone is free to write their own effort separately, of course.  There are several prototypes of varying degrees of harmony with the kubernetes architecture.  The Zalando effort is in production (AIUI) and would be architecturally more ready to merge IMO.

Why? What is there an architectural issue? What is the Skipper thing?

We built this ALB Ingress not being aware of Zalando's effort; they started around the same time.

 

The reason we haven't done this through sig-aws is because there wasn't a clear use case vs making ingress better; we asked every time it came up but there was never a clear advantage for ALB.  As it seems there is a use case, we should get one of these efforts into the ingress repo.

What do you mean by making ingress better? You mean the NGINX Ingress?

Thanks,

Brandon

 

> email to kubernetes-sig-ne...@googlegroups.com.

> To post to this group, send email to

> kubernetes-...@googlegroups.com.

[Justin Santa Barbara]

> Happy to move the code in there

I think at this point we'd do better to start from scratch, particularly as it's not a big effort (I'm guessing this took 1 or 2 days?).

The thing holding us back from doing it already has been the lack of a use case.  What was the use case for coreos?

> email to kubernetes-sig-network+unsub...@googlegroups.com.

> To post to this group, send email to

> kubernetes-sig-network@googlegroups.com.

[Tim Hockin]

My concern with a monorepo is the same as ever - someone has to review
those pull requests. In this case, it's a small number of disparate
implementations, but fast-forward -- do we want 20 implementations in
the same repo?

On Wed, May 10, 2017 at 2:37 PM, Justin Santa Barbara

>>>> > email to kubernetes-sig-ne...@googlegroups.com.

>>>> > To post to this group, send email to

>>>> > kubernetes-...@googlegroups.com.

[Brandon Philips]

On Wed, May 10, 2017 at 2:38 PM Justin Santa Barbara <jus...@fathomdb.com> wrote:

The thing holding us back from doing it already has been the lack of a use case.  What was the use case for coreos?

Our use case for Ingress? Or the use case for ALB Ingress vs NGINX Ingress & ELB?

[Justin Santa Barbara]

I agree with the concerns Tim; but we should also experiment with different approaches.  Here, for example, I suspect ALB would likely share a lot of code with GCELB ingress.  We'd probably make both better if we wrote ALB in the GCELB framework.  The question is whether that is a bigger win than having separate issue trackers for each implementation.  My hunch is yes, but I would also like to collect the data before we make a decision.

And yes Brandon, CoreOS's use case for ALB ingress vs NGINX ingress with ELB (or nginx with alternatives, but typically ELB beats alternatives for the use cases I've heard).

[Tim Hockin]

Really? I don't know that codebase very well but the last I looked,
the GCP code was the vast majority of it..

On Wed, May 10, 2017 at 3:35 PM, Justin Santa Barbara

[Justin Santa Barbara]

I feel like I'm being nerd sniped, but I'm going to be disciplined and wait for a use case :-)

[Tim Hockin]

I'm not sure I understand.

On Wed, May 10, 2017 at 6:52 PM, Justin Santa Barbara

[Sandor Szuecs]

Hi!

On 10 May 2017 at 23:11, Brandon Philips <brandon...@coreos.com> wrote:

Anyone is free to write their own effort separately, of course.  There are several prototypes of varying degrees of harmony with the kubernetes architecture.  The Zalando effort is in production (AIUI) and would be architecturally more ready to merge IMO.

Why? What is there an architectural issue? What is the Skipper thing?

To answer the last question, github.com/zalando/skipper is Zalando's main shop router. Features that make it better for Kubernetes then nginx:
1) it does not need to be reloaded to get new routes

2) it automatically detects new/updated ingress entries and their backends to route traffic to the defined Kubernetes service

Our current work on skipper is to enable traffic shaping from ingress to support for example blue/green deployments.

Best, sandor

--

Sandor Szücs | 418 I'm a teapot

[kr...@bigkraig.com]

The alb-ingress-controller is using the ingress framework that aledbf had created in the kubernetes/ingress repo. I don't know the entire history but from what I can tell, he either refactored or wrote an nginx controller based around that ingress framework. The GCE ingress in that repo is not (yet?) using that framework.

[kr...@bigkraig.com]

The alb-ingress-controller was born to solve ingress at Ticketmaster. We have a fairly complicated network topology that requires the specific placements of our ALBs and we also get a pretty heavy load of traffic, so individual ALBs in specific subnets per service is a requirement for us. We also didn't have requirements beyond what the ALB configuration offers, so adding in another hop of nginx/skipper/etc was undesired, it complicated the infrastructure further and added another layer for us to debug. We've got a few hundred products and even more microservices, so even small nginx pods add up very quickly.

[Tim Hockin]

Would it make sense tobreak the framework out to a repo, and then make
each ingress its own repo?

> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-ne...@googlegroups.com.

> To post to this group, send email to

> kubernetes-...@googlegroups.com.

[kr...@bigkraig.com]

I think so. It would clear up that there is a framework there. I spent some time digging around the repo to understand how it was laid out and it would incentivize more ingress controller development if people knew you could just drop in logic and go.

> email to kubernetes-sig-network+unsub...@googlegroups.com.

[Justin Santa Barbara]

I don't understand what problem we are trying to solve.

> email to kubernetes-sig-network+unsubscri...@googlegroups.com.

> To post to this group, send email to

[Tim Hockin]

The problem *I* am trying to solve is that omnibus repos are painful
once N gets more than 2 or 3 (see volumes, cloud providers, even kube
binaries). I can see the `ingress` repo going there, and I'd like to
get ahead of it.

I have no opinion on whether ALB is a good or useful ingress
implementation (though by default I assume it is, since it solves
SOMEONE's problems).

On Fri, May 12, 2017 at 10:15 AM, Justin Santa Barbara

>>> > email to kubernetes-sig-ne...@googlegroups.com.

>>> > To post to this group, send email to
>>> > kubernetes-...@googlegroups.com.
>>> > Visit this group at
>>> > https://groups.google.com/group/kubernetes-sig-network.
>>> > For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-ne...@googlegroups.com.

[Justin Santa Barbara]

My vote would be that we defer splitting the ingress repo until the problem arises.

I hypothesize that the more similar the functionality, the bigger the threshold N will be (i.e. this might not be imminent).  Further I hypothesize that identification and iteration on common functionality is facilitated by being in a single repo.

>>> > email to kubernetes-sig-network+unsub...@googlegroups.com.

>>> > To post to this group, send email to
>>> > kubernetes-...@googlegroups.com.
>>> > Visit this group at
>>> > https://groups.google.com/group/kubernetes-sig-network.
>>> > For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "kubernetes-sig-network" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to kubernetes-sig-network+unsub...@googlegroups.com.

> To post to this group, send email to

> kubernetes-sig-network@googlegroups.com.

[Tim Hockin]

Secondary problem - vendoring in an onmibus is messy at best. I am
fine leaving it as-is, since N=3 is barely at the threshold.

I am all for reuse, but honestly, it shouldn't be a goal on its own.
If having N repos means similar but different code, I am OK with that.
But if N hits 4 or 5, expect this topic to come back.

On Fri, May 12, 2017 at 10:30 AM, Justin Santa Barbara

>> >>> > email to kubernetes-sig-ne...@googlegroups.com.

>> >>> > To post to this group, send email to
>> >>> > kubernetes-...@googlegroups.com.
>> >>> > Visit this group at
>> >>> > https://groups.google.com/group/kubernetes-sig-network.
>> >>> > For more options, visit https://groups.google.com/d/optout.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "kubernetes-sig-network" group.
>> > To unsubscribe from this group and stop receiving emails from it, send
>> > an

>> > email to kubernetes-sig-ne...@googlegroups.com.

[Brandon Philips]

OK, so should we submit a PR to add the ALB ingress code to the ingress repo then? Is that the consensus Nick, Alejandro, Prasanth?