polling folks who replace (or intend to replace) built-in kube-proxy

[vipi...@insiemenetworks.com]

I am poling to see if there are are folks who replace built-in kube-proxy in K8s. There could be many reasons for doing this:

- if the iptables based host-datapath doesn't work for your CNI plugin (this is my reason)

- or if you believe your implementation could be better/faster

If there is enough interest, perhaps we could combine the skeleton code that we all can leverage, esp catching up with API versions, etc.

Please chime back if this sounds like something needed.

Thanks,

-Vipin

[Irena Berezovsky]

Hi Vipin,

The kuryr  team [1] is working on integration of the kubernetes with neutron.

The ongoing specification is here [2].

For number of neutron backends (i.e. MidoNet) kube-proxy cannot fit due to the first reason you stated.

The idea is to replace kube-proxy functionality by the neutron Load Balancer.

Currently we plan to run API watcher (Raven service) to watch kubernetes services changes and call neutron API to create and populate Load Balancer's instance.

What is your idea for replacing kube-proxy?

[1] https://launchpad.net/kuryr

[2] https://review.openstack.org/#/c/281132/

Regards,

Irena

[Sarah Novotny]

This is also likely to be of interest to the SIG-Openstack

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

[Oliver Gould]

Hi Vipin-

We're interested in replacing kubeproxy with linkerd, which can use the Endpoints api as a service discovery backend.

I haven't done any of the setup work for this yet, and so I'm be curious to know what others are doing.

[Tim Hockin]

I am very interested in seeing people build environments that don't
use the built-in kube-proxy, or that need to mutate what it does.
Consider me a supporter.

That said, there's been other discussions about putting things like
DNS into kube-proxy, which would make it less of a single-purpose
thing and more of a local network-ish agent. I think this is all very
much up for discussion as real requirements emerge.

I know that, for example, using ipvlan in L2 mode is very interesting
to some folks, but that obviously does not work with kube-proxy.

[vipi...@insiemenetworks.com]

Tim - thanks for +1

Also, love the replaceability of components in k8s architecture. In fact, addition of new features (like DNS you mentioned) makes it a whole lot more attractive to not repeat the code/logic outside by many. Sounds like it would serve as local proxy dns for all requests within cluster and fall back to external ones when not needed. And perhaps it would eliminate the need for bundled dns container? Can you please send pointers to the discussion (github and/or mailer list)?

Also thanks Irene, Oliver and others who responded showing some interest in joining some efforts.

Just to fill more details on our implementation: Our kube-proxy replacement allows us to have open-flow datapath (using OVS) to implement a distributed load balancer, by listening to APIserver (like kube-proxy) and generate mapper events (Service -> container, and new flows detection to map to backend containers for a service). I will send the pointer to the code if anyone is interested in taking a look.

Thanks,

-Vipin

> email to kubernetes-sig-network+unsub...@googlegroups.com.

[Akash Gangil]

Hi Vipin, 

I have been evaluating replacing kube-proxy too, and would be interested in helping out with review/design/code etc. I haven't sketched out anything formally yet but can do it if we need a base to start off. 

> email to kubernetes-sig-ne...@googlegroups.com.

> To post to this group, send email to

> kubernetes-...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-sig-network.
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

To post to this group, send email to kubernetes-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

--

Akash

[Michael Bridgen]

Hi Vipin, all,

We've been working on a "service routing" layer for Docker, called
Flux[1]. Inevitably, one of the moving parts is a proxy, which is
pretty similar to kube-proxy; the salient difference being that it
exposes HTTP request metrics for display in our UI (and is driven by
our controller rather than the k8s API, of course).

It's been an idea from the get-go that eventually, the proxy could act
as a replacement for kube-proxy. Some things that would help that
would be: skeleton controller code which we could adapt to control our
proxy, and a well-defined and stable boundary to kube-proxy's
responsibility. (Perhaps these things exist already or are in the
works, in which case, great!)

[1] https://github.com/weaveworks/flux

[Mike Spreitzer]

I am considering this.  I want to introduce multi-tenancy with link layer isolation between tenants.  The existing kube proxy violates this, regardless of whether it uses the userspace or iptables datapath.  However, I have not finished thinking through exactly what I want to do.

Thanks
Mike

[Rajat Chopra]

The contrail plugin also replaces the kube proxy with its own implementation to enable multi tenancy of services.

/Rajat

[Dan Williams]

For OpenShift we have simple interaction with kube-proxy and our
multitenant configuration.  This is currently only to ensure that pods
of one tenant cannot access services of other tenants, and is done
through an endpoint update hook that filters the list of endpoints
before the iptables proxy acts.

We want to run a standalone kube-proxy but we currently build the proxy
code into openshift itself, which is why the endpoint update hook works
for us right now.  As a separate process we'd want kube-proxy to have
some kind of extension API, but...

...it turns out that that our implementation isn't great scale-wise and
we'll need to do something different in the near future here.  I'm not
sure whether that will still include modifying kube-proxy to use our
endpoint filter.

Dan

[Rudi Chiarito]

I was toying with the idea of gRPC- or HTTP-aware load balancing. If you use iptables proxying and have e.g. frontend pods with a persistent connection and/or high traffic to a service IP, they tend to hotspot toward the oldest backend pods behind the service, without round robin. At that point, you either do the work yourself in the client, through headless services, or let kube-proxy handle that, i.e. you're back in userland. It'd be halfway between the current state of things and Ingress objects (the latter would require more configuration).

--

Rudi

[Ado Tony]

Hi Vipin,

Noticed this too late, really hope that I have noticed this at the beginning.

Actually our team is using OVS too, and also we are planning replace kube-proxy, but still no good solution. It seems that you have some good suggestions,

really appreciate if we can have a discussion about how to do this. Hope to hear from you. :)

在 2016年3月15日星期二 UTC+8下午1:06:45，vipi...@insiemenetworks.com写道：

[vipi...@insiemenetworks.com]

Hi Ado,

Any chance you'd be joining the sig-net call today? If so we can briefly chat about it on the call...

I work on Contiv (open sourced at https://github.com/contiv/netplugin), that utilizes OVS pipeline based on OF 1.3 onwards pipeline.

The proxy code uses standard kube apis to:

https://github.com/contiv/netplugin/blob/master/mgmtfn/k8splugin/kubeClient.go#L213

and programmed in OF pipeline using https://github.com/contiv/ofnet repository - ofnet is a golang library to manipulate openflow pipeline.

Thanks,

-Vipin

[Alex Pollitt]

It's tomorrow I think...

Sent from my iPhone

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.

[Ado Tony]

Hi Vipin,

Thanks very much for your quick response. I am afraid I can't join the sig-net meeting today, since I am out for a short holiday.
I will deep into your code, and hope we can have a discussion later if we both are available.
Thanks again. :)

在 2016年4月28日星期四 UTC-4下午2:34:13，vipi...@insiemenetworks.com写道：

[Mike Spreitzer]

On Wednesday, March 16, 2016 at 6:19:48 PM UTC-4, Dan Williams wrote:

...

For OpenShift we have simple interaction with kube-proxy and our
multitenant configuration.  This is currently only to ensure that pods
of one tenant cannot access services of other tenants, and is done
through an endpoint update hook that filters the list of endpoints
before the iptables proxy acts.

...

I am having trouble understanding what you mean.  The existing iptables proxy sets up iptables rules that are not sensitive to which tenant owns the client.  How could a pre-proxy endpoint filter keep one tenant's client from using another tenant's endpoint?
 

[Dan Winship]

Yeah, he described that incorrectly... that's not what the endpoint
filter is for.

The problem is that the service proxy throws away information about the
source of the traffic (in particular, if you're using the userspace
proxy, you lose the original source IP, but even if not, you still lose
the contents of OVS registers that were accompanying the traffic flow
through OVS). So once traffic comes out of the service proxy, we no
longer have the ability to validate that it is properly isolated, so
instead we have to make sure to set things up to guarantee that only
properly-isolated traffic can come out of the service proxy in the first
place.

Originally we just added OVS rules to ensure that pods can only send to
service IPs in the same namespace as the pod. But that doesn't
completely solve the problem because you can create Endpoint objects by
hand, and so someone could add endpoints to a service in one namespace
that pointed to pod IPs in another... So that's what the endpoint filter
is for; to prevent that from happening, so that we know that if we allow
the traffic into the service, it will end up in a pod in the same
namespace. (Or outside of the cluster.)

-- Dan

[Ado Tony]

As a replacement of built-in kube-proxy, our team is using a more traditional way. Compared with the distributed proxy implementation, we are using a centralized proxy. More specifically, we are using HAProxy. For every kubernetes service, we will bind service cluster ip to network interface of the HAProxy instance, and also we will write the service, endpoints info to the HAproxy configuration file. There are some tradeoff to use a centralized proxy. We chose to use a centralized proxy, because we have used HAProxy in production for a long time, and the Ops team have good knowledge about HAProxy. This approach could be a direct way for companies to adapt kubernetes and also integrate with the existing infrastructure.

在 2016年5月2日星期一 UTC-4下午3:20:18，Dan Winship写道：