replace kube-proxy with centralized HAProxy

959 views
Skip to first unread message

Ado Tony

unread,
Jun 13, 2016, 12:31:59 PM6/13/16
to kubernetes-sig-network
Recently, our team managed to replace built-in kube-proxy with centralized HAProxy in our new production Kubernetes cluster, and everything goes well. To be honest, kube-proxy is so awesome, we have used in production for almost a year, it works well most of time, but as we have more and more services in our cluster, we found it was getting hard to debug and maintain. There is no iptables expert in our team, we do have HAProxy&LVS experts, as we have used these for several years, so we decided to replace this distributed proxy with a centralized HAProxy. I think this maybe useful for some other people who are considering using HAProxy with kubernetes, so we just update this  project and make it open source: https://github.com/AdoHe/kube2haproxy. If you found it's useful , please take a look and give a try.

Prashanth B

unread,
Jun 13, 2016, 1:25:17 PM6/13/16
to Ado Tony, kubernetes-sig-network

On Mon, Jun 13, 2016 at 9:31 AM, Ado Tony <coo...@gmail.com> wrote:
Recently, our team managed to replace built-in kube-proxy with centralized HAProxy in our new production Kubernetes cluster, and everything goes well. To be honest, kube-proxy is so awesome, we have used in production for almost a year, it works well most of time, but as we have more and more services in our cluster, we found it was getting hard to debug and maintain. There is no iptables expert in our team, we do have HAProxy&LVS experts, as we have used these for several years, so we decided to replace this distributed proxy with a centralized HAProxy. I think this maybe useful for some other people who are considering using HAProxy with kubernetes, so we just update this  project and make it open source: https://github.com/AdoHe/kube2haproxy. If you found it's useful , please take a look and give a try.

--
You received this message because you are subscribed to the Google Groups "kubernetes-sig-network" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-sig-ne...@googlegroups.com.
To post to this group, send email to kubernetes-...@googlegroups.com.
Visit this group at https://groups.google.com/group/kubernetes-sig-network.
For more options, visit https://groups.google.com/d/optout.

Maru Newby

unread,
Jun 13, 2016, 1:54:39 PM6/13/16
to Prashanth B, Ado Tony, kubernetes-sig-network, Clayton Coleman
It looks like kube2haproxy has taken code from the openshift router
for niceties like minimizing unnecessary reloads and ensuring safe
handoff from old process to new. These features are required for
haproxy reloads to work at scale and are missing from the contrib
code.

While streamlining may be desirable, I would hope that a concerted
upstream effort to support haproxy load balancing would be based on
ingress rather than services.

On Mon, Jun 13, 2016 at 10:25 AM, 'Prashanth B' via
kubernetes-sig-network <kubernetes-...@googlegroups.com>
wrote:

Prashanth B

unread,
Jun 13, 2016, 2:03:21 PM6/13/16
to Clayton Coleman, Maru Newby, Ado Tony, kubernetes-sig-network
Agreed. The Ingress model is more flexible and allows one to satisfying the 60% use case in a cross platform way. At the same time there is a niche for loadbalancing based on Services and annotations, and several people have already deployed it as a similar kubeproxy replacement. 

There are efforts underway to teach the existing serviceloadbalancer about Ingress. It's a great starter project for anyone wanting to get involved.
 

On Mon, Jun 13, 2016 at 11:00 AM, Clayton Coleman <ccol...@redhat.com> wrote:
I do think there is a use case for haproxy as a kube-proxy replacement
for TCP, for sure.  The ideal outcome would be a set of core reusable
code that is workable across both kube-proxy and ingress use cases
with the appropriate specialization (things like templatizing the
config and the standard flags / options / watch cache behavior).

Clayton Coleman

unread,
Jun 13, 2016, 2:04:30 PM6/13/16
to Prashanth B, Maru Newby, Ado Tony, kubernetes-sig-network
I do think there is a use case for haproxy as a kube-proxy replacement
for TCP, for sure. The ideal outcome would be a set of core reusable
code that is workable across both kube-proxy and ingress use cases
with the appropriate specialization (things like templatizing the
config and the standard flags / options / watch cache behavior).

Ado Tony

unread,
Jun 16, 2016, 2:31:33 AM6/16/16
to kubernetes-sig-network, ccol...@redhat.com, ma...@redhat.com, coo...@gmail.com
     a niche for loadbalancing based on Services and annotations

what you mean about this? 

    There are efforts underway to teach the existing serviceloadbalancer about Ingress

Also what about this, you mean the service-loadbalancer?
在 2016年6月14日星期二 UTC+8上午2:03:21，Prashanth B写道：

>>> To post to this group, send email to
>>> kubernetes-...@googlegroups.com.
>>> Visit this group at
>>> https://groups.google.com/group/kubernetes-sig-network.
>>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kubernetes-sig-network" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

Ado Tony

unread,
Jun 16, 2016, 2:33:27 AM6/16/16
to kubernetes-sig-network, be...@google.com, ma...@redhat.com, coo...@gmail.com
I do think so, and I am willing to do much more things about this, but where should I start?

在 2016年6月14日星期二 UTC+8上午2:04:30，Clayton Coleman写道：
>> >>> To post to this group, send email to
>> >>> kubernetes-...@googlegroups.com.
>> >>> Visit this group at
>> >>> https://groups.google.com/group/kubernetes-sig-network.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "kubernetes-sig-network" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an

Prashanth B

unread,
Jun 16, 2016, 1:49:09 PM6/16/16
to Ado Tony, kubernetes-sig-network, Maru Newby
On Wed, Jun 15, 2016 at 11:33 PM, Ado Tony <coo...@gmail.com> wrote:
I do think so, and I am willing to do much more things about this, but where should I start?


Keep filing issues as things come up

Ideally we should mimize code duplication for the common case. Maybe a good first cut would be to add a --backend option to the nginx ingress controller, that wrote out a haproxy config instead of nginx. Everything else should be more or less the same. 
 
>> >>> To post to this group, send email to
>> >>> Visit this group at
>> >>> https://groups.google.com/group/kubernetes-sig-network.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "kubernetes-sig-network" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> To post to this group, send email to

Ado Tony

unread,
Jun 16, 2016, 10:28:30 PM6/16/16
to kubernetes-sig-network, coo...@gmail.com, ma...@redhat.com
Thanks very much, and I will do it asap.

在 2016年6月17日星期五 UTC+8上午1:49:09，Prashanth B写道：


>> >>> To post to this group, send email to
>> >>> Visit this group at
>> >>> https://groups.google.com/group/kubernetes-sig-network.
>> >>> For more options, visit https://groups.google.com/d/optout.
>> >>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "kubernetes-sig-network" group.
>> >> To unsubscribe from this group and stop receiving emails from it, send
>> >> an
>> >> To post to this group, send email to
Reply all
Reply to author
Forward
0 new messages