Client Loadbalancing with Kubernetes SVC

[julien...@gmail.com]

Hello,

Does it make sense to have client loadbalancer with gRPC when we are using gRPC server in a Kubernetes cluster? 

Because client will dial a service DNS and will always retrieve IP of service and not IP of Pods behind it.

NB: If already seen this blog entry https://github.com/grpc/grpc/blob/master/doc/load-balancing.md

Thanks

[Josh Humphries]

Unless things have changed recently, the default kubedns result for a standard k8s service will have a single IP: the virtual IP of the service. This in turn causes the gRPC client to configure just a single socket connection and route all requests on the one socket. Since kubeproxy load balancing through that virtual IP is layer 4, this can result in very poor load balancing, especially if you have unbalanced clients (e.g. a small number of clients that constitute the majority of RPC traffic).

If you use "headless" services in k8s on the other hand, the DNS query should return multiple pod IPs. This causes the gRPC client to in turn maintain multiple connections, one to each destination pod. However, I am not sure how responsive this will be to topology changes (as pod instances are auto-scaled, or killed and rescheduled, they will move around and change IP address). It would require disabling DNS caching and making sure the service host name is resolved/polled regularly.

Another solution is to use a custom gRPC resolver that talks to the k8s API to watch the service topology and convey the results to the gRPC client. For Go, this is implemented in an open-source package: github.com/sercand/kuberesolver

(Most of my experience is with Go. So your mileage may vary if using a runtime other than Go. But I think the various implementations largely behave quite similarly.)

----
Josh Humphries
jh...@bluegosling.com

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/63490220-692a-49fd-84bd-62ef13b19e09%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[julien...@gmail.com]

Thanks a lot for your explanation. I will have a look on 2 solution provided.