HelpNeeded: in RoundRobinLoadBalancerFactory usage properly
39 views
Skip to first unread message
Vinay Kumar Marpina
Jan 31, 2019, 7:03:58 PM1/31/19
to grp...@googlegroups.com
Dear Concerned,
Scenario: I have a grpc service which hosted AWS EKS with a single load balancer(ELB) pointing to 5 replicas. My grpc client is creating a single channel and in that single channel trying to send 1000 of requests. If i use multiple channels load balancer properly distributing load if not it is sending to only pod out of 5 replicas.
So what i did is i used RoundRobinLoadBalancerFactory instance in ManagedChannel Builder to build channel. Now when i sending 100's of requests through a single channel it able to distribute the load but it only distributes to 60% of replicas based on its choice and never send any results to rest 40%. Example: 3 replicas only 2 hits, 5 replicas 3 get hit etc...
When i started debugging i found out irrespective number of replicas DnsNameResolver always resolving only 3 servers only. Its pretty weird. Am i missing some thing ?? resolutionResults or Servers should be respective to number of replicas right. Even when i have 2 replicas i have 3 servers in DnsNameResolver . I think my understanding of Source code is bit off. Can some one please Help me in finding what i am missing.
If this is not the right place can you please guide me to correct place where i can find help. If i need to provide any more info please let me know.
Thanking you in Advance for your valuable time.
Regards,
-Vinay,
Carl Mastrangelo
Feb 1, 2019, 6:44:09 PM2/1/19
to grpc.io
Is there another way to get the addresses of all your backends? If the DNS query only returns some of the results, gRPC won't try to query it again until some of the connections fail (or a timeout expires). You can see what backends are returned by calling the 'dig' tool on Linux a few times to see what response you get.
One alternative is to implement a custom NameResolver (not as hard as it sounds!) to issue multiple queries and aggregate the backends. That would get you more backends (making some assumptions about your setup) to pass to the load balancer.
Eric Anderson
Feb 1, 2019, 7:03:46 PM2/1/19
to Carl Mastrangelo, grpc.io
You're probably using ELB in L4 LB mode (as a TCP proxy). The three addresses you see are probably the ELB's own addresses. ELB itself would see the greater number of backends.
You can verify that you are understanding the code using a tool like "dig" to issue DNS requests. It shows you all the addresses discovered via DNS. For example:
...
In this example, you see 6 addresses (we only care about the A, not CNAME). You can double-check that what's returned by dig matches what you see when debugging.
--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/db263c58-f65f-4a83-a3e2-37cc60a8622a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages