1-2% GRPC traffic gets Dropped at scale With Service Unavailable Error

[Pawan Kumar Sharma]

Hi Guys,

I am seeing 1-2% grpc Traffic gets dropped even after following best practices.

I am using channel polling of size 10-20, best practices suggested in for polling is we have a unique channel name and even with retry added these numbers did not change.

For channel creation I using bellow code:

Map<String, Object> customArgs = new HashMap<>();

customArgs.put("channelNumber", String.valueOf(channelNumber));

ManagedChannel channel = ManagedChannelBuilder.forTarget(grpcUrl)
.usePlaintext()
.defaultLoadBalancingPolicy("round_robin")
.enableRetry()
.maxRetryAttempts(6)
.defaultServiceConfig(customArgs)
.build();

And the Error msg I am getting is:

RankingServiceSort EXCEPTION : UNAVAILABLE: unavailableio.grpc.stub.ClientCalls|toStatusRuntimeException|262
io.grpc.stub.ClientCalls|getUnchecked|243
io.grpc.stub.ClientCalls|blockingUnaryCall|156
proto.RankingServiceGrpc$RankingServiceBlockingStub|sortInventoryData|169
RankingService.RankingGrpcConnector|GetInventoryRanking|80

Please help us we are stuck here from 3-4 months, We are using aws grpc supported ALB in between and these best practices though helped us in reducing DeadLine Acceded issue but service Unavailable cases are not getting reduced. Did now found any detailed documentation as well. We are having 2 instances for this service and both CPU and memory and network bandwidth is not at exhausted, used bellow 5%. 

Thanks & Regards,

Pawan Sharma

[Pawan Kumar Sharma]

Pawan Kumar Sharma

6:03 PM (now) 

to grpc.io

In Local I am using plain Text for production useTransport security is used as aws ALB only supports ssl settings.

So just a corrections:

Map<String, Object> customArgs = new HashMap<>();

customArgs.put("channelNumber", String.valueOf(channelNumber));

ManagedChannel channel = ManagedChannelBuilder.forTarget(grpcUrl)
.useTransportSecurity()

.defaultLoadBalancingPolicy("round_robin")
.enableRetry()
.maxRetryAttempts(6)
.defaultServiceConfig(customArgs)
.build();

Thanks & Regards,

Pawan Sharma

[Larry Safran]

Which version of grpc are you using?

::DISCLAIMER::
----------------------------------------------------------------------------------------------------------------------------------------------------

The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus.com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/9e6eb62d-7125-4978-a231-ed625ef462cbn%40googlegroups.com.

[Larry Safran]

That's an over 5 year old version, can you try it with 1.64.0 and see if it still happens.  Note that there have been a number of security updates over the intervening years so it would be valuable just from that standpoint to be on a relatively current version.

Thanks,

  Larry

On Wed, Jun 19, 2024 at 2:07 AM Pawan Kumar Sharma <pawank...@redbus.com> wrote:

Hi Larry,

I am using go1.21.0.

Thanks & Regards,

Pawan Sharma

[Larry Safran]

I believe that you are talking about the version of the go language rather than the version of grpc-go.

See https://github.com/grpc/grpc-go/releases

On Thu, Jun 20, 2024 at 9:53 PM Pawan Kumar Sharma <pawank...@redbus.com> wrote:

Hello Larry,

1.64 does that even exist, as go documentation these are the only one and last year when I created this service I used the latest one 1.21.

[Kannan J]

If you are using Java client (as seen from the stacktrace in your first message) you can upgrade to 1.64 for Java from https://github.com/grpc/grpc-java/releases.