How to correctly handle UNAVAILABLE: Abrupt GOAWAY returned from nginx

[chetan sood]

Hi

We are seeing the following exceptions in our grpc-java client when it's trying to send a rpc to a grpc-server that is configured behind nginx.

io.grpc.StatusRuntimeException: UNAVAILABLE: Abrupt GOAWAY closed sent stream. HTTP/2 error code: NO_ERROR

I went through the detailed conversation here - https://github.com/grpc/grpc-java/issues/8310.

I understand that nginx is not sending 2 GOAWAY frames and thats the reason for the issue in our case. I believe this is the case since we don't see the requests reaching the grpc server.

It sounds like the grpc-client should retry once automatically as part of transparent retry mechanism. Is my understanding correct here? 

We don't enable retry on the managed channel (https://grpc.github.io/grpc-java/javadoc/io/grpc/ManagedChannelBuilder.html#enableRetry--) . Should we still expect the grpc-client to retry once in case of transparent retries?

In our case we see that the request doesn't reach the server and is terminated by nginx, so we would expect a subsequent retry to work and we shouldn't see these errors. Is there another reason why we might be seeing this? At the end of the conversation, ejona86 mentions that a netty bug is also fixed in 1.41 grpc version. We currently use 1.37 grpc version. Is that the reason the transparent retry is not working for us and we are seeing the above errors? Will upgrading to 1.41 resolve the issue?

I'm planning to upgrade our client to 1.41 however I'm not certain if that's the only change that's needed here. Please let me know if I need to add something else.

Thanks

[sanjay...@google.com]

Upgrading to grpc 1.41 is definitely worth trying because of this https://github.com/grpc/grpc-java/pull/8359 .

[Eric Anderson]

On Sat, Mar 12, 2022 at 10:13 PM 'sanjay...@google.com' via grpc.io <grp...@googlegroups.com> wrote:

Upgrading to grpc 1.41 is definitely worth trying because of this https://github.com/grpc/grpc-java/pull/8359 .

I agree, although that fix was backported to v1.40.0 (but use newer v1.40.2). Nginx's behavior will fail RPCs. You need retries to avoid that. v1.40.0 enabled retries, and has the fix to improve this specific situation. But in general, we'd recommend upgrading to one of the two most recent minor releases.

[chetan sood]

Thanks for the response. I have updated the version to 1.41 and waiting for it to roll out to Production to see if it helps.