Issue with client closing connections in grpc java

[shishi...@go-jek.com]

Hello folks. I'd appreciate some help with an issue we faced recently. Some context, our organization has been using gRPC for quite some time, so there are legacy services running older versions of gRPC. One such service is running version 1.0.1 and we faced an issue where we got a JVM OOM and the netty server was unable to create more threads. Before that, for quite some time there were `java.io.IOException: Connection reset by peer` exceptions (caused by a timeout configured at a fronting Envoy setup). The server uses server side streaming RPCs.

I was wondering if this was reported earlier and has been fixed in recent releases and if an upgrade would fix this. We are regardless going to upgrade our gRPC version.

[Eric Anderson]

On Mon, Sep 3, 2018 at 7:53 AM <shishi...@go-jek.com> wrote:

One such service is running version 1.0.1 and we faced an issue where we got a JVM OOM and the netty server was unable to create more threads.

If you get a OOM, frequently the process is hosed. It is simply impossible to recover from OOMs in some parts of the code. Since every line of Java is impacted differently from an OOM, these can only be discussed on a case-by-case basis.

Before that, for quite some time there were `java.io.IOException: Connection reset by peer` exceptions (caused by a timeout configured at a fronting Envoy setup).

That error is benign. Later versions of gRPC squelch it (hide it).