How to analyze potential memory leak in Grpc-Java

[Sammi Chen]

There is a lot "io.grpc.StatusRuntimeException: CANCELLED: client cancelled" error in server side.

And occationally,  I will get "io.netty.util.internal.OutOfDirectMemoryError: failed to allocate *** byte(s) of direct memory (used: ***, max: ***)" error on server side too.

Set "io.netty.leakDetection.level=paranoid",  there is no leak related LOGs.

Server side function looks like this,

@Override
public StreamObserver<CommandRequestProto> send(
StreamObserver<CommandResponseProto> responseObserver) {
return new StreamObserver<CommandRequestProto>() {

@Override
public void onNext(CommandRequestProto request) {
try {
CommandResponseProto resp =
function(request);
responseObserver.onNext(resp);
} catch (Throwable e) {
LOG.error("Got exception when processing"
+ "CommandRequestProto {}", request, e);
responseObserver.onError(e);
}
}

@Override
public void onError(Throwable t) {
// for now we just log a msg
LOG.error("Command send on error. Exception: ", t);
}

@Override
public void onCompleted() {
LOG.debug("Command send completed");
responseObserver.onCompleted();
}

};

My question is SHOULD I call responseObserver.onCompleted() in onError too?

I found there is inconsistent answers on stack overflow,

This one

https://stackoverflow.com/questions/64257725/can-someone-explain-to-me-whats-the-proper-usage-of-grpc-streamobserver-onerror

Says that onError/onCompleted should not call requestObserver.

While this one

https://stackoverflow.com/questions/65488395/grpc-oncomplete-for-bidistream

Says if you don't call requestObserver.onCompleted, there will be a memory leak.

I'm not sure which one is corrent.  I'd like to hear the more authoritative answer.

Should server side request StreamObserver.onError/onCompleted call response's StreamObserver.onCompleted?

Thanks in advance.

[sanjay...@google.com]

> My question is SHOULD I call responseObserver.onCompleted() in onError too?

It will be a good idea to call responseObserver.onError() inside your server's request-observer e.g.

@Override
public void onError(Throwable t) {

responseObserver.onError(t);   // or alternatively return a fixed throwable instead of the throwable you received 

LOG.error("Command send on error. Exception: ", t);
}

The reason it's a good idea is because even though gRPC code doesn't need it you could have some interceptor code that leaks resources in such situations.

> I found there is inconsistent answers on stack overflow,

One of the links is for the client side so that logic doesn't apply here.

> And occationally,  I will get "io.netty.util.internal.OutOfDirectMemoryError: failed to allocate *** byte(s) of direct memory (used: ***, max: ***)" error on server side too.

This is unlikely to be caused by the above but could possibly be because of lack of flow control in your implementation.