grpc call timeout & exception handling

[Eugene Strulyov]

Hi all,

I am trying to use grpc for a new project. The service is in java and the client in python. I have questions about exception handling and timeouts.

A grpc call seemed to get stuck. The client wasn't getting a response and the server didn't seem to do anything. This particular call uses client -> server streaming. Turned out that a RuntimeException was thrown in onNext() method and was not caught, so neither onError() nor onCompleted() ended up getting called. I changed the code to do the following:

@Override

public void onNext(...) {

  try {

    // do stuff

  } catch (Exception ex) {

    responseObserver.onError(ex);

  }

} 

This fixed the problem, but it raised the following questions:

1. Do I really need to catch all exceptions and explicitly call onError() ? Uncaught exception does not abort the call?

2. I would expect the call to time out at some point. It would be really *really* undesirable to leave both the client and the server in a zombie state. The timeout does not seem to be happening.

Please advise. Thanks.

Eugene

[Nathaniel Manista]

On Wed, Dec 14, 2016 at 5:08 PM, Eugene Strulyov <eugene....@gmail.com> wrote:

2. I would expect the call to time out at some point. It would be really *really* undesirable to leave both the client and the server in a zombie state. The timeout does not seem to be happening.

Please share with us your client-side code? It would be good for us to see what your timeout value is and how you are passing it when making your call.

-Nathaniel

[Eugene Strulyov]

I'm not setting explicit timeout on the client. It would be too error-prone to set it on every call. Is there a way to set default timeout on either the client or the server?

thanks,

Eugene

[Nathaniel Manista]

On Wed, Dec 14, 2016 at 6:01 PM, Eugene Strulyov <eugene....@gmail.com> wrote:

I'm not setting explicit timeout on the client. It would be too error-prone to set it on every call.

Please reconsider this - after years working on different Google production systems I came to the gRPC project with the attitude that making an RPC without a timeout (or deadline) is asking for trouble and initially designed gRPC Python with mandatory timeouts. I only made them non-mandatory in the final API to match the APIs of the other gRPC languages, not because something has changed my mind and I now think making an RPC without a timeout is a responsible way to code a distributed system.

Is there a way to set default timeout on either the client or the server?

There might be in other languages, but not in Python - we tried it in our Alpha API and it didn't bring the user benefit we wanted from it so it is no longer present.

On Wednesday, December 14, 2016 at 5:11:37 PM UTC-8, Nathaniel Manista wrote:

On Wed, Dec 14, 2016 at 5:08 PM, Eugene Strulyov <eugene....@gmail.com> wrote:

2. I would expect the call to time out at some point.

If you don't pass a (client side) timeout value then you should not have any (client side) expectation that the RPC will time out.

It would be really *really* undesirable to leave both the client and the server in a zombie state.

Agreed - but without having passed a timeout value on the client side, if your server side code has a bug in which it hangs the RPC forever then your client side code will hang forever waiting for the conclusion of the hung RPC.

The timeout does not seem to be happening.

This is expected for RPCs invoked without a timeout.

-N

[Eugene Strulyov]

Is there a way to set timeout on the server (the server is in java) ?

Is there a way to set a default timeout for ALL calls on the client? I definitely would not want to do it before every single call (unless it actually overrides the default).

Also, getting back to (1), why doesn't uncaught exception cause the RPC to abort immediately? Instead, both the client and the server simply get stuck.

thanks,

Eugene

[Carl Mastrangelo]

If an exception is thrown in a StreamObserver, it will be caught in the JumpToApplicationThreadServerStreamListener and cause the stream to be closed.  Eventually the exception will propagate back back to the serializing executor contained in the Server, which will log the exception.   

Server side deadlines are set before the ServerInterceptors are added.  These are normally derived from the deadline provided by the client.   You can add your own ServerInterceptor to add a deadline if you would like, though. 

[Eugene Strulyov]

If an exception is thrown in a StreamObserver, it will be caught in the JumpToApplicationThreadServerStreamListener and cause the stream to be closed.  Eventually the exception will propagate back back to the serializing executor contained in the Server, which will log the exception.   

That is not the behaviour I observed. Uncaught exception in onNext() went nowhere and was not logged (using grpc version 1.0.1).

 

Server side deadlines are set before the ServerInterceptors are added.  These are normally derived from the deadline provided by the client.   You can add your own ServerInterceptor to add a deadline if you would like, though. 

If the client does not specify a deadline, is there no deadline at all? Can you point me to an example to set up a default deadline on the server?

thanks,

Eugene

[Carl Mastrangelo]

If that is not the case, it is a bug.  Could you file an issue at https://github.com/grpc/grpc-java/issues along with a reproduce-able case?