gRPC streams unexpected disconnections
Amit Waisel
I encountered a weird behavior in gRPC.
The symptom - an active RPC stream is signaled as cancelled on server side (happens from time to time, I couldn't find any correspondence with other events in the environment) although the client is active and the stream shouldn't be closed.
It happens for streams initialized as response streams in RPC calls from both C++ and NodeJS clients. It happened on gRPC 1.3.6 and still happens on gRPC v1.6.0.
The problem does not reproduced easily - the system is executed under heavy load for many hours until this happens.
In my code, I have 2 main types of streams:
- Control stream (C++→C#)
- the client initiates an RPC call to the server, which keeps the RPC's
response stream opened.
Those streams are used as control channels with the C++ clients and are kept open to allow server-to-client requests. When they are closed, both client and server clean up all data related to the connection. So, the control stream is critical to the session.
The server registers on call cancellation notification:
ServerCallContext context; // Received from RPC call as a parameter
// ...
context.CancellationToken.Register(() => System.Threading.ThreadPool.QueueUserWorkItem(async obj => { handle_disconnection(...); }));
The total number of opened control streams (AKA number of connected C++ clients) is ~1200. - Command stream (NodeJS→C#) - There are many many other streams for server-to-client command response communication, which are kept opened in parallel by the server with NodeJS clients. The total number of opened streams is 20K-30K.
The problem is noticeable when the control streams get disconnected.
Further investigation of the client (C++) and server (C#) logs of control stream disconnection, revealed to following:
- For some reason, the server's cancellation token (the one registered above) is signaled - and the server does its cleanup (`handle_disconnection` which also closes many command streams intentionally). According to the client, the connection should have remained opened.
- After some time, the client realizes the connection was closed unexpectedly and does its cleanup - throwing the error I posted here (NodeJS in that case). The clients disconnects itself only after the server disconnects the connection and control stream.
Another note - I set the servers' RequestCallTokensPerCompletionQueue value for both C++/NodeJS client interfaces, to 32768 (32K) per completion queue.
I have 2 server interfaces (for node clients and C++ clients, which have different API), and 4 completion queues (for 8 cores machine). I don't really know if the 4 completion queues are global, or per-server.
Do you think it might cause those streams to be closed under heavy load?
In any case, my suspicious is on the C# server behavior - the CancellationToken is signaled for no apparent reason.
I didn't rule out network instability yet - although both clients and server are located on the same ESX server with 10-gig virtual adapters between them, so this is quite a long-shot.
Do you have any idea how to solve this?
Thanks!
