Async response never received
38 views
Skip to first unread message
randy....@gmail.com
Nov 9, 2017, 10:58:45 AM11/9/17
to grpc.io
gRPC 1.6.7 and Ubuntu 17.04 installed on a MacBook Pro.
I created a multi-threaded server based on the canonical async server example where threads block on Next(), get a state object in the tag, do synchronous calls to another server to get some data, then build a response and use ServerAsyncResponseWriter to send a response. The state object is basically just like the CallData object in the example and creates a new request when a RPC comes in the same was as the CallData object.
90-ish% of the time, this works fine. I have tested the server with 1 up to 4 threads. A client program (provided to me, i.e. I can't modify it) can spawn a specified number of threads to make simultaneous synchronous calls to my server. If the client ever uses more than one thread, regardless of the number of threads in the server, periodically the client will hang waiting for a response from my server.
I was able to verify the following:
* The Finish() process on the ServerAsyncResponseWriter completes successfully. A Next() call succeeds with `ok` set to True for that state object after the Finish() call.
* Using Wireshark, I am able to see the actual response packet leave my server. However, the client never ACKs the packet, a TCP Retransmission occurs that is ACK'd by the client, but the client hangs at this point.
Has anyone seen something similar? With a single thread, my server is essentially the async server example.
I created a multi-threaded server based on the canonical async server example where threads block on Next(), get a state object in the tag, do synchronous calls to another server to get some data, then build a response and use ServerAsyncResponseWriter to send a response. The state object is basically just like the CallData object in the example and creates a new request when a RPC comes in the same was as the CallData object.
90-ish% of the time, this works fine. I have tested the server with 1 up to 4 threads. A client program (provided to me, i.e. I can't modify it) can spawn a specified number of threads to make simultaneous synchronous calls to my server. If the client ever uses more than one thread, regardless of the number of threads in the server, periodically the client will hang waiting for a response from my server.
I was able to verify the following:
* The Finish() process on the ServerAsyncResponseWriter completes successfully. A Next() call succeeds with `ok` set to True for that state object after the Finish() call.
* Using Wireshark, I am able to see the actual response packet leave my server. However, the client never ACKs the packet, a TCP Retransmission occurs that is ACK'd by the client, but the client hangs at this point.
Has anyone seen something similar? With a single thread, my server is essentially the async server example.
randy....@gmail.com
Nov 14, 2017, 1:57:24 PM11/14/17
to grpc.io
Not really sure what the problem was, but updating to 1.7.2 fixed it with zero code changes on my part.
Christopher Warrington - MSFT
Nov 14, 2017, 9:59:23 PM11/14/17
to grpc.io
On Tuesday, November 14, 2017 at 10:57:24 AM UTC-8, randy....@gmail.com wrote:
> Not really sure what the problem was, but updating to 1.7.2 fixed it with
> zero code changes on my part.
I observed something similar on Windows. An upgrade to 1.7.1 fixed it for me
as well. I bisected the gRPC implementation and commit be9b8142, "Add
ChannelConnectivityWatcher" appears to be the one with the fix. Though, the
connectivity watching has been refactored from a C++ library implementation
into the C core in commit 7269667f, "Add client channel connectivity
watcher". There's some speculation here, as its quite difficult to prove
that a hang has gone away. :-)
Reply all
Reply to author
Forward
0 new messages