Unexpected cancel of unrelated streaming clients

47 views
Skip to first unread message

Svante Karlsson

unread,
May 10, 2019, 7:50:39 AM5/10/19
to grpc.io
I've started looking at server streaming in C++ with greeter_async_server.cc as starting point.
I've overridden AsyncNotifyWhenDone() to capture when the client closes the stream. All is good in the "normal" state.

Many clients streams live data endlessly. When I kill a client either with ctrl-c or nicely so it's exiting with "channel.reset()" the server is for sure notified of the exiting connection going away but other unrelated streaming rpc's  (not always and not all of them) also gets closed (and then restarted from the client)  What is going on and is it possible to prevent this unexpected stream close?

/svante




Yash Tibrewal

unread,
May 17, 2019, 2:27:07 PM5/17/19
to grpc.io
That seems strange. What is the error status/message received on the clients for such RPCs?

Svante Karlsson

unread,
May 17, 2019, 4:52:51 PM5/17/19
to Yash Tibrewal, grpc.io
I'll have to get back on that since I solved my "real" problem for now with a rewrite. But I do think that there's a gremlin in the grpc server code. First of all I checked that the time of the server and the clients where within 1 ms. Then I added logging on all three nodes. When I kill a client - the disconnect notification pop up in the other client BEFORE the server gets the notification. Actually a "new" rpc from the disconnected client gets to the server before the disconnect notification of the failed rpc pops up. All this seems strange to me. 

While this was interesting and somewhat annoying my "real world" problem was that I was streaming from kafka and that client has a 3 second set up time. I solved the the problem by caching and sharing consumers that were at somewhat the same position in the stream. Way more logic but startup times went from 3s to less that 1ms.

So if you think there might be a bug I can give a try of chasing it down, but I'm new to grpc and have no knowledge of the internals. Otherwise I'm off the hook.

Thanks

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/86372bc4-11ad-430f-a6ae-72b0f5044dff%40googlegroups.com.

Yash Tibrewal

unread,
May 21, 2019, 6:11:43 PM5/21/19
to grpc.io
I'm glad that you have gotten it to work. If you are able to provide a reproduction, please file a bug on https://github.com/grpc/grpc/issues
To unsubscribe from this group and stop receiving emails from it, send an email to grp...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages