(golang) Detecting a dead client

[Aaron Beitch]

Hello,

tl;dr: Does HTTP2 do any heartbeat with the client? Can a streaming rpc's context.Done channel be closed if the client dies ungracefully?

In my application I have bidirectional communication. I implement this with one unary rpc for client-to-server messages and one streaming rpc for server-to-client communications. When a client dies ungracefully my server leaks resources because there is a goroutine sitting around waiting for data to send to the client, but never notices the client disconnected.

I was hoping that just waiting on the ctx.Done channel would notify me when the client disappears, but that doesn't appear to work. I could add a heartbeat mechanism, but I was hoping to not have to do that. Does HTTP2 do any heartbeat with the client? Can a streaming rpc's context.Done channel be closed if the client dies ungracefully?

Thanks,

Aaron

[Qi Zhao]

Depending on what the definition of "ungracefully", we have different answers:

i) If TCP stack gets a chance to send RST to the server, the server will find this connection error immediately and close all contexts;

ii) if it is really "ungraceful" (e.g., unplug the network cable), right now grpc relies on TCP user timeout to detect it, which may take tens of minutes depending on the kernel configuration. Unfortunately, we have not fleshed out a server side heart beat to detect this (in our radar though) by now. Thus, users might need to rely on their own health checking mechanism for now.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/3d923856-3c73-4590-bb36-7b9af29e9a8c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,

-Qi

[Paul Breslin]

This seems to still be an issue. We have a server side stream read implementation that is forced to manually timeout because the Done channel doesn't notify when the client side goes away.

[Qi Zhao]

We will have heart-beat mechanism ready to tackle this kind of issues soon (Q1 2017).

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.

To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.

Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/af4f2bb6-07ac-43bf-9bd4-4f1b6ad72d72%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

-Qi

[pawel...@gmail.com]

Just a notice for future readers. As today (2017.07.25) it's enough to listen on context.Done().

func (p *XImpl) Listen(_ *empty.Empty, rec services.X_ListenServer) error {

c := make(chan *services.X, 16)

p.add(c)

defer p.remove(c)

ctx := rec.Context()

for {

select {

case v := <-c:

if err := rec.Send(v); err != nil {

return err

}

case <-ctx.Done():

return nil

}

}

}

[Eric Anderson]

On Wed, Aug 30, 2017 at 1:39 AM, <ismail...@gmail.com> wrote:

Was the heartbeat solution ever implemented?

See https://godoc.org/google.golang.org/grpc#KeepaliveParams . ServerParameters.Time and Timeout.

[yz]

My concern is that if the client connection is closed, no matter gracefully or not, the application layer might need to do some cleaning according to the event, e.g., to remove authentication information from the server. Haven't find a way to do that.

[Eric Anderson]

On Tue, Jan 23, 2018 at 3:32 AM, 'yz' via grpc.io <grp...@googlegroups.com> wrote:

My concern is that if the client connection is closed, no matter gracefully or not, the application layer might need to do some cleaning according to the event, e.g., to remove authentication information from the server. Haven't find a way to do that.

Any RPCs that were active at the time will be closed. That is a notification is good to use. If you're using the TCP connection as a session, then I'd suggest you strongly consider another option, like some simplistic caching; there are lots of things that can go wrong with using knowledge of which TCP connection and RPC is using.