Detecting client disconnections in the server side

[gust...@gmail.com]

Hi all,

How can I detect in the server side when a client gets disconnected?  The typical use case could be a chat server where you want to notify other users when somebody leaves ungracefully.  

I haven't been able to find a clear answer to this question yet.    I'm using Go implementation.

Regards,

[Carl Mastrangelo]

You can't really tell if the client is accidentally disconnected, but you can find out with a few seconds by turning on keep-alives.  But, for your question: why do you want to know?  What different course of action would you take if you did know?

[Gustavo García]

Hi Carl,  Thank you for your answer.

This is a chat application, so when a user disconnects you want to tell everybody else in the room that somebody left.    That's why I need to detect it.  Does it makes sense?

If I turn on keep-alives, how would I detect the connection drop in a GRPC Go server? (or other language if there is any limitation in Go).

Thx a loot

[Carl Mastrangelo]

I'm actually more familiar with Java, though the architectures aren't that different.  Most of gRPC is centered around the Call, rather than the Connection. If it were me, I would make a custom RPC that implies "Disconnecting".  This would also allow you to embellish the disconnect (like add a going away message).   If you really care about the connection dropping, at least in Java, you can tell by installing a ServerTransportFilter and checking onTerminated, thought this doesn't help with your Go program.  I'll let someone else speak for that.  Still though, my personal take is to use an RPC to implement it.

[Arpit Baldeva]

For Go, this thread claimed that something like this would be available in Q1 2017 - https://groups.google.com/forum/#!topic/grpc-io/C0rAhtCUhSs 

Regardless of the language (I am working with C++ impl), my plan to implement this functionality is also RPC based (streaming Ping rpc). On server side, you can detect the client disconnection via AsyncNotifyWhenDone as discussed here (https://github.com/grpc/grpc/issues/3956 )

[Constantine]

I have a nodejs server and a nodejs client. With a "streaming client <--> streaming server" when

• I call the .end() method on the client object, the .on('end') event is called on the server
• I disconnect the client code with ctrl+c, also the .on('end') event is called on the server

(I shared my code below.)

it means that the server will understand that a client is dead(disconnected), so there will be no more resource consumption on the server. Did I miss something or not?

I don't know how to implement the heart-beat on this situations, any clue would be appreciated.


Server.js

var PROTO_PATH = __dirname + '/../../protos/hellostreamingworld.proto';

var grpc = require('grpc');

var hello_proto = grpc.load(PROTO_PATH).hellostreamingworld;

/**

 * Implements the SayHello RPC method.

 */

function sayHello(call) {

  call.on('data', () => {

    call.write('recieved')

  })

  call.on('end', () => {

    console.log(call.request);

  })

  console.log("call object is: ", call);

}

/**

 * Starts an RPC server that receives requests for the Greeter service at the

 * sample server port

 */

function main() {

  var server = new grpc.Server();

  server.addProtoService(hello_proto.MultiGreeter.service, {sayHello: sayHello});

  server.bind('localhost:50051', grpc.ServerCredentials.createInsecure());

  server.start();

  console.log('server is running :) ')

}

main();

Client.js

var PROTO_PATH = __dirname + '/../../protos/hellostreamingworld.proto';

var grpc = require('grpc');

var hello_proto = grpc.load(PROTO_PATH).hellostreamingworld;

var client = new hello_proto.MultiGreeter('localhost:50051', grpc.credentials.createInsecure());

var user;

if (process.argv.length >= 3) {

  user = process.argv[2];

} else {

  user = "world";

}

call = client.sayHello()

call.on('data', (message) => {

  console.log("server said: [%s]", message.message);

})

call.on('end', () => {

  console.log("end @ client");

})

call.write({name: "hadi", num_greetings: "a"})

call.write({name: "hadi", num_greetings: "a"})

call.end() // you can ommit this and stop the process by ctrl-c. same behaviour

[Michael Lumish]

I don't understand what the problem is. You say that you get the 'end' event on the server side when the client disconnects. What else do you need?

Also, with streaming calls, you need to call call.end() on the server side too. The callback for the 'end' event is usually a good place to do that.

--
You received this message because you are subscribed to the Google Groups "grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email to grpc-io+u...@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/1e69ed74-6db4-4d2a-966e-a5302a696dea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Constantine]

Thank you for .end() on the server, I forgot it. I just want to say that it is working in the nodejs server. But guys here talking about heart-beat and other stuff, I was just curious how to implement it within gRPC if it is not supported yet :) Sorry if it was annoying. 

[mmu...@google.com]

We are working on providing a solution for such a case: If the other side of the connection becomes unresponsive due to some reason the connection is closed and the any stream(RPC) reading on it will get an error that the connection was closed.

For the chat application, I'm assuming a streaming RPC would be established between server and client and each side must have a routine reading on this stream. In case of a connection error this reading routine will get an error.

We're calling this mechanism a point-to-point healthcheck. The client-side work is done and the server-side is underway.

[mmu...@google.com]

ps The last message was pertaining to golang.

[Mahak Mukhi]

https://github.com/grpc/grpc-go/blob/master/keepalive/keepalive.go

On Wed, Aug 30, 2017 at 1:46 AM, <ismail...@gmail.com> wrote:

Was this ever implemented? Specifically, a way for grpc-go to detect if a client is disconnected from a server->client stream? Can't find it on the grpc-go repo...

[Arpit Baldeva]

I think most people asking this question are interested in knowing whether a particular client/user is dead. The keepalive stuff is useful for grpc to internally manage the network resources clean up. However, as the network layer is hidden, anybody who wants to detect the state of a particular client has to do it at their application code level (say by implementing a a streaming rpc and using heartbeat mechanism). 

https://github.com/grpc/grpc-go/blob/master/keepalive/keepalive.go

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/C9rTVKZqqp0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAD6XTdesZi-fpzLFmJ8Z_dyX9%2BW9-ccvbkPa_bVAuyVDCqU4XA%40mail.gmail.com.

[Carl Mastrangelo]

The short answer is that you can't tell when a connection has gone away.  Even gRPC can't really be sure when a connection is gone (we can guess).  Like I suggested before, why not send a "Going away" message just before disconnect?  This is the same solution that HTTP/2 uses under the hood (called "go away").  

[Arpit Baldeva]

>>Like I suggested before, why not send a "Going away" message just before disconnect?

I am not sure when/what context you made that suggestion. 

Anyhow, "Going away" message is good for graceful close. For ungraceful exits, a heartbeat rpc is the way to go in gRPC. That rpc can be implemented in a specific fashion that the application needs. To clarify, yes, I agree, this isn't something gRPC can do for the application. 

[ismail...@gmail.com]

Is listening on the context.Done() channel in Go (on the server) sufficient? I ran a client and server locally with a client listening on a server->client stream, killed the client process, and saw that context.Done() was indeed called on the server. 

I'm assuming that killing a client process is what some of you are referring to as an "ungraceful exit". Under what scenario would listening on context.Done() not work?

As of now I'm happy to listen on context.Done() and implement a heartbeat RPC, but am curious to know when context.Done() will fail.

https://github.com/grpc/grpc-go/blob/master/keepalive/keepalive.go

To unsubscribe from this group and all its topics, send an email to grpc-io+u...@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

[ismail...@gmail.com]

Just to add, this is the context.Done() solution that I tried: https://groups.google.com/d/msg/grpc-io/C0rAhtCUhSs/SzFDLGqiCgAJ

[Arpit Baldeva]

It just depends on your application. From what I can tell based on your description, you have a single rpc executing between client and server. If you have many rpcs exposed, you wouldn't want to run a 'clean up' for each onDone call.

Also, by "ungraceful disconnect", I also include any situation that results in an absence of communication between client and server (for example, if a cable is pulled on one of them, it might take a while for the other machine to notice this).

Some applications may only want to detect client state at server or server state at client or both. They may also want to know the state 'on demand'. So all these can be accomplished with a 'heartbeat rpc' per client depending on the need of a specific application.

Frameworks I have worked in the past expose the actual socket/client connection which can make this particular case slightly easier. Grpc hides network layer (and rightly so) so it just needs a bit of additional work to accomplish but can be done.

Thanks.   

To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.

To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/36b01f54-f741-4bae-a6e3-90aa7d5b9cc7%40googlegroups.com.