Force close a connection from the JS client

[Nick Fishman]

Hi Tobias et al,

It's been a while since I was active on here. I'm happy to share that my engineering team has migrated from our previous custom pubsub transport layer (Tornado + redis) to Autobahn (AutobahnJS and AutobahnPython) and in the following months we've seen a noticeable improvement in performance. Our throughput is at least 3500% greater with Autobahn, measured in messages delivered per second without performance degradation.

I've taken notes on a handful of issues, questions, and feedback that I'd like to share in the near future. For now, I wanted to focus on a particular issue I recently discovered.

In the JS client, autobahn.Connection.close doesn't immediately close the socket when there is an active session but instead sends a GOODBYE request in Session.leave. It only closes the socket once it gets a GOODBYE acknowledgement from the server (relevant code: https://github.com/tavendo/AutobahnJS/blob/v0.9.4/package/lib/connection.js#L351). This leads to several unexpected behaviors:

A) RPC callbacks from calls that were made before the close request could be invoked after connection.close() is called.

B) PubSub subscription callbacks can be invoked after connection.close() is called.

Because there's no guarantee when the Autobahn server will process the client's GOODBYE request relative to other messages intended for the client, the client may or may not observe either of these behaviors. Both of these are race conditions and have caused errors in our application. In our case, here's what is happening:

1) We call connection.close() if the Autobahn client stops getting responses to its keepalive messages after 30 seconds. We implement these keepalives as RPCs though it would be nice if Autobahn implemented client/server keep-alive itself, as I'm sure other developers would benefit from it.

2) We then immediately create a new Connection object, call open on it, resubscribe to all relevant topics, and kick off a new round of RPC keepalives.

3) In some cases the original Autobahn client (the one we called close on in #1) regains connectivity and suddenly receives a whole bunch of RPC callbacks and subscription callbacks. That's because the server hasn't yet processed its GOODBYE request. This causes all sorts of havoc for our application since we now have two Autobahn connections (the "zombie" connection in #1 that temporarily came back from the dead then gets terminated by the server, and the new connection we created in #2).

A few questions/comments:

- Would it be possible to prevent these behaviors in the JS library? It would be nice for the Session to invalidate any incoming messages if self._goodbye_sent is True.

- At the very least, it would be helpful to mention these race conditions in the documentation so other developers don't encounter errors from false assumptions like I did.

- In the meantime, do you have suggestions on the best way to force a hard disconnect to eliminate the possibility of these behaviors? In this case, the client would close its socket immediately without waiting for a GOODBYE acknowledgement. My initial implementation involves calling connection.close() followed by connection._transport.close() but I'm wondering if there's a better way.

Thanks,

Nick

[Tobias Oberstein]

Hi Nick,

> It's been a while since I was active on here. I'm happy to share that my
> engineering team has migrated from our previous custom pubsub transport
> layer (Tornado + redis) to Autobahn (AutobahnJS and AutobahnPython) and
> in the following months we've seen a noticeable improvement in
> performance. Our throughput is at least 3500% greater with Autobahn,
> measured in messages delivered per second without performance degradation.

This is awesome! Also: real world user experience like this might be of
interest to developers in general - means: probably you have a blog post
about this I could share/RT?

>
> I've taken notes on a handful of issues, questions, and feedback that
> I'd like to share in the near future. For now, I wanted to focus on a
> particular issue I recently discovered.
>
> In the JS client, autobahn.Connection.close doesn't immediately close
> the socket when there is an active session but instead sends a GOODBYE
> request in Session.leave. It only closes the socket once it gets a
> GOODBYE acknowledgement from the server (relevant code:
> https://github.com/tavendo/AutobahnJS/blob/v0.9.4/package/lib/connection.js#L351).
> This leads to several unexpected behaviors:
> A) RPC callbacks from calls that were made before the close request
> could be invoked after connection.close() is called.
> B) PubSub subscription callbacks can be invoked after connection.close()
> is called.

From my point of view, this is exactly what I'd expect: everything
_before_ calling close is still processed.

>
> Because there's no guarantee when the Autobahn server will process the
> client's GOODBYE request relative to other messages intended for the

SThe is guarantee: it will not process any messages _after_ having
received GOODBYE.

> client, the client may or may not observe either of these behaviors.
> Both of these are race conditions and have caused errors in our
> application. In our case, here's what is happening:
> 1) We call connection.close() if the Autobahn client stops getting
> responses to its keepalive messages after 30 seconds. We implement these
> keepalives as RPCs though it would be nice if Autobahn implemented
> client/server keep-alive itself, as I'm sure other developers would
> benefit from it.

Yeah, IMO this shouldn't be handled at app level, but transport level.

Crossbar.io has configurable WebSocket ping/pong/keepalive/timeout.
Checkout the "auto_ping_XXX" options here

http://crossbar.io/docs/WebSocket-Options/

The revised WAMP-over-RawSocket transport also has transport level
ping/pong and Crossbar.io will have that too.

> 2) We then immediately create a new Connection object, call open on it,
> resubscribe to all relevant topics, and kick off a new round of RPC
> keepalives.
> 3) In some cases the original Autobahn client (the one we called close
> on in #1) regains connectivity and suddenly receives a whole bunch of
> RPC callbacks and subscription callbacks. That's because the server
> hasn't yet processed its GOODBYE request. This causes all sorts of havoc
> for our application since we now have two Autobahn connections (the
> "zombie" connection in #1 that temporarily came back from the dead then
> gets terminated by the server, and the new connection we created in #2).
>
> A few questions/comments:
> - Would it be possible to prevent these behaviors in the JS library? It
> would be nice for the Session to invalidate any incoming messages if
> self._goodbye_sent is True.

We could add an option to make it behave like above. If you are
interested, please file an issue to AutobahnJS.

One open question then is: what about outstanding promises for calls not
fired yet? Reject them so the app can react? Or just do nothing with those?

There is a related issue already:
https://github.com/tavendo/AutobahnJS/issues/27

> - At the very least, it would be helpful to mention these race
> conditions in the documentation so other developers don't encounter
> errors from false assumptions like I did.

I wouldn't call it a race condition, since it is deterministic behavior:
everything _before_ the client starts to close is still processed.
Everything after not. But I nevertheless can see your point/issue. So
let's address that.

> - In the meantime, do you have suggestions on the best way to force a
> hard disconnect to eliminate the possibility of these behaviors? In this
> case, the client would close its socket immediately without waiting for
> a GOODBYE acknowledgement. My initial implementation involves calling
> connection.close() followed by connection._transport.close() but I'm
> wondering if there's a better way.

Yes, you can just hard close the underyling transport (WebSocket).

Cheers,
/Tobias

>
> Thanks,
>
> Nick
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/940e4358-ed2d-4a68-a1a4-c509fe295d34%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/940e4358-ed2d-4a68-a1a4-c509fe295d34%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.