PING and PONG message type

[Matt Bonneau]

I have been working on methods to test if a transport is still good.

Different transports implement varying ways to validate the connection, some easier than others to use.

I think it may be helpful to add PING and PONG message types to the basic spec. Either peer can send a PING with a request id and an arbitrary payload and the other side MUST respond with a PONG and the corresponding request id and payload.

This would provide a few different advantages to using transport-specific methods:

- There is no transport-specific implementation needed to validate the connection

- It would allow measurement of latency of the entire protocol stack

The reason it would be good to add it to the basic spec is that it would create a standard way for all peers basic and advanced to check the connection.

Thanks,

Matt

[Tobias Oberstein]

Hi Matt,

Am 05.08.2014 04:33, schrieb Matt Bonneau:
> I have been working on methods to test if a transport is still good.
>
> Different transports implement varying ways to validate the connection,
> some easier than others to use.

Yes. WebSocket has pings/pongs built in, but RawSocket/TCP does not have
anything (since it's basically plain TCP plus minimal framing).

>
> I think it may be helpful to add PING and PONG message types to the
> basic spec. Either peer can send a PING with a request id and an
> arbitrary payload and the other side MUST respond with a PONG and the
> corresponding request id and payload.
>
> This would provide a few different advantages to using
> transport-specific methods:
>
> - There is no transport-specific implementation needed to validate the
> connection
> - It would allow measurement of latency of the entire protocol stack
>
> The reason it would be good to add it to the basic spec is that it would
> create a standard way for all peers basic and advanced to check the
> connection.

I can see your arguments. I have reopened

https://github.com/tavendo/WAMP/issues/3

to discuss this further. Currently, the advanced profile has

https://github.com/tavendo/WAMP/blob/master/spec/advanced.md#heartbeat-1

Cheers,
/Tobias



>
> Thanks,
> Matt
>
> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com
> <mailto:wampws+un...@googlegroups.com>.
> To post to this group, send email to wam...@googlegroups.com
> <mailto:wam...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/wampws.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/wampws/f1a273fb-c2f3-455c-978b-aa1144f24d34%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/f1a273fb-c2f3-455c-978b-aa1144f24d34%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Matt Bonneau]

The heartbeat would be useful in detecting certain problems, but the PING/PONG gives a definitive answer that the peer on the remote end is in fact still processing messages and the transport is good.

There are certain cases where it would be beneficial for the router to be able to probe a client. If, for example, you have a client that registers an RPC call, and then disappears from the network without closing the connection. That same client then comes back online and tries to register the RPC again, only to get an error because the previous session still holds the registration. The only way right now to overcome this quickly (but still not 100% effectively) would be to try and call the RPC to generate traffic and hope that the transport shuts down quickly from some network error.

If the PING/PONG messages existed, the router could check the existing session to make sure it is up.

The HEARTBEAT would probably be able to generate the traffic to do this (assuming underlying TCP) if the router wanted to send some traffic - but it would require the advanced spec, and would not necessarily give the router feedback to know if the client is still processing messages.

How are cases like this handled in crossbar.io?

Matt

[Bas Wegh]

It would be great to have that feature in the offical WAMP spec.

I also did implement some kind of PING/PONG messages to keep track of the connection status.

Also we should consider some kind of meta event in case of disconnection/new connection in the advanced protocol.

Cheers,

Bas

[Alexander Gödde]

Hi!

Current Crossbar (0.9.7-3) implements a meta event for new sessions and session disconnects:

"wamp.metaevent.session.on_join"

and

"wamp.metaevent.session.on_leave"

Unfortunately, neither has made it into the WAMP documentation or the WAMP advanced profile spec yet.

Is this what you would like to see as a feature?

Regards,

Alex

--

You received this message because you are subscribed to the Google Groups "WAMP" group.

To unsubscribe from this group and stop receiving emails from it, send an email to wampws+un...@googlegroups.com.
To post to this group, send email to wam...@googlegroups.com.

Visit this group at http://groups.google.com/group/wampws.

To view this discussion on the web visit https://groups.google.com/d/msgid/wampws/7c69ed2f-aece-411f-8839-10fd16042444%40googlegroups.com.

[Matt Bonneau]

While these are nice features that I didn't know about, I think a test for "liveness" is still good to have and can aid in other ways too. I assume the meta events are sent from the router. If the router has no idea that the client is no longer there, or no longer processing messages, then would the meta events ever get sent?

Also, is there a list of "standard" meta events?

Regards,

Matt

[Bas Wegh]

Would be great in combination with The Ping and pong message type.
So the router can check the connections and kill the session on timeout.
And then send the meta event.

Cheers,
Bas

[Tobias Oberstein]

Am 09.08.2014 20:00, schrieb Matt Bonneau:
> While these are nice features that I didn't know about, I think a test
> for "liveness" is still good to have and can aid in other ways too. I

Yes. Generating meta events for session join/leave is _orthogonal_ to
the the _detection_ of "leave".

> assume the meta events are sent from the router. If the router has no
> idea that the client is no longer there, or no longer processing
> messages, then would the meta events ever get sent?

No.

You are right: when a client just disappears (WAN network link gets
broken or such), the router currently will not detect that immediately.

This is TCP. And this is why sth like ping/pong (or heartbeat with ACK)
is necessary. It needs to be addressed.

>
> Also, is there a list of "standard" meta events?

No. This is still in flux in the spec (advanced profile).

As Alex mentioned, Crossbar.io has

http://crossbar.io/docs/Session-Metaevents/

It's not described in WAMP AP
(https://github.com/tavendo/WAMP/issues/80). But we just needed it for
an app right now.

>
> Regards,
> Matt
>
> On Saturday, August 9, 2014 6:10:58 AM UTC-4, Alexander Gödde wrote:
>
> Hi!
>
> Current Crossbar (0.9.7-3) implements a meta event for new sessions
> and session disconnects:
>
> "wamp.metaevent.session.on_join"
>
> and
>
> "wamp.metaevent.session.on_leave"
>
> Unfortunately, neither has made it into the WAMP documentation or
> the WAMP advanced profile spec yet.
>
> Is this what you would like to see as a feature?
>
> Regards,
>
> Alex
>
>

> 2014-08-08 16:02 GMT+02:00 Bas Wegh <bas....@gmail.com <javascript:>>:

> <http://crossbar.io>?

> it, send an email to wampws+un...@googlegroups.com <javascript:>.

> To post to this group, send email to wam...@googlegroups.com

> <javascript:>.

> Visit this group at http://groups.google.com/group/wampws

> <http://groups.google.com/group/wampws>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/wampws/7c69ed2f-aece-411f-8839-10fd16042444%40googlegroups.com

> <https://groups.google.com/d/msgid/wampws/7c69ed2f-aece-411f-8839-10fd16042444%40googlegroups.com?utm_medium=email&utm_source=footer>.

>
> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com

> <mailto:wampws+un...@googlegroups.com>.

> To post to this group, send email to wam...@googlegroups.com

> <mailto:wam...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/wampws.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wampws/b4122a1c-0c6c-4912-8bf5-233f05fe7851%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/b4122a1c-0c6c-4912-8bf5-233f05fe7851%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tobias Oberstein]

Yes.

FWIW, I now implemented automatic ping/pongs in AutobahnPython (both
client and server) - WebSocket based for now:

https://github.com/tavendo/AutobahnPython/blob/autopingpong/examples/twisted/websocket/pingpong_keepalive/server.py

That works: detects broken connections even when you simply "pull the plug".

This is obviously only working for WebSocket transports.

We'll expose that in Crossbar.io soon

http://crossbar.io/docs/WebSocket-Options/
https://github.com/crossbario/crossbar/issues/105

==

I need to contact the Twisted/asyncio guys, since I am concerned about
scalability of having massive numbers of timers.

Until then, it's on above branch ..

Cheers,
/Tobias


>
> Cheers,
> Bas
>

[42o8o1oo]

for me the callback of "wamp.metaevent.session.on_leave" gets called in notime if i unplug a module. Perfect!

The only thing I really miss is some information about the "dead-buddy". In the callback i get an empty object and the regular details (where "publisher" is "undefined" by default).

Looking to session.py [339]:

msg = message.Publish(0, u'wamp.metaevent.session.on_leave', [self._session_details])
self._router.process(self, msg)

It seems to me  the "session_details".are getting lost somewhere on the way...

[Matt Bonneau]

Hi Tobias,

I would still like to see PING and PONG added to the basic profile:

Ping is sent from Client to Router or Router to Client to test the connection end-to-end and to verify that the other end is handling messages

All parameters after request id can be omitted

[PING, Request|id, Options|dict, Echo|list, Discard|string]

PONG must be sent upon receive a ping request. Echo is sent back to the peer that requested.

[PONG, PING.Request|id, Details|dict, Echo|list]

A possible usage scenario:

Suppose you have an important RPC call that you need to have high availability. If the callee's session becomes unresponsive for any reason, it will not able to reconnect and reregister until the first (and now hung) client session is removed.

I would like to build an option into the register handler to request that the server "PING" the client that currently is holding the registration. If that times out, then proceed with destroying the non-responsive session and go ahead and register the latter registration request. If the server receives a timely PONG message, it can reject the latter registration attempt.

This would require the full stack to be responsive, not just the transport (as with TCP keepalive or Websocket ping/pong). This would also allow transports that don't have keepalive support to have the ping functionality.

Regards,

Matt

[Tobias Oberstein]

Hi Matt,

Am 27.08.2014 00:00, schrieb Matt Bonneau:
> Hi Tobias,
>
> I would still like to see PING and PONG added to the basic profile:
>
> Ping is sent from Client to Router or Router to Client to test the
> connection end-to-end and to verify that the other end is handling messages

What do you mean with "end-to-end"?

The transport between a WAMP client and a WAMP router is "point-to-point".

In case of a WAMP caller calling a WAMP callee, there are 2 legs:

C1 <-- leg 1 --> R <-- leg 2 --> C2

Both leg1 and leg2 can be kept alive / fast connection loss detection
using WebSocket pings/pongs when using WebSocket transport.

This only works on WebSocket (since it has it's own transport-level
facilities for that).

It does not work on RawSocket. Is that what you are concerned about?

There is no mechanism for C1 sending a WAMP level Ping thing that
crosses the router and is replied by C2.

That would introduce a coupling between C1 and C2. C1 isn't (and
shouldn't) be aware that a call it issues is handled by C2.

>
>
> All parameters after request id can be omitted
>
>
> [PING, Request|id, Options|dict, Echo|list, Discard|string]
>
> PONG must be sent upon receive a ping request. Echo is sent back to the peer that requested.
>
>
> [PONG, PING.Request|id, Details|dict, Echo|list]
>
>
> A possible usage scenario:
>
> Suppose you have an important RPC call that you need to have high
> availability. If the callee's session becomes unresponsive for any
> reason, it will not able to reconnect and reregister until the first
> (and now hung) client session is removed.
>
> I would like to build an option into the register handler to request
> that the server "PING" the client that currently is holding the
> registration. If that times out, then proceed with destroying the
> non-responsive session and go ahead and register the latter registration
> request. If the server receives a timely PONG message, it can reject the
> latter registration attempt.
>
> This would require the full stack to be responsive, not just the
> transport (as with TCP keepalive or Websocket ping/pong). This would
> also allow transports that don't have keepalive support to have the ping
> functionality.

I have re-read what you wrote a couple of times. I don't get what you mean.

Let's make it more concrete.

Say Callee1 has registered "com.myapp.add2".

Now Callee1 transport becomes unresponsive. This can be detected on
WebSocket using WebSocket pings/pongs (let's continue the example with
that transport only for now).

When the Router recognizes that the transport for Callee1 is broken,
it'll kick the WAMP session, and unregister "com.myapp.add2" automatically.

At that point, another session can register "com.myapp.add2".

Above introduces a "time gap" in which no one (reachable) has
"com.myapp.add2" registered.

Are you concerned about this gap?

If so, my thinking would probably start in more in this direction:

Automatic callee failover to a "hot-standby" callee.

Concrete:

Callee1 has registered "com.myapp.add2".

Now Callee2 (a different WAMP session) registered _also_
"com.myapp.add2" providing the option "hot_standby = true" or something.

The router allows this, since it is a hot standby callee. Normally it
would reject registering a 2nd callee for the _same_ URI.

If the router now processes a call to "com.myapp.add2" that fails
because the transport got unresponsive, or the call doesn't return at
all within a given timeout, it'll failover "com.myapp.add2" to Callee2.

This could happen completely transparently with regard to the caller.

This is of course a highly advanced feature: definitely WAMP AP.

Not sure -- let me know if I misunderstood you / I now got wild;)

/Tobias

>
> Regards,
> Matt
>
> On Sunday, August 10, 2014 8:06:02 AM UTC-4, Tobias Oberstein wrote:
>
> Am 10.08.2014 06:54, schrieb Bas Wegh:
> > Would be great in combination with The Ping and pong message type.
> > So the router can check the connections and kill the session on
> timeout.
> > And then send the meta event.
>
> Yes.
>
> FWIW, I now implemented automatic ping/pongs in AutobahnPython (both
> client and server) - WebSocket based for now:
>
> https://github.com/tavendo/AutobahnPython/blob/autopingpong/examples/twisted/websocket/pingpong_keepalive/server.py
> <https://github.com/tavendo/AutobahnPython/blob/autopingpong/examples/twisted/websocket/pingpong_keepalive/server.py>
>
>
> That works: detects broken connections even when you simply "pull
> the plug".
>
> This is obviously only working for WebSocket transports.
>
> We'll expose that in Crossbar.io soon
>
> http://crossbar.io/docs/WebSocket-Options/

> <http://crossbar.io/docs/WebSocket-Options/>
> https://github.com/crossbario/crossbar/issues/105

> <https://github.com/crossbario/crossbar/issues/105>
>
> ==
>
> I need to contact the Twisted/asyncio guys, since I am concerned about
> scalability of having massive numbers of timers.
>
> Until then, it's on above branch ..
>
> Cheers,
> /Tobias
>
>
> >
> > Cheers,
> > Bas
> >
>

> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com

> <mailto:wampws+un...@googlegroups.com>.

> To post to this group, send email to wam...@googlegroups.com

> <mailto:wam...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/wampws.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wampws/daf1d36d-d3a7-408f-a093-feaeafc05f63%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/daf1d36d-d3a7-408f-a093-feaeafc05f63%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Tobias Oberstein]

Am 26.08.2014 14:31, schrieb 42o8o1oo:
> for me the callback of "wamp.metaevent.session.on_leave" gets called in

> notime if i unplug a module. *Perfect!*

I'd say hooray, but I'm not so sure=( The problematic cases are these:

- WAMP component is connected to router over a WAN connection - which
breaks or just becomes unresponsive

- WAMP component is connected to router over LAN connection - and you
hard kill the component (not just CTRL-C it .. which will likely close
the TCP orderly) or plug the cable

Fast detection of transport (and hence WAMP session) loss in above cases
is a concern ..

> The only thing I really miss is some information about the "dead-buddy".
> In the callback i get an empty object and the regular details (where
> "publisher" is "undefined" by default).
>
> Looking to session.py [339]:
>
> msg = message.Publish(0, u'wamp.metaevent.session.on_leave',
> [self._session_details])
> self._router.process(self, msg)
>
>
> It seems to me the "session_details".are getting lost somewhere on the
> way...

Oh. That might be the case: self._session_details being deleted before
the event gets sent.

If so, this is a bug. Could you pls file it on CB repo?

>
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com

> <mailto:wampws+un...@googlegroups.com>.

> To post to this group, send email to wam...@googlegroups.com

> <mailto:wam...@googlegroups.com>.

> Visit this group at http://groups.google.com/group/wampws.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wampws/7e9f08c9-44bd-4dcc-a2c3-bbc4b112293f%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/7e9f08c9-44bd-4dcc-a2c3-bbc4b112293f%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Matt Bonneau]

Hi Tobias,

I did mean point-to-point. The messages are just between the router and the client.

The ping and pong WAMP messages would serve almost the same function as the websocket ping and pong, except it brings it up to the next protocol level and verifies that the client (or router) is still processing and responding to WAMP messages.

This would give us a universal method that would work regardless of transport (PING/PONG message types).

Regards,

Matt

[Tobias Oberstein]

Hi Matt,

> I did mean point-to-point. The messages are just between the router and
> the client.

Ah. Ok.

>
> The ping and pong WAMP messages would serve almost the same function as
> the websocket ping and pong, except it brings it up to the next protocol
> level and verifies that the client (or router) is still processing and
> responding to WAMP messages.
>
> This would give us a universal method that would work regardless of
> transport (PING/PONG message types).

The only transport defined in WAMP BP is WebSocket. Which does have
ping/pong already.

So we are talking about RawSocket and LongPoll transports (or possible
future ones), right?

Both of these are WAMP AP - hence anything we come up with would reside
there.

Regarding your proposal:

[PING, Request|id, Options|dict, Echo|list, Discard|string]

[PONG, PING.Request|id, Details|dict, Echo|list]

Why is Echo a list, and not a string?

Cheers,
/Tobias

> > an email to wampws+un...@googlegroups.com <javascript:>
> > <mailto:wampws+un...@googlegroups.com <javascript:>>.

> > To post to this group, send email to wam...@googlegroups.com

> <javascript:>
> > <mailto:wam...@googlegroups.com <javascript:>>.

> > Visit this group at http://groups.google.com/group/wampws

> <http://groups.google.com/group/wampws>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/wampws/daf1d36d-d3a7-408f-a093-feaeafc05f63%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/daf1d36d-d3a7-408f-a093-feaeafc05f63%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/wampws/daf1d36d-d3a7-408f-a093-feaeafc05f63%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/wampws/daf1d36d-d3a7-408f-a093-feaeafc05f63%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com
> <mailto:wampws+un...@googlegroups.com>.
> To post to this group, send email to wam...@googlegroups.com
> <mailto:wam...@googlegroups.com>.
> Visit this group at http://groups.google.com/group/wampws.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wampws/f160a776-6ee6-44ff-851a-bc122852c32c%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/f160a776-6ee6-44ff-851a-bc122852c32c%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Matt Bonneau]

Hi Tobias,

I should have realized that Websocket was the only transport for basic protocol. I think Websocket ping/pong is sufficient for basic profile.

I do think adding the PING and PONG messages to the advanced profile is a good idea.

You are right - a string would be a more appropriate type for the Echo argument.

Regards,

Matt

[Tobias Oberstein]

Hi Matt,

> I do think adding the PING and PONG messages to the advanced profile is
> a good idea.

https://github.com/tavendo/WAMP/issues/90

Cheers,
/Tobias

[Tobias Oberstein]

Hi Matt,

> I do think adding the PING and PONG messages to the advanced profile is
> a good idea.

After the discussions on #63 and #90, I now think it's wrong to add a
transport level feature like loss detection / keep alive at the WAMP level.

For RawSocket, here is a proposal to add ping/pong at transport level

https://github.com/tavendo/WAMP/issues/63#issuecomment-53856688

With above, we have transport loss detection / keep alive for WebSocket
and RawSocket. For Long-poll, it needs to be done differently anyway: by
adjusting the poll timeout. And I won't think about yet to be invented
transports for now. The former 3 already cover a lot. E.g. stuff like
function-call based transports or shared-memory queues don't need
ping/pong anyway.

I would like to push #63 and #90 to an end now so we can move on to
other features in WAMP AP - there are plenty which need more love.

What do you think? Would you be fine with RawSocket transport-level
ping/pong and closing #90?

Cheers,
/Tobias

[Matt Bonneau]

Hi Tobias,

I am good moving on. Like you said - the transports in use now have methods for what I am concerned about. We should move on to other things.

Regards,

Matt