Publishing disconnects (onLeave) from RouterSession to joined front- and backends.

[Daniel Faust]

Hi there,

I've got multiple SBC's (olimex, raspberry) which control house lighting. Each SBC is a backend and publishes it's existence as soon as they join the router. Frontends can also publish a request for existence as soon as they join the router, to which the backends respond by publishing their hostname and id to that frontend.

My problem is that I'm unable to notify the frontends when a backend goes down. In the router I've subclassed the RouterSession class in order to track connected front- and backends, and came to the conclusion that it would be best to publish a general on_leave message containing the id of the leaving session the to everyone who is connected. I don't want frontends to be polling for backends existance.

In short, I'm unable to call a publish method from within a RouterSession. As RouterSessions seem to be siblings to ApplicationSessions, which do contain the publish methods on them, I fear that a RouterSession is unable to publish at all.

If that is true, is it possible to create RouterSessions and ApplicationSessions with the same reactor, so that the RouterSession's onLeave could call a method on a local ApplicationSession which would then call the publish method?

Thanks in advance,
Daniel

[Greg Fausak]

Hi Daniel,

It sounds like an interesting application!

I messed with a copy of basicrouter a couple days ago and posted my changes for an issue I was trying to figure out.  In that experiment, I added a new class to basicrouter and recorded in an in memory class all client join and leave events.  I then registered some rpc calls from the router itself so list the active sessions, and to kill a session.  Anyway, I just modified that a bit to add a 'publish' message for the join/leave events.  I did a little experiment here where I publish 'com.client.add' with the session id when a new client joins, and 'com.client.delete' when a session leaves.  I subscribed to that, then I had sessions come and go, all the while the subscription announced the event.  I thought this might be what you are looking for.   You can publish from a routersession, but you have to do it from a self registered component.  That is, you create a component subclass of applicationsession, then connect it to the router basic router.  I can forward you my basicrouter.py code as well as the test client.py subscription if you want to take a look.  Using this technique, your frontend subscripts to the add/delete publications, and when the backends come and go the frontend will receive a message to that effect.

-g

[Tobias Oberstein]

Am 10.09.2014 23:35, schrieb Daniel Faust:
> Hi there,
>
> I've got multiple SBC's (olimex, raspberry) which control house
> lighting. Each SBC is a backend and publishes it's existence as soon as
> they join the router. Frontends can also publish a request for existence
> as soon as they join the router, to which the backends respond by
> publishing their hostname and id to that frontend.
>
> My problem is that I'm unable to notify the frontends when a backend
> goes down. In the router I've subclassed the RouterSession class in
> order to track connected front- and backends, and came to the conclusion
> that it would be best to publish a general on_leave message containing
> the id of the leaving session the to everyone who is connected. I don't
> want frontends to be polling for backends existance.

WAMP AP defines (or will properly define) so-called meta-events for
stuff like this. E.g. Crossbar has imlements this:

http://crossbar.io/docs/Session-Metaevents/

>
> In short, I'm unable to call a publish method from within a
> RouterSession. As RouterSessions seem to be siblings to
> ApplicationSessions, which do contain the publish methods on them, I
> fear that a RouterSession is unable to publish at all.

Yes. This is by design. RouterSessions don't run application specific code.

>
> If that is true, is it possible to create RouterSessions and
> ApplicationSessions with the same reactor, so that the RouterSession's

Yes. If you run a router and side-by-side components, instances of above
will exist in the same process, and run from the same event loop.

You can think of a WAMP session as 2 parts:

ApplicationSession <==== WAMP Session ====> RouterSession

ApplicationSession is the client-side representation of a WAMP session
in AutobahnPython, whereas RouterSession is the router-side
representation of a WAMP session in AutobahnPython's basic router.

> onLeave could call a method on a local ApplicationSession which would
> then call the publish method?

No.

You can dig into Crossbar to find out how metaevents/procedure are done
there:

https://github.com/crossbario/crossbar/blob/master/crossbar/crossbar/router/session.py#L58
https://github.com/crossbario/crossbar/blob/master/crossbar/crossbar/router/session.py#L343
https://github.com/crossbario/crossbar/blob/master/crossbar/crossbar/router/session.py#L380

===

As stressed in my other mails: all of above is absolutely irrelevant to
WAMP application developers.

It might only be of interested if you write a custom router using
AutobahnPython. And you should think twice before doing that, since
there is already an advanced router built on AutobahnPython: Crossbar.

/Tobias


>
> Thanks in advance,
> Daniel
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/6a714859-c23c-46ce-8580-e33750c0cb78%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/6a714859-c23c-46ce-8580-e33750c0cb78%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Tobias Oberstein]

Hi Greg,

> I messed with a copy of basicrouter a couple days ago and posted my
> changes for an issue I was trying to figure out. In that experiment, I
> added a new class to basicrouter and recorded in an in memory class all
> client join and leave events. I then registered some rpc calls from the
> router itself so list the active sessions, and to kill a session.
> Anyway, I just modified that a bit to add a 'publish' message for the
> join/leave events. I did a little experiment here where I publish
> 'com.client.add' with the session id when a new client joins, and
> 'com.client.delete' when a session leaves. I subscribed to that, then I
> had sessions come and go, all the while the subscription announced the
> event. I thought this might be what you are looking for. You can

No. Please do not recommend this approach to others (see below).

> publish from a routersession, but you have to do it from a self
> registered component. That is, you create a component subclass of
> applicationsession, then connect it to the router basic router. I can
> forward you my basicrouter.py code as well as the test client.py
> subscription if you want to take a look. Using this technique, your
> frontend subscripts to the add/delete publications, and when the
> backends come and go the frontend will receive a message to that effect.

What you are doing is creating a custom router with very specific,
application dependent code. In short: don't do that.

This breaks WAMP paradigms and is unsupported by AutobahnPython (your
code may break any time, with each minor release).

I feel I need to write all of this up much more clearly, but in short,
WAMP applications developers should _not_ write custom routers/drivers,
but only use these classes:

https://github.com/tavendo/AutobahnPython/wiki/Programming-Guide#writing-wamp-applications

* ApplicationSession
* ApplicationRunner
* Application

I will remove the code examples that don't follow above:

https://github.com/tavendo/AutobahnPython/issues/268

Cheers,
/Tobias

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/ee51e839-2359-485f-81c9-9f7740b93daf%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/ee51e839-2359-485f-81c9-9f7740b93daf%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Greg Fausak]

Tobias,

The description that Daniel proposed seems to be basic functionality to me.  I took the basicrouter.py code (in the example directory) and added a class to it.  That class built upon a concept already presented in the UserDb class.  The *only* deviant thing that was done was I dipped into the _session_id because there was no first class way to get the _session_id.  Other than that, there isn't any hacking going on here.

Isn't basicrouter.py an 'example' of a router that I can write.  I'm confused.

-g

> <mailto:autobahnws+unsub...@googlegroups.com>.

[Tobias Oberstein]

Hi Greg,

Am 11.09.2014 16:00, schrieb Greg Fausak:
> Tobias,
>
> The description that Daniel proposed seems to be basic functionality to
> me. I took the basicrouter.py code (in the example directory) and added
> a class to it. That class built upon a concept already presented in the
> UserDb class. The *only* deviant thing that was done was I dipped into
> the _session_id because there was no first class way to get the
> _session_id. Other than that, there isn't any hacking going on here.
>
> Isn't basicrouter.py an 'example' of a router that I can write. I'm
> confused.

The thing is: why do you feel the need to write a custom router anyway?
There are certainly valid cases, but could you expand on the former?

With WAMP2 / AutobahnPython, writing custom routers/drivers is something
expected to be unneeded and discouraged in most cases.

Developers writing WAMP applications certainly don't need to write
custom routers.

As said, I'll remove the confusing examples that apparantly lead people
to think they should start writing apps by writing a router.

> > an email to autobahnws+...@googlegroups.com <javascript:>
> > <mailto:autobahnws+...@googlegroups.com <javascript:>>.

> > To post to this group, send email to autob...@googlegroups.com

> <javascript:>
> > <mailto:autob...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/autobahnws/ee51e839-2359-485f-81c9-9f7740b93daf%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/ee51e839-2359-485f-81c9-9f7740b93daf%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/autobahnws/ee51e839-2359-485f-81c9-9f7740b93daf%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/autobahnws/ee51e839-2359-485f-81c9-9f7740b93daf%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+...@googlegroups.com>.

> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/16f091a5-d3eb-4e0e-b2ec-aabc3b697268%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/16f091a5-d3eb-4e0e-b2ec-aabc3b697268%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Daniel Faust]

Hi Greg and Tobias,

I've spent coding on this issue.  I've done the following things:
Subclassed a WampWebSocketServerFactory to hold shared variables, and shared methods. The variables are a mapping of id's to backed hostnames (and backend hostnames to id's for performance), the methods are add_backend(self, hostname, id) and remove_endpoint(self, id).
In addition I added subclassed ApplicationSession to the reactor, which connects to the router in the same memory space. The ApplicationSession listens to announcements from backends, which also contain the hostnames, and calls add_backend().
I also subclassed the RouterSession, which calls the remove_endpoint() during an onLeave event.
The remove_endpoint() method publishes an event telling everyone that the backend(id/hostname ) has left the router. If the id didn't belong to a backend (wasn't in the map), it ignores the disconnect.

It works really nice, since everybody now gets the announcements as well as the disconnects. I know that it's not nice to modify the router, as I've built one based on WAMP v1 and now I'm clueless about how to upgrade the system to v2, since there are so many customizations. It's a heavy, reliable system, also containing Android clients.

But I really think that backend disconnects, either dropped ones or clean shutdowns, should be notified to frontends by the router, and possibly also frontend disconnects to backends. Or possibly general all disconnects should be published on a public channel to everyone, containing only the id, as this would not be that much of a security issue. But this disconnection stuff should get included in the basic router. It's not ok to rely on periodic RPC calls between back- and frontends just to determine presence, if the router has the capability of detecting disconnects in realtime. But it's hard to determine a way to do this. Maybe the correct way would be for an ApplicationSession to subscribe under the hood to a special presence topic. I'm not sure about this, as I don't know anything about the inner workings of WAMP.

Yesterday I spent a short time looking at crossbar.io and I got the feeling that it is similar to what I'm doing here, but I'm not sure yet. I will continue developing the stuff I'm doing, and checking out crossbar.io in parallel. Maybe I can swich over to crossbar.io.

Currently I've got other issues, like deciding how devices should talk to each other. I'm now thinking that I should avoid RPC at all costs, since there is the issue of RPC names having to be unique on the router. So I'm thinking about only pubsub with a heavy use of PublishOptions::exclude, but I need to find a way to handle responses which come back in a subscribed channel to functions which should get performed after a request-for-action got published. I feel like I'm reinventing the wheel, but need to do this because it's not possible to make a RPC call to multiple backends at once, which would be ideal because it wouldn't require the use of a unique procedure name. But then deciding how to receive the results of the calls to multiple backends raises a whole lot of questions. I'm thinking about taking a lot of logic out of the backends into the application residing in the router, and have frontends communicate direclty with the application which then communicates with the backends. What a mess.

Getting back to the disconnection stuff: If I have a web page which is connected to the router, and this web page is interested in a specific backend, ie the LED-Dimming in the living room, if the living room backend is down, the web page will just be publishing the values from the slider without any reason, getting the UI out of sync with reality. It wouldn't be possible to place a backend online / offline icon on the page without having the backend constantly send a heartbeat, and have the cliend always checking for that heartbeat and wait to see if the heartbeat is no longer present. Really, the presence info is in the router, it just has to publish it somehow. I'm using an ApplicationSession, it works, but doing it this way feels wrong. But maybe it isn't.

@Greg, when you say "doing it from a registered component", am I missing something? I have a whole ApplicationSession running on the same reactor as the router, is there a more lightweight method of doing this? The examples are all a bit shuffled together, it's hard fo find a good overview which tells one what which example is doing. If not, it would be Nice to be able to insert components directly into the router, but then again, maybe crossover.io is doing this.

Daniel

[Daniel Faust]

Looks like I totally missed the following

WAMP AP defines (or will properly define) so-called meta-events for stuff like this. E.g. Crossbar has imlements this:
http://crossbar.io/docs/Session-Metaevents/

will be checking out crossbar more intensively now.

On Wednesday, September 10, 2014 11:35:22 PM UTC+2, Daniel Faust wrote:

[Daniel Faust]

Hey Tobias, these meta-events, "wamp.metaevent.session.on_leave", "wamp.metaevent.session.on_join" do they only get sent in crossbar.io? basicrouter doesn't send them, right? (at least my componets aren't receiving then when connected to the basicrouter)

Could you give me a short description of how to add them in the cleanest possible way into the basicrouter so that I can try to implement it there with the goal to send you a pull request?

[Tobias Oberstein]

Am 12.09.2014 03:30, schrieb Daniel Faust:
>
> Hey Tobias, these meta-events, "wamp.metaevent.session.on_leave",
> "wamp.metaevent.session.on_join"do they only get sent in crossbar.io?
> basicrouter doesn't send them, right? (at least my componets aren't
> receiving then when connected to the basicrouter)

Yes. This is a feature of WAMP AP ("Advanced Profile"). The basic router
included with AutobahnPython is called "basic" because it only
implements the WAMP BP - plus those features from the AP which had been
in WAMP1. Other features of the AP won't be implemented in the basic router.

>
> Could you give me a short description of how to add them in the cleanest
> possible way into the basicrouter so that I can try to implement it
> there with the goal to send you a pull request?

Why replicate functionality that is already in Crossbar? You might just
use Crossbar ..

Cheers,
/Tobias

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/ec5e35bc-1854-4aa5-bc7a-dafe06b8e587%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/ec5e35bc-1854-4aa5-bc7a-dafe06b8e587%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Greg Fausak]

On Thursday, September 11, 2014 2:21:11 PM UTC-5, Tobias Oberstein wrote:

Hi Greg,

Am 11.09.2014 16:00, schrieb Greg Fausak:
> Tobias,
>
> The description that Daniel proposed seems to be basic functionality to
> me.  I took the basicrouter.py code (in the example directory) and added
> a class to it.  That class built upon a concept already presented in the
> UserDb class.  The *only* deviant thing that was done was I dipped into
> the _session_id because there was no first class way to get the
> _session_id.  Other than that, there isn't any hacking going on here.
>
> Isn't basicrouter.py an 'example' of a router that I can write.  I'm
> confused.

The thing is: why do you feel the need to write a custom router anyway?
There are certainly valid cases, but could you expand on the former?

I am porting from v1 to v2.  In v1 I subscribe to messages and let the router figure out who to publish to in the 'include' part of the publish message.  I determine the endpoints that are authorized for a specific message.  Ie.  if one of my devices is going to reboot I'll publish a 'REBOOT' message.  On the subscribe side, I only send that message to those subscribers that have the authorization to see it.  I could do the same thing in v2. However, it puts a load on the router rather than making it a simply relay, it becomes a database dip for each message. So, I chose to do it my making the message more specific, like:

com.device.42.REBOOT

Now, everybody that subscribes to that message will see it.  It doesn't make sense for my clients to subscribe to that, though, because there are hundreds. I'd rather subscribe to:

com.device.*.REBOOT

and have the router figure out at subscription time the actual subscriptions to subscribe to.  That's step one. Step two is as new devices come online, they also are auto-subscribed.  Step three is when the relationships change (I add or subtract devices from my login relationship) the subscription/unsubscription is automatic.

In a nutshell that is what I am currently trying to make work.  The * denotes an 'axis' of security.  There can be many of them, and the control for each axis can come from different client.  I am struggling with that part right now.  In the addressing scheme there is:

AXIS1.AXIS2.AXISn

where each AXIS can have publish/subscribe/register and call governed by independent authority.  Either that or I group topics into 'super targets' and explicitly map which topics belong to it.

-g

[Daniel Faust]

Hi Tobias, Greg,

Tobias, I've now been fiddling around with crossbar and it's going to replace my router. Up to this point it does exactly what I need. But I still think that there should be a way for frontends to get notified about disconnects from backends.

Maybe the best way would be to allow frontends (or AppSessions in general) to register to RPC method avaliability notifications. For example if a backend implements an com.myapp.hello RPC method, then a frontend can tell the router that it should get notified when the backend implementing this method joins / leaves the router. Even if crossbar could implement this, it's a basic enough feature to justify its existence in the basicrouter / WAMP BP. A backend or frontend could add a flag to a RPC method it implements, to optionally enable realtime avaliability information about that method in the system.

On a system I built on WAMP v1 I also dealt with routing. All clients would register to the router by telling the router which tags the clients subscribes to. Then, endpoints would publish all to the same topic specifying a message, category and tag, where tag would allow the router to filter out clients which didn't subscribe to them, category would ease clients to select the action to perform on the message or the extra data associated with the message. These do also get stored in the database.

Basically a database entry looks like

/* 46 */
{
  "_id" : ObjectId("541276c8cf1e8c0c6c63e818"),
  "category" : "notification/message",
  "status" : "ok",
  "uuid" : "752eda40-3a35-11e4-aabf-00012e2fc013",
  "tags" : ["dev"], // <============================= only to developer devices
  "when" : ISODate("2014-09-12T04:30:00.163Z"),
  "message" : "Licht Wohnzimmer Katrin 80->1024",
  "options" : {
    "hidden" : false,
    "store" : true
  }
}

/* 68 */
{
  "_id" : ObjectId("5412f623cf1e8c0c6c63e82e"),
  "category" : "phone/call",
  "status" : "ok",
  "uuid" : "5e6b4cc0-3a81-11e4-9247-00012e2fc013",
  "tags" : ["all"], // <============================= to every device
  "when" : ISODate("2014-09-12T13:33:23.723Z"),
  "message" : "Armin XXXXXXXX (Seba) → K'",
  "options" : {
    "direction" : "in",
    "uuid" : "2014-09-12T13:33:22.351+0000",
    "tts" : true,
    "say" : "Armin XXXXXXXX (Seba) für Katrin",
    "action" : "request",
    "store" : true
  }
}

/* 70 */
{
  "_id" : ObjectId("5412f696cf1e8c0c6c63e830"),
  "category" : "phone/call",
  "status" : "ok", // <============================= to every device
  "uuid" : "a2886f00-3a81-11e4-9d39-00012e2fc013",
  "tags" : ["all"],
  "when" : ISODate("2014-09-12T13:35:17.998Z"),
  "message" : "Armin XXXXXXXX (Seba) → K'",
  "options" : {
    "direction" : "in",
    "uuid" : "2014-09-12T13:33:22.351+0000",
    "duration_pretty" : "1m 36s",
    "extension_short" : "KDCT",
    "action" : "disconnect",
    "duration" : 96.844009,
    "store" : true
  }
}

Keep in mind that these messages belong to a messaging system, not to a control system, they are just intended to deliver text notifications to endpoints, where the notifications can be enriched with additional information. For example the SBC's for home lighting post such a message to this system, or the telephony system. They can do this easily since the router has an integrated web server so that http://router.example.com/publish?message=Licht Wohnzimmer Katrin 80->1024&category=notification/message&tags=dev&options=... results in such a message. They can be optionally stored in the database so that web pages can query the database to get a history of the notifications.

In another app I was doing I focused on the tag-subscription mechanism which is where this topic got handled http://stackoverflow.com/questions/10281863/in-python-how-can-i-query-a-list-of-words-to-match-a-certain-query-criteria so that tags could contain hierarchical structures ( level1/level2/level3, like home/lighting/livingroom ), and publishers could use levels to select the ganularity of who is a recipient (ie users/kids/* vs users/kids/seba) even use regular expressions to select the targets: (users/kids/* dev.* where dev.* is a regex while users/kids/* is a wildcard). I will check this out again after I have a better understanding of crossbar. BTW this was part of a test system which also contained reflection, where components which registered at the router would announce all the mentods they have avaliable, including their parameters, and the configuration state (variables) of the component, as well as avaliable configuration sets (ie ping every 10 seconds to tag=dev or every 30 seconds to tag=all, or a realtime stock tracking component where stocks=fb or stocks=fb,goog,amz). Frontends are able to shutdown/spawn backends and load configurations. See the attached image. It starts with a client/html, which is the page, there's a system/chat backend registered at the router, which would post messages to everyone subscribed to teh tag "chat" (taargets: chat). Then there's a service/stock backend, to which client/html registers at 17:31:11.237. Tags can be denied access to if other tags are not present, which is a very rudimentary access control. This is a lot of info, I hope I find time to reuse and share this system.

[Daniel Faust]

[Daniel Faust]

A more customized stock client would look like this

[Daniel Faust]

Had to dump crossbar.io, as I also need a twisted web server in the same reactor :(