List all topics and RPC's

[Lasse Vestergaard]

Hi all.

I'm just starting looking into crossbar.io, and it looks very promising.

What I need is the possibility to query the crossbar router for existing topics and registered RPC's. It could also be an extension, but I don't know how to do that yet.

In my concrete project I need a central unit (could be a crossbar node with a running route), and I then need to be able to connect arbitrary entities on to that central unit. An entity could be anything from a sensor to an application that ex. scrapes a web page for information. All-in-all, I just want to be able to have some arbitrary software that can hook up to a central place, announce it's presence and provide some information about it self.

All this seems to be implemented in WAMP. The only thing that is missing is the discoverability. If I have a piece of software that hooks up to the system, I can't ask for anything. I don't know which topics that I could subscribe to and I don't know which RPC's I could call.

An example could be that I have a simple javascript client (a web page), where I want to monitor all lamp posts in the system. How do I know if there exists any lamp posts in the system? I could start out by just asking completely randomly for arbitrary topics, but this would not be very smart. Instead I would like to ask the system (crossbar) for a list of all connected entities or topics. The same goes for RPC's. What could be really cool is if I could ask for a list of all registered entities and then ask the single entity for topics that it is publishing and what RPC's it exposes. 

Regards

Lasse Vestergaard

[Tobias Oberstein]

Hi Lasse,

Am 01.07.2014 17:48, schrieb Lasse Vestergaard:
> Hi all.
>
> I'm just starting looking into crossbar.io, and it looks very promising.

You are welcome!

>
> What I need is the possibility to query the crossbar router for existing
> topics and registered RPC's. It could also be an extension, but I don't
> know how to do that yet.

Once we have this kind of features specified in the WAMP Advanced
Profile, e.g.

https://github.com/tavendo/WAMP/blob/master/spec/advanced.md#subscriber-list

we will implement it in Crossbar.io.

It will be implemented as regular WAMP procedures and events, but built
into the router.

Note: Some bits are already there ... eg the metaevents for sessions.

>
> In my concrete project I need a central unit (could be a crossbar node
> with a running route), and I then need to be able to connect arbitrary
> entities on to that central unit. An entity could be anything from a
> sensor to an application that ex. scrapes a web page for information.
> All-in-all, I just want to be able to have some arbitrary software that
> can hook up to a central place, announce it's presence and provide some
> information about it self.
>
> All this seems to be implemented in WAMP. The only thing that is missing
> is the discoverability. If I have a piece of software that hooks up to
> the system, I can't ask for anything. I don't know which topics that I
> could subscribe to and I don't know which RPC's I could call.

The procedures registered / available for calling from a client: yes.

However, topics are not "registered" in advance. A publisher simply
publishes to a topic, and if it is authorized to do so, it'll succeed.

Subscribers subscribe to topics. So here we can provide a list of
subscribers. But publisher don't pre-register a topic they want to
publish to.

The router does not know in advance to which topics publishers will publish.

>
> An example could be that I have a simple javascript client (a web page),
> where I want to monitor all lamp posts in the system. How do I know if
> there exists any lamp posts in the system? I could start out by just
> asking completely randomly for arbitrary topics, but this would not be
> very smart. Instead I would like to ask the system (crossbar) for a list
> of all connected entities or topics. The same goes for RPC's. What could

Again, for RPC no problem. For PubSub, topics aren't "registered".

Probably the kind of "reflection" you think about is more like

https://github.com/tavendo/WAMP/blob/master/spec/advanced.md#reflection

Caution: Above section isn't properly thought through in any way. I can
see the usefulness, but it needs more design first.

If you want to join that discussion, please comment on

https://github.com/tavendo/WAMP/issues/61

Note: I remember this topic being discussed already .. probably on this
list. I lost oversight. Hence, pls comment on the issue ..

> be really cool is if I could ask for a list of all registered entities
> and then ask the single entity for topics that it is publishing and what
> RPC's it exposes.

Yes. I can see the usefulness. Even more so: would be great to be able
to access some kind of documentation for procedures/topics also.

The latter points to another direction: while the router does not have a
list of "registered topics" in general, we might allow application
components to call special procedures on the router to "self-document" /
reflect themselfes, e.g.

RPC used by an app to define metadata for procs/topics at the router:

wamp.reflection.define({"uri": "com.example.mytopic1", "type": "topic",
"help": "A awesome topic"});

RPC used by an app to retrieve metadata for a specific item:

wamp.reflection.describe("com.example.mytopic1") => {"type": "topic",
"help": "A awesome topic"})

RPC used to list items:

wamp.reflection.list() => [{"uri": "com.example.mytopic1", "type":
"topic", "help": "A awesome topic"}, ...]


Not sure if I lost you .. does above make sense to you?

/Tobias

>
> Regards
>
> Lasse Vestergaard
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Lasse Vestergaard]

Hi Tobias.

It seems like reflection does what I want to do. My immediate question is that I'm a little uncertain whether reflection is already implemented in crossbar or if it's something that will come in the future? If it's already implemented do you then have some documentation on usage?

Regarding registration of topics. I like the idea that a topic is not predefined. That way you remove one step when initiating publishing - you don't have to both (1) register the topic and (2) then publish to the topic. I would say that it has great potential if a topic would still be registered (for reasons already discussed) though. What I'm thinking is that crossbar could automatically register a topic when someone is publishing to it for the first time. This might call for a specific RPC for deleting unused/dead topics.

If I understand you correctly, then I really like your RPC suggestions in the last part of your response. Completely simple ways to define and retrieve a description and get a list of all items.

Regards

Lasse

> <mailto:autobahnws+unsub...@googlegroups.com>.

[Tobias Oberstein]

Hi Lasse,

Am 02.07.2014 10:00, schrieb Lasse Vestergaard:
> Hi Tobias.
>
> It seems like reflection does what I want to do. My immediate question
> is that I'm a little uncertain whether reflection is already implemented
> in crossbar or if it's something that will come in the future? If it's
> already implemented do you then have some documentation on usage?

No, it's not implemented. And before we can implement it, we need a
coherent design for the feature. Means: the spec needs more text.

> Regarding registration of topics. I like the idea that a topic is not
> predefined. That way you remove one step when initiating publishing -

Yes. It's also about symmetry: while Callees register (and hence a list
of registered procedures is known to the router), Callers don't apply
for calling before they actually issue a call. Of course the router can
track which procedures were actually called in the past, but it has no
clue which procedures Callers _might_ call in the future.

And this translates to Subscribers (similar to Callees rgd above) and
Publishers (similar to Callers rgd above).

> you don't have to both (1) register the topic and (2) then publish to
> the topic.

Yes.

> I would say that it has great potential if a topic would
> still be registered (for reasons already discussed) though. What I'm

"register" in the reflection sense: more like "declare" or "define"

> thinking is that crossbar could automatically register a topic when
> someone is publishing to it for the first time. This might call for a
> specific RPC for deleting unused/dead topics.

This is possible of course: have the router track topics and procedures
_actually_ published to / called _in the past_.

And then we could have meta API in the router exposing above tracked
info. and probably reset also.

I'd call that yet another approach, since it is neither related to
actual WAMP register/subscribe level nor reflection (where the app must
actively provide info).

Here: https://github.com/tavendo/WAMP/issues/76

>
> If I understand you correctly, then I really like your RPC suggestions
> in the last part of your response. Completely simple ways to define and
> retrieve a description and get a list of all items.

I agree, the approach of having 3 reflection procedures for: declare,
describe and list

https://github.com/tavendo/WAMP/issues/61#issuecomment-47763425

might be a good start.

Now, if you want to help, you could try to map your requirements to
above. Means: check if above 3 procedures would be sufficient for what
you want, and in particular _what_ kind of information those procedures
would carry.

Since: at this point in design we only have: "ok, lets make 3 router
procs for reflection".

Not enough. What signatures? Example usage for a concrete scenario?

Thats "thought work", but important. Just comment on the issue.

Once we have the feature spec'ed in WAMP, the implementation in
Crossbar.io won't be that much.

I know: you just discovered Crossbar, and now are in the middle of an
involved discussion - welcome to the bleeding edge;)

And I won't try to hide this: Crossbar is still young. We haven't got
the full picture yet, but a robust/correct basis.

/Tobias

>
> Regards
>
> Lasse
>
> Den tirsdag den 1. juli 2014 19.27.50 UTC+2 skrev Tobias Oberstein:
>
> Hi Lasse,
>
> Am 01.07.2014 17:48, schrieb Lasse Vestergaard:
> > Hi all.
> >

> > I'm just starting looking into crossbar.io <http://crossbar.io>,

> > an email to autobahnws+...@googlegroups.com <javascript:>
> > <mailto:autobahnws+...@googlegroups.com <javascript:>>.

> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+...@googlegroups.com>.

[Alexander Gödde]

Hi!

in order to better understand the necessity for publication topic lists in WAMP/Crossbar.io: 

- What precisely is it about your usage scenario that prevents your subscribers from knowing what topics to expect?

- Would prefix subscription (part of the WAMP Advanced Profile https://github.com/tavendo/WAMP/blob/master/spec/advanced.md#prefix-matching) solve your needs? More precisely: In your scenario, would it be possible and enough to subscribe with a prefix that is general enough to receive all publishes, and then process them as they first occur for each topic, or do you need advance knowledge of precise topics that might be published to?

Note: prefix subscriptions are a feature of WAMP which is sufficiently specified (unlike reflection), but it is not yet implemented in Crossbar.io.

Regards,

Alexander Gödde

[Lasse Vestergaard]

Hi guys.

Thanks for your engagement!

I'm having a hard time figuring out whether WAMP can actually solve my imagined paradigm or if my mental work is just plain of :-)

My project is about creating an application where people can vote whether they want a lamppost to be on or off. The idea is that I want to use web sockets, because I can then work with real-time orientation. Concretely, a home owner should be able to start a browser, or a smartphone app, and then see the status of lampposts around his house (ex. luminance or each lamppost). Additionally he should be able to suggest changes for a lamppost (ex. turning it of in 10 minutes). Neighbours, that would be affected by the changes should be able to vote the suggestion up or down, and at some point the suggestion will take effect or not happen (depending on the voting result).

That being my scenario, I want everything to be loosely coupled and dynamic. It should ex. be possible to add new lampposts real-time. This further means that the system should be self adapting - when a new lamppost is deployed it should automatically be connected to the system, and every other clients should be able to see the new lamppost and it's functionality.

If I, as a user, go into my web browser and want to see all lampposts, near me, I need to be able to distinguish between them, and I need to be aware of each of them. This means that each lamppost should broadcast their existence and I should be able to subscribe to each of the lampposts separately. This is where I see the issue with Crossbar.io (and maybe WAMP) - if I as a programmer don't already know exactly what is going on in the entire system I don't have a chance to figure it out.

In the lamppost application I, as a programmer, know how a lamppost is described, and what it can do, but I don't know which lampposts are already connected. Therefore I need to ask randomly to test for existence of specific lampposts. Furthermore, two lampposts are more or less copies of each other and therefore they would be publishing to the same topic. How would I be able to figure out which exact lamppost published latest, and wouldn't I run into concurrency issues as well?

In the lamppost application I, as the programmer, am completely aware of the functionality but do not necessary know how many devices/entities that are actually connected. This means that the lamppost application is rather comprehensible - I only have one unknown factor.

If we look at this system a little more generic, then I could in essence actually have all kinds of applications running in this system (which is also what WAMP describe, as far as I understand). What it all comes down to is that Crossbar.io is just a hub where applications, sensors, actuators etc. could hook up to, and then I could create my own application that exploits the already connected entities. Let's say the system is running, and a flower pod is connected (exposing humidity and the ability to water the plant), a lamppost, a home brewery, and a solar panel. I now want to create a monitoring application that connects to the Crossbar.io system, and then exploits the previous mentioned connectivities - I ex. want to see the status of the flower pod, and be able to water it if I so feel, and I would like to do the same for the other entities (lamppost, brewery, solar panel).

All this would be hard to do if I can't get an idea of what is already connected, and how to use each of the connected entities.

What all this come down to, is that I need a system much like Crossbar.io, but I would need to be able to ask for existing entities, functionality of each entity and subscribe to each entity. Additionally, I imagine that an object oriented approach would fit best. Instead of just having lists of all topics and RPC's, I would need to move these lists into each connected entity. This means that I would ask a specific entity about it's own functionality (methods/RPC's) and it's publishing topics.

To me it seems like I'm thinking of a different paradigm than the one WAMP represents. Would it make sense to pursue this paradigm in the future developments of WAMP or does this more seem like a parallel project?

Regards

Lasse Vestergaard

[Alexander Gödde]

Hi Lasse!

Just a few thoughts and comments for now - we'll definitely be talking about this at the office some more.

• I see the fundamental issue of discoverability at the protocol level. WAMP in its present state does not help there.
• This issue should be addressed, and it is likely that the we'll add more reflection features to the advanced profile. Specifically, a client should be able to announce that it is a publisher to a topic independently of publishing to that topic.
• Your scenario obviously rules out using prefix subscriptions, since you want users to have an overview of existing nodes when connecting their votes app.
• Nothing at the protocol level would solve your second problem after discoverability - displaying the present state. The WAMP router should never contain any application logic, and storing state for existing nodes would be that.
• This means that the state has to be either stored somewhere, or you need to query all nodes for their present state on each new votes app connecting, 

So, practically speaking:

• In the first case, you would need some kind of coordinating instance which is permanently connected and subscribed to the state update channel. This could then be queried by newly connected clients for both the list of lamp posts and their present state.
• In the second case, each new app connection leads to network traffic anyway, i.e. to querying each lamp post for its state. Then why not send an initial "new_client_connected" from the vote app, to which all lamp posts are subscribed. On receiving this, they announce their present state. This means the new client both knows who's connected, and what their state is.

Regarding your more general considerations:

[Alexander Gödde]

Sorry, pressed 'post' accidentally, so this is 2/2:

Regarding your more general considerations:

• WAMP handles RPC and PubSub. RPC endpoints should be discoverable, as should, probably, be publishers.
  
• The higher-level concept of entities which offer the RPC endpoints, call procedures, subscribe and publish to topics are not something the protocol can handle. As you say yourself, this is something that needs to be done on the application level.
• Having said that, I certainly think that this can be implemented using WAMP and the messaging patterns it provides.

Anyway, the above thoughts are just my two cents worth at the moment. I'm hope there's something in there that you find useful.

Regards, 

Alex

[Lasse Vestergaard]

Hi Alex.

Thanks for your pointers!

I have now decided to start out from scratch (starting from the web sockets level), but will keep you guys in the loop, if you are interested.

My initial thoughts are to implement a simplistic pub/sub pattern, that essentially isn't anything else than an array on the web socket server, that contains all entities registered. Each entity will be described with id, type, list of methods and a list of observers. A method will consist of a name, list of parameters (each with name and type) and a return type. When ever a client is publishing, the server will just iterate through all observers registered on the specific entity, and send the notification. This means that I don't work with topics, but objects - when ever an object/entity says something (anything really) all observers gets notified.

Regarding RPC, a client can iterate through all connected entities, get all methods for each entity and call a specific method on a specific entity with the relevant parameters. Essentially, the server is just an index of all connected entities and their functionalities.

Regards

Lasse Vestergaard

[Lasse Vestergaard]

Hi guys.

I have now done the first iteration of a WAMP inspired platform. It's extremely light weight (more or less excluding security and performance). I have chosen to call the platform arip (awesome real-time IoT platform). Here is the code: https://github.com/lassesvestergaard/arip

The aim is to develop it for interaction designers for rapid prototyping. I don't know if you can use it for anything, but hopefully it can provide you with some kind of inspiration (either in the AHAAA or OHHH NOO sense :-) ).

Regards

Lasse Vestergaard

[Tobias Oberstein]

Hi Lasse,

Am 18.08.2014 09:28, schrieb Lasse Vestergaard:
> Hi guys.
>

> I have now done the first iteration of a WAMP inspired platform. It's
> extremely light weight (more or less excluding security and
> performance). I have chosen to call the platform arip (awesome real-time
> IoT platform). Here is the code: https://github.com/lassesvestergaard/arip

Mmh. Why reinvent the wheel (WAMP)?

Also: I have a hard time believing that 211 lines of code

https://github.com/lassesvestergaard/arip/blob/master/genericplatform.py

would provide anything near the functionality of WAMP / Autobahn not to
speak of Crossbar.io;)

> The aim is to develop it for interaction designers for rapid
> prototyping. I don't know if you can use it for anything, but hopefully
> it can provide you with some kind of inspiration (either in the AHAAA or
> OHHH NOO sense :-) ).

Well;)

I have only one question: what was your motivation or requirement that
made you skip WAMP?

Cheers,
/Tobias

> * WAMP handles RPC and PubSub. RPC endpoints should be

> discoverable, as should, probably, be publishers.

> * The higher-level concept of entities which offer the RPC

> endpoints, call procedures, subscribe and publish to topics
> are not something the protocol can handle. As you say
> yourself, this is something that needs to be done on the
> application level.

> * Having said that, I certainly think that this can be

> <https://github.com/tavendo/WAMP/blob/master/spec/advanced.md#prefix-matching>)

> solve your needs? More precisely: In your scenario,
> would it be possible and enough to subscribe with a
> prefix that is general enough to receive all publishes,
> and then process them as they first occur for each
> topic, or do you need advance knowledge of precise
> topics that might be published to?
>
> Note: prefix subscriptions are a feature of WAMP which
> is sufficiently specified (unlike reflection), but it is
> not yet implemented in Crossbar.io.
>
> Regards,
>
> Alexander Gödde
>
>
> Am Dienstag, 1. Juli 2014 17:48:58 UTC+2 schrieb Lasse
> Vestergaard:
>
> Hi all.
>
> I'm just starting looking into crossbar.io

> <http://crossbar.io>, and it looks very promising.

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.

> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/806f9794-8b9a-404f-95a1-61b95d46d774%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/806f9794-8b9a-404f-95a1-61b95d46d774%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Lasse Vestergaard]

Hi Tobias.

Thanks for your reply.

First of all, I'm completely aware that this simple code do not hold more than a fraction of what WAMP does.

The reason why I wanted to "reinvent" the wheel is because it doesn't seem like WAMP is considering discoverability (as we discussed earlier in this thread). The reason why I wanted to provide you with this simple code is just to show how one implementation of discoverability could be thought of.

As an example, you can ask the server for all existing entities connected to the platform. Additionally, you can ask a specific entity for exposed RPC's (this also works as a simplified documentation). And since you can now figure out who is actually connected to the platform, you can find out who you want to subscribe to.

All in all; the point is just to add a prototype for further discussions on how/if discoverability should be integrated into WAMP. The only extra detail about arip, is that is actually works for simple setups (Interaction designers love these simple tools - I know because I'm one of them :-) ).

I hope this makes the purpose a little clearer. It's not an attempt to compete with WAMP, but to be part of the discussion.

Regards

Lasse Vestergaard

> <mailto:autobahnws+unsub...@googlegroups.com>.

[Tobias Oberstein]

Hi Lasse,

> The reason why I wanted to "reinvent" the wheel is because it doesn't
> seem like WAMP is considering discoverability (as we discussed earlier
> in this thread). The reason why I wanted to provide you with this simple
> code is just to show how one implementation of discoverability could be
> thought of.

Ahh. I like that! Since it shows you are serious about this and provides
code to play/discuss.

Alright. Let's make that more concrete and "persistent" (the mailing
list isn't optimal for systematic development of new features .. at
least I forget easily):

https://github.com/tavendo/WAMP/issues/83

>
> As an example, you can ask the server for all existing entities
> connected to the platform. Additionally, you can ask a specific entity

What would be an entity in the context of WAMP?

> for exposed RPC's (this also works as a simplified documentation). And

In fact, I started on this in Crossbar.io
(https://github.com/tavendo/WAMP/issues/61)

It's totally missing in the spec, but works like this: a client that
exposes a procedure can _additionally_ can attach a JSON schema to the
procedure, and Crossbar is then able to

a) validate that the call arguments match the schema
b) generate online docs for procedures

Is that something of interest to you?

> since you can now figure out who is actually connected to the platform,
> you can find out who you want to subscribe to.

The main "issue" / difference with topics is: a procedure only exists if
someone registered it. A topic isn't pre-registered in any way. The
router knows when there is at least 1 subscriber, but it doesn't know if
someone would subscribe in the future.

Now:

- a topic that was described by the reflection API above can be queried
- a router might be configured that it only allows publish/subscribe for
topics which have been previously described via reflection API

With both of above, you could then query for an "authorative" list of
topics. Is that going in a direction you would like to see?

>
> All in all; the point is just to add a prototype for further discussions
> on how/if discoverability should be integrated into WAMP. The only extra

I now understand. That's perfect - I just couldn't decipher that from
your mail;)

> detail about arip, is that is actually works for simple setups
> (Interaction designers love these simple tools - I know because I'm one
> of them :-) ).
>
> I hope this makes the purpose a little clearer. It's not an attempt to
> compete with WAMP, but to be part of the discussion.

You are welcome! Let's discuss .. ideally on the issue #83 and/or #61.

Cheers,
/Tobias

>
> Regards
>
> Lasse Vestergaard
>

> > an email to autobahnws+...@googlegroups.com <javascript:>
> > <mailto:autobahnws+...@googlegroups.com <javascript:>>.

> > To post to this group, send email to autob...@googlegroups.com

> <javascript:>
> > <mailto:autob...@googlegroups.com <javascript:>>.

> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/autobahnws/806f9794-8b9a-404f-95a1-61b95d46d774%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/806f9794-8b9a-404f-95a1-61b95d46d774%40googlegroups.com>
>
> >

> <https://groups.google.com/d/msgid/autobahnws/806f9794-8b9a-404f-95a1-61b95d46d774%40googlegroups.com?utm_medium=email&utm_source=footer

> <https://groups.google.com/d/msgid/autobahnws/806f9794-8b9a-404f-95a1-61b95d46d774%40googlegroups.com?utm_medium=email&utm_source=footer>>.
>
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+...@googlegroups.com>.

> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/b61913f7-603d-4460-b1a8-0c23253dbdd3%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/b61913f7-603d-4460-b1a8-0c23253dbdd3%40googlegroups.com?utm_medium=email&utm_source=footer>.