trying to understand options for serving embedded system data...

[Dave Barndt]

Hi,

First off, let me say that after reading through the various sites' docs, WAMP, Autobahn, and Crossbar all seem terrific. I am excited about the possibility of using them!

I'm writing because I would very much appreciate a little guidance with understanding what my implementation options are given my project's requirements. I apologize in advance for my ignorance or gaps in my understanding.  There's so much to try to wrap my head around!

So here's my project in a nutshell. I have some number of identical embedded devices on a network. There will be some backend system that they all connect to (and they also may potentially connect to each other, if it makes sense).  Each embedded device will serve a local "console" website (HTML/JS/static files), but for "serving" dynamic data for the website, and in fact "serving" any data from the device in general (ex. to a browser, or to the backend system), I would like to use Websockets. Additionally, I'm hoping users will be able to create their own clients to interact with the device.  I would also like the embedded devices to be able to "register" on the network when they come online (ex. perhaps they could potentially have a config setting pointing to the (backend) system to register with).

Anyway, with that quick description, here are a few questions I have:

1) I think I want to at least put Autobahn/Python on each embedded device.  Would it be able to serve (the same data to) both WAMP-enabled clients and simple "vanilla" Websocket clients?

2) I assume my own browser client (for the local website) should be WAMP-enabled (via Autobahn/JS)?

3) Does it make any sense at all to put a router like Crossbar on each device?  If so, the same question from 1) applies.  Or does it make more sense to have the router on the backend system?

4) I'm thinking when a device 'registers" on the network, it would register the RPCs and channels it wants to offer with the router. Is it possible somehow to query to get a digest/directory of all RPCs registered with a Dealer?  Same question for all channels available to be subscribed to?

I hope I haven't written too much for one topic/question, but any answers you can give will save me hours or even days of trial and error as I continue to learn.  Thank you again for a great set of tools to help move the "IoT" forward!

Dave B.

[Tobias Oberstein]

Hi Dave,

Am 02.11.2014 05:27, schrieb Dave Barndt:
> Hi,
>
> First off, let me say that after reading through the various sites'
> docs, WAMP, Autobahn, and Crossbar all seem terrific. I am excited about
> the possibility of using them!

Great! Also good to hear that you are not confused by this stuff
spreading over some sites;) Seems this can be somewhat frustrating for
users .. but we want e.g. WAMP be a "neutral" place (not tied to
Autobahn or Crossbar.io).

>
> I'm writing because I would very much appreciate a little guidance with
> understanding what my implementation options are given my project's
> requirements. I apologize in advance for my ignorance or gaps in my
> understanding. There's so much to try to wrap my head around!

Nothing to excuse for! Don't hesitate to ask any amount of Qs ..

>
> So here's my project in a nutshell. I have some number of identical
> embedded devices on a network. There will be some backend system that
> they all connect to (and they also may potentially connect to each
> other, if it makes sense). Each embedded device will serve a local
> "console" website (HTML/JS/static files), but for "serving" dynamic data

Why does each and every device serve static content? Just trying to
understand your goals ..

> for the website, and in fact "serving" any data from the device in
> general (ex. to a browser, or to the backend system), I would like to
> use Websockets. Additionally, I'm hoping users will be able to create
> their own clients to interact with the device. I would also like the

If your devices expose their functionality via WAMP, then users will be
able to talk to devices using any WAMP support language .. which is a
growing list that already contains many popular ones.

> embedded devices to be able to "register" on the network when they come
> online (ex. perhaps they could potentially have a config setting
> pointing to the (backend) system to register with).

Yep, they could have a local config with minimum:

- WAMP router URL ("wss://...")
- WAMP realm ("realm1")

plus possibly authentication information.

>
> Anyway, with that quick description, here are a few questions I have:
>
> 1) I think I want to at least put Autobahn/Python on each embedded
> device. Would it be able to serve (the same data to) both WAMP-enabled
> clients and simple "vanilla" Websocket clients?

Using AutobahnPython, you can create a server that listens on 1 port,
but serves both vanilla WebSocket as well as WAMP .. on that 1 port.
In fact, you can have even more stuff being served on that port.

E.g. server2.py here:

https://github.com/tavendo/AutobahnPython/tree/master/examples/twisted/websocket/multiproto

Crossbar.io uses this technique to serve a whole bunch of services:

http://crossbar.io/docs/Web-Transports-and-Services/

> 2) I assume my own browser client (for the local website) should be
> WAMP-enabled (via Autobahn/JS)?

Yep, if you want to speak WAMP from the browser, AutobahnJS is your
first stop.

There is a 2nd JS WAMP implementation for browsers: wampy.js

http://wamp.ws/implementations/#libraries

which you can also use. They have a comparison table with AutobahnJS.

> 3) Does it make any sense at all to put a router like Crossbar on each
> device? If so, the same question from 1) applies. Or does it make more
> sense to have the router on the backend system?

I'd start simple, and have a WAMP router on the backend system where all
devices connect to.

If using Python, you only need AutobahnPython. If you use Python 3.4+
with asyncio (incl. with Python 3), you get away with very little
installation.

Reasons to have a router on a device:

- even the locally running app parts are build as WAMP components to
talk to each other (on the device).
- the device is some kind of gateway (with both LAN and WAN ports) and
local WAMP traffic should be routed locally

Please note that scenarios like the latter, where a WAMP component
connects to a (local) router, and that router in turn connects to an
upstream router to essentially form a router network is a feature we not
yet have in Crossbar.io - needs router-to-router connections.

> 4) I'm thinking when a device 'registers" on the network, it would
> register the RPCs and channels it wants to offer with the router. Is it
> possible somehow to query to get a digest/directory of all RPCs
> registered with a Dealer? Same question for all channels available to
> be subscribed to?

Listing and reflecting upon registered procedures, as well as listing
subscribers: this is being defined in WAMP "Advanced PRofile". Not yet
finalized in the spec, and only partially implemented (and undocumented)
in Crossbar.io

Same for reflecting on "all topic available to subscribe to".

Note my different use of "listing" and "reflecting".

E.g. Crossbar.io implements the approach described at the end of

https://github.com/tavendo/WAMP/issues/61

You can "wamp.reflection.describe" on both a procedure or topic which
you have previously stored info for via "wamp.reflection.declare".

>
> I hope I haven't written too much for one topic/question, but any
> answers you can give will save me hours or even days of trial and error
> as I continue to learn. Thank you again for a great set of tools to
> help move the "IoT" forward!

Cool! Welcome to the WAMP ecosystem!

Cheers,
/Tobias

>
> Dave B.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/36e160fd-c05b-4203-a1f9-3aea554e8453%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/36e160fd-c05b-4203-a1f9-3aea554e8453%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Dave Barndt]

Hi Tobias,

Thanks so much for your reply.  I'm sorry for not returning to this topic sooner. I am encouraged by your note; it sounds like I'm generally on the right track. I have a few clarifications and follow-up questions if that's OK...

On Wednesday, November 5, 2014 6:35:20 PM UTC-5, Tobias Oberstein wrote:

Am 02.11.2014 05:27, schrieb Dave Barndt:
> So here's my project in a nutshell. I have some number of identical
> embedded devices on a network. There will be some backend system that
> they all connect to (and they also may potentially connect to each
> other, if it makes sense).  Each embedded device will serve a local
> "console" website (HTML/JS/static files), but for "serving" dynamic data

Why does each and every device serve static content? Just trying to
understand your goals ..

Each device needs to be able to be browsed to "locally", without any additional network infrastructure. Right now the website is served via NodeJS (already comes with the Linux distro on the device) and is a simple single-page template; once the page is in the browser, all data will be requested by an AutobahnJS client in the page, to be returned by a AutobahnPython (basic) WAMP router/server.  (It's Python because we have some other logic in Python that the WAMP server has to call to interact with the device).

There will also be a network backend system that will probably have another, centralized router (like Crossbar.io) as well, for doing things across multiple devices at the network level. But right now I'm more focused on the "local" device interaction.

> embedded devices to be able to "register" on the network when they come
> online (ex. perhaps they could potentially have a config setting
> pointing to the (backend) system to register with).

Yep, they could have a local config with minimum:

- WAMP router URL ("wss://...")
- WAMP realm ("realm1")

plus possibly authentication information.

OK, cool. Here's a follow-up question I have: What would be your recommendation for the best way to "partition" WAMP API/resources so that individual devices or potentially groups of devices could be "addressed" correctly? Here's a little more background:

Each device will offer the same basic set of data. A device's RPC "Procedure URIs" could be CRUD-like to get specific data, e.g.: "domain.device.create", "domain.device.read", etc.(and the data could be specified as call parameters with URIs like "/api/v1/some-data").

It seems that if I want to "address" a single device, a realm for each device (perhaps using its unique name) could be defined. The other way I could see to do it would be to specify a device name in the RPC URI, e.g. "domain.device.device_name.create", etc. But this seems "less correct". Is using realms the correct approach?

And a couple follow-up questions:

1) For a "centralized" WAMP router on the network (i.e., not in a device): Can a router like Crossbar.io handle/manage more than one realm?  I think the answer's yes, but I'm not sure. For example, could a realm define a group of devices (ex. in a given area), and another realm define an individual device, and a "caller" or 'subscriber" could get data from the desired device(s) just by referencing the correct realm?

2) Thinking about 1) - are routing RPCs to multiple devices in a realm supported? Not sure how the returned data would be managed.

3) Thinking about 1) - subscriptions to any device in a realm are supported, correct?

> 4) I'm thinking when a device 'registers" on the network, it would
> register the RPCs and channels it wants to offer with the router. Is it
> possible somehow to query to get a digest/directory of all RPCs
> registered with a Dealer?  Same question for all channels available to
> be subscribed to?

Listing and reflecting upon registered procedures, as well as listing
subscribers: this is being defined in WAMP "Advanced PRofile". Not yet
finalized in the spec, and only partially implemented (and undocumented)
in Crossbar.io

Same for reflecting on "all topic available to subscribe to".

Note my different use of "listing" and "reflecting".

E.g. Crossbar.io implements the approach described at the end of

https://github.com/tavendo/WAMP/issues/61

You can "wamp.reflection.describe" on both a procedure or topic which
you have previously stored info for via "wamp.reflection.declare".

Wow, I will look into this, thanks. Perhaps if necessary, as a stop gap until the Crossbar.io spec is ready I could create some sort of a "list" of RPCs and topics that could be registered.

 

> I hope I haven't written too much for one topic/question, but any
> answers you can give will save me hours or even days of trial and error
> as I continue to learn.  Thank you again for a great set of tools to
> help move the "IoT" forward!

Cool! Welcome to the WAMP ecosystem!

Glad to be here. :^)  Thanks for the support and any corrections in my understanding.  Again, this is some great s/w.

Dave

[Tobias Oberstein]

Hi Dave,

> OK, cool. Here's a follow-up question I have: What would be your
> recommendation for the best way to "partition" WAMP API/resources so
> that individual devices or potentially groups of devices could be
> "addressed" correctly? Here's a little more background:
>
> Each device will offer the same basic set of data. A device's RPC
> "Procedure URIs" could be CRUD-like to get specific data, e.g.:
> "domain.device.create", "domain.device.read", etc.(and the data could be
> specified as call parameters with URIs like "/api/v1/some-data").
>
> It seems that if I want to "address" a single device, a realm for each
> device (perhaps using its unique name) could be defined. The other way I
> could see to do it would be to specify a device name in the RPC URI,
> e.g. "domain.device.device_name.create", etc. But this seems "less
> correct". Is using realms the correct approach?

Realms are completely separate name (URI) and routing domains. That
means, a subscriber on realm A won't receive event published on realm B
(and vice versa). Same for RPC.

If you want your devices being able to talk to each other, realms are
not the way, but device specific URIs

com.myapp.device.123.turn_on_led
com.myapp.device.123.read_sensor

where 123 is the specific device.

>
> And a couple follow-up questions:
>
> 1) For a "centralized" WAMP router on the network (i.e., not in a
> device): Can a router like Crossbar.io handle/manage more than one
> realm? I think the answer's yes, but I'm not sure. For example, could a

Yes. A single Crossbar.io can manage/handle any number of realms (well,
until you run out of memory/cpu).

> realm define a group of devices (ex. in a given area), and another realm
> define an individual device, and a "caller" or 'subscriber" could get
> data from the desired device(s) just by referencing the correct realm?

A single WAMP session is alway attached to exactly one specific realm,
and WAMP calls/events on one realm never cross over to another realm.

> 2) Thinking about 1) - are routing RPCs to multiple devices in a realm
> supported? Not sure how the returned data would be managed.

This is an upcoming feature of WAMP "Advanced Profile":
"distributed/partitioned calls". The router will have different
policies: accumulate all results (make a list of results) and only
return the accumulated result to the caller when all are available, or
return individual results immeditately as "progressive results" (already
implemented today) to the caller.

> 3) Thinking about 1) - subscriptions to any device in a realm are
> supported, correct?

If you mean, subscribe to URI "com.myapp.*" (a wildcard subscription),
this is also WAMP "Advanced Profile" and not yet released. But the
feature is hashed out and definitely coming.

>
> Glad to be here. :^) Thanks for the support and any corrections in my
> understanding. Again, this is some great s/w.

Great! Sorry for so many "yes, but not yet" answers - this whole thing
is quite an effort, still moving fast and growing.

Cheers,
/Tobias

>
> Dave

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/6a6235f2-6bf7-4b28-931a-c0de1edba978%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/6a6235f2-6bf7-4b28-931a-c0de1edba978%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Dave Barndt]

Hi Tobias,

Thank you very much again. Your replies plus some initial good old-fashioned trial-and-error prototyping have helped a lot.

If I may, I'd like to summarize the updated design - if you can tell me I'm making good/correct/intended use of WAMP/Autobahn I'd appreciate it. I'm hoping the scenario is common enough that someone else out there reading this could benefit from it.

So assume devices A1, A2, A3 ... Ax will be on the network.  A backend system will also be on the network.

1) I have a requirement for a "local" use case - for example, where a user walks up to and connects directly to device A1 without the backend system involved. For this case, the A1 device will serve a local website (via NodeJS) consisting of a single page containing AutobahnJS. AutobahnJS will connect back to a WAMP basic router on the A1 device (AutobahnPython) in order to call (RPC) and/or subscribe for (PubSub) data from the A1 device. In this case, the WAMP basic router on the device will create one session/realm, which will be the name of the device ("A1").

2) I have a requirement for the networked case where a user connects to the network backend system to talk to one or more devices. As setup for this case, the Crossbar.io advanced router will be running on the backend system. Each device will have settings in its configuration that direct it to connect to and "register" with Crossbar.io on the backend system. "Registering" means the device will connect once using its own name as one realm (ex. "A1"), once using an "all"-type realm (i.e., the device is part of the realm of "all devices", and once for each group-level realm (ex. "northeast", which contains devices A1, A2, and A3) its configured to belong to. The Crossbar.io router will simply create/add any realms that don't exist (the basic router spec seems to indicate this possible in the WELCOME message discussion). Each device will then register all its procedures that can be called and all its topics that can be subscribed to on all its realms.

So then for example: if a client connects to the "all" realm on the backend system router (Crossbar.io) and subscribes to a topic, will the client automatically receive all messages published by all devices to that topic? And if a client makes an RPC call to a procedure in the "northeast" realm, will all devices in the "northeast" realm be automatically called and the results returned (I understand individually for now, possibly aggregated in the future)?

If so, I'm pretty much in awe of what this architecture can do. :^)

Dave

[Alexander Gödde]

Hi Dave,

some answers below:

 

Hi Tobias,

Thank you very much again. Your replies plus some initial good old-fashioned trial-and-error prototyping have helped a lot.

If I may, I'd like to summarize the updated design - if you can tell me I'm making good/correct/intended use of WAMP/Autobahn I'd appreciate it. I'm hoping the scenario is common enough that someone else out there reading this could benefit from it.

So assume devices A1, A2, A3 ... Ax will be on the network.  A backend system will also be on the network.

1) I have a requirement for a "local" use case - for example, where a user walks up to and connects directly to device A1 without the backend system involved. For this case, the A1 device will serve a local website (via NodeJS) consisting of a single page containing AutobahnJS. AutobahnJS will connect back to a WAMP basic router on the A1 device (AutobahnPython) in order to call (RPC) and/or subscribe for (PubSub) data from the A1 device. In this case, the WAMP basic router on the device will create one session/realm, which will be the name of the device ("A1").

2) I have a requirement for the networked case where a user connects to the network backend system to talk to one or more devices. As setup for this case, the Crossbar.io advanced router will be running on the backend system. Each device will have settings in its configuration that direct it to connect to and "register" with Crossbar.io on the backend system. "Registering" means the device will connect once using its own name as one realm (ex. "A1"), once using an "all"-type realm (i.e., the device is part of the realm of "all devices", and once for each group-level realm (ex. "northeast", which contains devices A1, A2, and A3) its configured to belong to. The Crossbar.io router will simply create/add any realms that don't exist (the basic router spec seems to indicate this possible in the WELCOME message discussion). Each device will then register all its procedures that can be called and all its topics that can be subscribed to on all its realms.

- At some point in the past, Crossbar.io did automatically create realms on first connect, i.e. show the behavior you want. At the moment, as far as I know (and some quick tests indicate) realms are created explicitly as part of router configuration. Attempting to connect to a non-configured realm results in a refusal of the connection. It's also not possible to configure an "any" realm using a placeholder - though that might be a good idea. If you think this would be a good feature, then I suggest you file an enhancement request. 

 

So then for example: if a client connects to the "all" realm on the backend system router (Crossbar.io) and subscribes to a topic, will the client automatically receive all messages published by all devices to that topic? And if a client makes an RPC call to a procedure in the "northeast" realm, will all devices in the "northeast" realm be automatically called and the results returned (I understand individually for now, possibly aggregated in the future)?

- Publications are dispatched to all subscribers. This is per realm, so messages on e.g. the "northeast" realm for "topic1" will not be dispatched to subscribers to "topic1" on "all" relam.

- RPC calls are forwarded to the client within the realm which has registered them. 

Hope this helps a bit!

Regards,

Alex

[Dave Barndt]

Hi Alexander,

Yes, those answers are helpful - thank you! It sounds like I'm on the right track. It shouldn't be a big deal to have to configure devices and routers to connect to the correct realms, since it's not something that will change very often at all. Kind of "set it and forget it".

I will keep prototyping and report any issues and ask any questions I have.

So thank you again! You guys have been exceptionally helpful in your replies.

Dave