Building real-time database-backed web applications with Autobahn/Crossbar

409 views
Skip to first unread message

Jimmy Jia

unread,
Jan 12, 2015, 1:38:32 PM1/12/15
to autob...@googlegroups.com
Hi,

I'm interested in the use case of building a publicly facing multi-user web application, with support for synchronization of models in real time between the client and the server, with per-user ownership of objects in the database. Essentially, this would be the real-time equivalent of an app built with Django + Django Rest Framework + django-guardian, exposing an real-time API over WAMP rather a REST API over HTTP.

One example of a framework that makes it relatively easy to do such a thing is Meteor, see e.g. their (old) screencast for building a party planning app: https://www.youtube.com/watch?v=cSnANePlEjg (~10 minutes). Semantically this is not much more than just "TodoMVC with auth", if you're familiar with that example.

There are a couple of other Python-based packages that offer similar support, such as SwampDragon and webalchemy, but they're both somewhat questionable for various purposes, and don't give me the same flexibility that WAMP does. There's bits and pieces of this in the Crossbar roadmap with things like database integration. Other parts like user auth seem like they're deliberately set up as pluggable modules.

In general, does this seem like the sort of application that would make sense to build on top of Crossbar? Which parts of this kind of functionality are intended as part of Crossbar, while which should happen in a library built on top of it?

Thanks,
Jimmy Jia

Greg Miller

unread,
Jan 12, 2015, 7:12:33 PM1/12/15
to autob...@googlegroups.com
Back in the summer I spent quite a bit of time with Meteor.  It's quite impressive but I felt it was a little constricting, and for various reasons abandoned it.  But I didn't abandon the concept of reactivity.  Having found crossbar/autobahn helped me decide to ditch Meteor and to 'roll my own'.  I was mostly interested in knowing when records in a DB changed.  I use Kendo UI widgets and their version of MVVM.  From there it was just a matter of knowing when a record(s) changed in a table and then re-executing a read.  In Crossbar I have one generic procedure for each of the CRUD operations.  As soon as the operation is complete I publish the table/record to anyone that's listening, using crossbar's pub/sub. In the browser I have some code that registers what to do when a table/record is affected and then subscribes to that event.  When the event occurs that code runs and Kendo's MVVM updates the page.  It works fine, and is nice and quick.  I haven't yet tried to replicate Meteor's reactive select.  I was hoping to find a DB that takes care of it, and rethinkdb sort of does, but they aren't quite there yet.

I could make what I have better by having a more defined event as right now the browser reacts when any table/record is changed and it decides if it's interested in the change.  I could have one event per table, or perhaps one for each table/record. 

From my point of view I use crossbar for what it can do, pub/sub, RPC, and I do the rest.

Jimmy Jia

unread,
Jan 12, 2015, 7:47:24 PM1/12/15
to autob...@googlegroups.com
I should say that I have written up chunks of this on my own as well. I'm using the current Crossbar database integration with PostgreSQL triggers to automatically dispatch updates to tables onto topics of my choice, and have RPC bits for getting table snapshots (this along with the topic give me read functionality), along with other RPC bits for create, update, and delete.

The real problem here is user auth and permissioning. WAMP-CRA/ticket with dynamic authentication lets me do password-based authentication, except it looks to be a bit awkward for setting up a user registration - and realistically it'd be nicer for this to be offered at a library/framework level, like how Meteor users work. For example, since the challenge is made as soon as the connection is made, you end up needing to do awkward things like dropping-and-re-establishing the connection to allow a user to actually log in, and overall it seems a bit awkward, but perhaps the complexity is better resolved in a higher-level library.

The other issue is that the WAMP v2 subscriber whitelisting is not set up to allow this pattern in a transparent way. Ideally I'd like to be able to whitelist a set of users (auth ids and/or auth roles) to receive a message on a topic, rather than a set of subscription IDs. I feel like the right way to deal with object-level permissioning in this context is for all updates to go to a single topic, for the publisher to designate the authorized users, and for the broker to take care of making sure that only the properly authorized users see updates. I know that you can do this right now with per-user topics, but it's not ideal and doesn't really let you get to the level of richness or convenience that you get with a real object-level permissioning scheme.

It's possible that a lot of this can (and maybe should, since you'd probably want SQLAlchemy integration) live in a framework that wraps Crossbar, but I think the authorization and subscriber whitelisting are not ideal. You can wrap some or all of this, but it feels like it'd be re-inventing the wheel.

I'd also like to know how Tavendo see this working. Building a decent real-time CRUD analogue (with auth and object-level permissions) is a huge pain point right now when working in anything other than JavaScript, so having something that makes it possible to build such a system while maintaining the flexibility of using WAMP underneath seems like something that would be a very strong sell.

Greg Fausak

unread,
Jan 13, 2015, 12:28:56 AM1/13/15
to autob...@googlegroups.com
I wrote some stuff that does this in Autobahn, but I haven't ported to Crossbar.  I posted it to pypi (sqlauth), it manages roles/users and the permissions granted to them (in relationship to publish,subscribe,call,register).  I added another primitive called 'admin', which is the ability to delegate granting of permissions.  It sounds like we are all working on the same issues.

-g

Jimmy Jia

unread,
Jan 13, 2015, 1:39:20 AM1/13/15
to autob...@googlegroups.com
I talked this through with a co-worker.

On second thought, the auth isn't really an issue because you can do the actual auth/user management through an existing framework that handles everything for you and just use a token to authenticate with the router.

Looking at how the Meteor publish/subscribe API looks (http://docs.meteor.com/#/basic/Meteor-publish), you probably actually want to set up a new topic per subscription anyway, to support things like applying client-requested filters on collections on the server side anyway.

I think all that's missing then is support for subscriber meta events, so I know to clean up subscriptions for record sets after clients go away.

Tobias Oberstein

unread,
Jan 15, 2015, 4:35:21 PM1/15/15
to autob...@googlegroups.com
Hi Jimmy,

Am 12.01.2015 um 19:38 schrieb Jimmy Jia:
> Hi,
>
> I'm interested in the use case of building a publicly facing multi-user
> web application, with support for synchronization of models in real time
> between the client and the server, with per-user ownership of objects in
> the database. Essentially, this would be the real-time equivalent of an
> app built with Django + Django Rest Framework + django-guardian,
> exposing an real-time API over WAMP rather a REST API over HTTP.
>
> One example of a framework that makes it relatively easy to do such a
> thing is Meteor, see e.g. their (old) screencast for building a party

Meteor is different from WAMP as the core abstraction exposed to apps
are synchronized datasets, whereas in WAMP it is RPC + PubSub.

You can build the former on top of the latter in a natural way. The
other direction is not possible, or at least would be extremely hacky.

> planning app: https://www.youtube.com/watch?v=cSnANePlEjg (~10 minutes).
> Semantically this is not much more than just "TodoMVC with auth", if
> you're familiar with that example.
>
> There are a couple of other Python-based packages that offer similar
> support, such as SwampDragon and webalchemy, but they're both somewhat
> questionable for various purposes, and don't give me the same
> flexibility that WAMP does. There's bits and pieces of this in the
> Crossbar roadmap with things like database integration. Other parts like
> user auth seem like they're deliberately set up as pluggable modules.

Yes, authentication and authorization: Crossbar.io provides the
frameworks, but specific stuff is done in components - just regular WAMP
components _used_ for the AA purpose.

>
> In general, does this seem like the sort of application that would make
> sense to build on top of Crossbar? Which parts of this kind of
> functionality are intended as part of Crossbar, while which should
> happen in a library built on top of it?

If you want something Meteor like (synchronized datasets), I guess this
is pretty straighforward on top of Crossbar.io - means, synchronized
dataset is not something we want to provide in Crossbar.io itself. So
yes: library/framework on top of Crossbar.io.

Cheers,
/Tobias

>
> Thanks,
> Jimmy Jia
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/aa6d295e-bc6e-4c86-88cd-ff1fade177db%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/aa6d295e-bc6e-4c86-88cd-ff1fade177db%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

Jimmy Jia

unread,
Jan 15, 2015, 4:38:52 PM1/15/15
to autob...@googlegroups.com
That makes sense, and I think it's a pretty significant use case, and something that, like I've said before, is very painful unless you want to use either Meteor or maybe Firebase, neither of which give you any real control over your data.

The reason I bring this up in the first place is because of the direct database integration in Crossbar. It seems like one very natural thing to do with a database table in this sort of system is to provide the abstraction of a synchronized view. Since I'd rather minimize how much I re-invent the wheel, what do you see as the use case for the direct database integration in Crossbar (v. what should live in a separate library instead)?

Tobias Oberstein

unread,
Jan 15, 2015, 5:17:22 PM1/15/15
to autob...@googlegroups.com
Am 15.01.2015 um 22:38 schrieb Jimmy Jia:
> That makes sense, and I think it's a pretty significant use case, and
> something that, like I've said before, is very painful unless you want
> to use either Meteor or maybe Firebase, neither of which give you any
> real control over your data.
>
> The reason I bring this up in the first place is because of the direct
> database integration in Crossbar. It seems like one very natural thing
> to do with a database table in this sort of system is to provide the
> abstraction of a synchronized view. Since I'd rather minimize how much I
> re-invent the wheel, what do you see as the use case for the direct
> database integration in Crossbar (v. what should live in a separate
> library instead)?

The database integration of Crossbar.io works differently: the
abstractions exposed to apps are exactly the same: RPC + PubSub (not
synchronized datasets). That is, a database (like PostgreSQL or Oracle)
is integrated at the WAMP level.

You can call a database stored procedure exactly/transparently as any
other WAMP procedure.

You can publish a WAMP event from within a database stored procedure or
database trigger, and the event will be received as any other WAMP event.

A database table is not exposed as some kind of synchronized dataset.

The former is already (partially) in Crossbar.io. The latter is not.

Note that the database integration in Crossbar.io isn't via code
_inside_ the router. That is wrong for apps, and it is wrong even for
this stuff. It's regular components, but bundled with Crossbar.io

So if you want to build a Meteor style dataset synch, awesome! Go ahead!
Do it with regular WAMP components.

The question if you want to do that as your own, separate project or
have it bundled with Crossbar.io (as the database integration above) is
a different matter. Are you asking about this?
> > an email to autobahnws+...@googlegroups.com <javascript:>
> > <mailto:autobahnws+...@googlegroups.com <javascript:>>.
> > To post to this group, send email to autob...@googlegroups.com
> <javascript:>
> > <mailto:autob...@googlegroups.com <javascript:>>.
> <https://groups.google.com/d/msgid/autobahnws/aa6d295e-bc6e-4c86-88cd-ff1fade177db%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/6b552915-a762-4d6c-8009-783776c26fc2%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/6b552915-a762-4d6c-8009-783776c26fc2%40googlegroups.com?utm_medium=email&utm_source=footer>.

Tobias Oberstein

unread,
Jan 15, 2015, 5:21:20 PM1/15/15
to autob...@googlegroups.com
forget to add:

both approaches/abstractions

a) RPC/Event <=> database procedures / triggers

b) synchronized dataset <=> database table

are fine, and there a valid use cases for both, and each can "win"
depending on the situation.

Also: we can have it both. It's not either or. They are just "different" ..

Jimmy Jia

unread,
Jan 15, 2015, 5:29:05 PM1/15/15
to autob...@googlegroups.com
I think I just misunderstood what the motivation of the database integration was, and you've clarified it now.

I was under the impression that the exposed PubSub interface would be something for synchronizing table contents, instead of just for exposing triggers from the database side (plus RPC to expose stored procedures).

Given what you've said, it makes sense to me that the synchronized dataset bit would be another component that I could run in a system with a Crossbar router, rather than part of the core Crossbar package itself.

Thanks,
Jimmy Jia

Tobias Oberstein

unread,
Jan 15, 2015, 5:33:38 PM1/15/15
to autob...@googlegroups.com
Am 15.01.2015 um 23:29 schrieb Jimmy Jia:
> I think I just misunderstood what the motivation of the database
> integration was, and you've clarified it now.
>
> I was under the impression that the exposed PubSub interface would be
> something for synchronizing table contents, instead of just for exposing
> triggers from the database side (plus RPC to expose stored procedures).

The RPC-PubSub integration is far more than exposing triggers. It is for
writing business logic that runs close to your data inside the database.
E.g. in PostgreSQL you can write your stored procedures in PL/pgSQL,
JavaScript, Python, R, ...

>
> Given what you've said, it makes sense to me that the synchronized
> dataset bit would be another component that I could run in a system with
> a Crossbar router, rather than part of the core Crossbar package itself.

Exactly. Technically, definitely outside the router, implemented as
regular components.

This is what I mean with: the question if these components are then part
of a separate project, or bundled with Crossbar.io is orthogonal to the
technical thing above (that is, they should be regular WAMP components).

Cheers,
/Tobias

>
> Thanks,
> Jimmy Jia
>
> On Thursday, January 15, 2015 at 5:21:20 PM UTC-5, Tobias Oberstein wrote:
>
> forget to add:
>
> both approaches/abstractions
>
> a) RPC/Event <=> database procedures / triggers
>
> b) synchronized dataset <=> database table
>
> are fine, and there a valid use cases for both, and each can "win"
> depending on the situation.
>
> Also: we can have it both. It's not either or. They are just
> "different" ..
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/ffe50c3c-296e-4e2d-ab99-dea0ea9b8091%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/ffe50c3c-296e-4e2d-ab99-dea0ea9b8091%40googlegroups.com?utm_medium=email&utm_source=footer>.

Jimmy Jia

unread,
Jan 27, 2015, 6:48:33 PM1/27/15
to autob...@googlegroups.com
Following up on this - with RethinkDB announcing that they're adding native support for this, I'm thinking that this sort of table sync built on top of Crossbar is no longer the way I am going to go.

I'd of course prefer PostgreSQL over RethinkDB, but real-time database sync to browser as a database feature is too compelling of a feature to pass on.

Jimmy Jia

unread,
Jan 27, 2015, 6:48:55 PM1/27/15
to autob...@googlegroups.com
Oops, forgot to include link to RethinkDB's announcement: http://rethinkdb.com/blog/realtime-web/
Reply all
Reply to author
Forward
0 new messages