WAMP v2

323 views
Skip to first unread message

Tobias Oberstein

unread,
Dec 19, 2013, 1:04:22 PM12/19/13
to autob...@googlegroups.com, wam...@googlegroups.com
Hi,

I've consolidated all (most) ideas for WAMP v2 that accumulated over the
last year into a cohesive design

https://github.com/tavendo/WAMP/tree/master/spec

This isn't finished yet, but ready for review and feedback.

WAMP v2 brings major improvements:

- support for binary serialization (MsgPack)
- transport independence (besides WebSocket, it'll have HTTP long-poll
and possibly SPDY in the future)
- Acknowledgements for Subscribe/Publish
- Pattern-based Subscriptions
- Meta-Events
- Publisher and Event Identification
- Progressive Call Results
- Call Canceling
- no more "CURIEs" (PREFIX) .. no more HTTP URIs ..

but probably most important: complete symmetry.

We now have a clean, role-based fully symmetric protocol with the
following roles:

PubSub: Publisher, Subscriber and Broker
RPC: Caller, Callee and Dealer

This is (for me) the most exciting, since it will finally fully live up
to the original vision of enabling flexible, component based, decoupled
application architectures.

E.g., you will be able to implement RPC endpoints in JavaScript running
in the browser and call those from a backend. Or call RPC endpoints that
run in a database or your Arduino.

You might even develop/debug an RPC endpoint in the browser and later
move the code to a Node.js backend - without a single line of code changed!

This is a big deal. I think.

Please provide feedback / review .. now is the time.

Even more so since: large chunks of the implementation of WAMPv2 are
already started in AutobahnPython.

The other Autobahn's will follow (AutobahnJS and AutobahnAndroid).

At least with Autobahn, we will only support WAMPv2 upcoming - the
changes from WAMPv1 are too large to be worth supporting both.

Cheers,
/Tobias

Elad Zelingher

unread,
Dec 22, 2013, 5:31:34 PM12/22/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi,

I skimmed the spec and I have a few questions:

Serialization: both JSON and MsgPack support number, bool and null. I understand that some other protocols don't support these types, but I think it is not a good enough reason to disallow these types.

Pub/Sub: the EVENT message now contains custom details per client (SubscriptionId) - this prevents the serialization optimization that allowed serializing the same event's EVENT message for all clients in a specific topic. (I haven't implemented it, but it was implemented in autobahn python) - have you considered this?

Pub/Sub: I've received some feedbacks about WAMPv1, suggesting to allow passing a custom parameter to SUBSCRIBE which allows to define a custom subscription (for example, the subscriber can send an object that represents a filter, and only events that pass that filter will be sent to the subscriber).
Since SUBSCRIBE now receives a options parameter, can it be used to send custom data to subscription? (of course this will be supported in a specific implementation)

RPC: Is the capability which allows one client to call another
client's RPC method necessary? I understand the advantages of a capability that allows the server to call a client's RPC method, but I find it hard to understand the use cases in which I'd like to call some client's RPC method from another client.

I'll write here more questions if they'll pop.

Elad

Tobias Oberstein

unread,
Dec 22, 2013, 6:00:15 PM12/22/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi Elad,

thanks for your feedback .. I try to address the issues you raised.

> I skimmed the spec and I have a few questions:
>
> Serialization: both JSON and MsgPack support number, bool and null. I
> understand that some other protocols don't support these types, but I
> think it is not a good enough reason to disallow these types.

These types are of course allowed .. in _application_ payloads:

https://github.com/tavendo/WAMP/tree/master/spec#serialization

"WAMP itself only uses above types. The application payloads transmitted
by WAMP (e.g. in call arguments or event payloads) may use other types a
concrete serialization format supports."

>
> Pub/Sub: the EVENT message now contains custom details per client
> (SubscriptionId) - this prevents the serialization optimization that
> allowed serializing the same event's EVENT message for all clients in a
> specific topic. (I haven't implemented it, but it was implemented in
> autobahn python) - have you considered this?

It does _not_ prevent that optimization. The subscription ID is chosen
by the broker:

"Note. The Subscription ID chosen by the broker may be unique only for
the Topic (and possibly other information from Options, such as the
topic pattern matching method to be used). The ID might be the same for
any Subscriber for the same Topic. This allows the Broker to serialize
an event to be delivered only once for all actual receivers of the event."

https://github.com/tavendo/WAMP/tree/master/spec#subscribing-and-unsubscribing

>
> Pub/Sub: I've received some feedbacks about WAMPv1, suggesting to allow
> passing a custom parameter to SUBSCRIBE which allows to define a custom
> subscription (for example, the subscriber can send an object that
> represents a filter, and only events that pass that filter will be sent
> to the subscriber). Since SUBSCRIBE now receives a options parameter,
> can it be used to send custom data to subscription? (of course this will
> be supported in a specific implementation)

That's a good point! Yes, we might allow that .. having implementation
specific extensions.

The attribute names in details/options dictionaries should then probably
be prefixed like

"x_..."

so that no collisions with "officially" specified behavior occurs.
Similar to what is done in CSS ..

https://github.com/tavendo/WAMP/issues/27

If you have further ideas or requirements, please comment on that issue ..

>
> RPC: Is the capability which allows one client to call another client's
> RPC method necessary? I understand the advantages of a capability that
> allows the server to call a client's RPC method, but I find it hard to
> understand the use cases in which I'd like to call some client's RPC
> method from another client.

Think of "client" in a more abstract way (not "client" = "user"):

You might implement your _backend_ as a WAMP client .. eg have parts of
your backend implemented in NodeJS, have "adapters" that expose hardware
on an Arduino as RPCs, etc.

But a WAMP implementation is NOT required to implement the "Callee" role.

Personally, the full symmetry of WAMPv2 is what I see as the most
powerful .. it'll allow complete new system architectures.

>
> I'll write here more questions if they'll pop.

Yes, please!

Tobias
> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

Greg Fausak

unread,
Dec 23, 2013, 10:32:52 AM12/23/13
to autob...@googlegroups.com, wam...@googlegroups.com
Tobias,

Very nice.  There are a lot of good ideas in v2.  I am very interested in the rpc routing.  The any/all/partition calling endpoints can be very useful.  I know this is still a work in progress.  I couldn't figure out what a 'partition' call was about from the spec.  Can you explain what that is and how it is different than the 'all'?  Does a partition endpoint get selected based upon input criteria?

I think the 'any' can be extremely useful for implementing fault tolerant architectures.  I've got a few ideas that I'll just throw out.  The proposed random selection of the endpoint could use some ios 'queue' behaviors (round robin, weighted, etc..).  The endpoint selection could also benefit from external metrics feedback (like a geo-latency selector, shorter queries first, etc).  Also, the 'any' might benefit from some stubborn behaviors, like, keep selecting an endpoint until one succeeds.

Best,

-greg

Tobias Oberstein

unread,
Dec 23, 2013, 1:04:49 PM12/23/13
to autob...@googlegroups.com, wam...@googlegroups.com
Hi Gre,

Am 23.12.2013 16:32, schrieb Greg Fausak:
> Tobias,
>
> Very nice. There are a lot of good ideas in v2. I am very interested
> in the rpc routing. The any/all/partition calling endpoints can be very
> useful. I know this is still a work in progress. I couldn't figure out
> what a 'partition' call was about from the spec. Can you explain what
> that is and how it is different than the 'all'? Does a partition
> endpoint get selected based upon input criteria?

Yes. With "all" and "any", the endpoints the Dealer routes a call to are
either implicit ("all") or decided by the Dealer ("any").

With "partition", the Callee provides a "partition key" (e.g. a hash of
a customer number) in the CALL.Details (since the Dealer should stay
completely agnostic to the application payload).

The Dealer then uses this "partition key" to route the call.

Here is the scenario: say you have your backend/data partitioned across
10 nodes (say 1/10 of customer data on each node). Then you want to
route calls to the right node (holding the 1/10 data that includes the
data relevant to the respective call).

>
> I think the 'any' can be extremely useful for implementing fault
> tolerant architectures. I've got a few ideas that I'll just throw out.

Exactly. Fault tolerance. Load balancing. That stuff ..

> The proposed random selection of the endpoint could use some ios
> 'queue' behaviors (round robin, weighted, etc..). The endpoint
> selection could also benefit from external metrics feedback (like a
> geo-latency selector, shorter queries first, etc). Also, the 'any'
> might benefit from some stubborn behaviors, like, keep selecting an
> endpoint until one succeeds.

Those are all very good ideas. I'll try to add those to the spec. In
essence, with "any", it's the Dealer's decision to which endpoint to
route .. and that decision might be based on a number of policies (and
the "random" selection we have in the spec right now is just one).

If you have more comments and/or ideas, yes please!

/Tobias

>
> Best,
>
> -greg
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com.

Jordi Mariné Fort

unread,
Dec 23, 2013, 3:19:36 PM12/23/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi,

What about these message simplifications:

1) Avoid correlation of request|Id and subscription|Id in the unsubscription process:
-  [UNSUBSCRIBE,Request|id,Subscription|Id] --> [UNSUBSCRIBE,Subscription|Id]
-  [UNSUBSCRIBED,Request|id] --> [UNSUBSCRIBED,Subscription|Id]

2) Unify some messages types:
- CALL and INVOCATION  (the INVOCATION.REGISTERED.Registration|id parameter value could be send in the CALL.Options parameter)
- CANCEL_CALL and CANCEL_INVOCATION
- CALL_RESULT and INVOCATION_RESULT
- CALL_ERROR and INVOCATION_ERROR
- CALL_PROGRESS and INVOCATION_PROGRESS


Tobias Oberstein

unread,
Dec 23, 2013, 3:56:02 PM12/23/13
to wam...@googlegroups.com, autob...@googlegroups.com
Am 23.12.2013 21:19, schrieb Jordi Marin� Fort:
> Hi,
>
> What about these message simplifications:
>
> 1) Avoid correlation of request|Id and subscription|Id in the
> unsubscription process:
> - [UNSUBSCRIBE,Request|id,Subscription|Id] -->
> [UNSUBSCRIBE,Subscription|Id]
> - [UNSUBSCRIBED,Request|id] --> [UNSUBSCRIBED,Subscription|Id]

UNSUBSCRIBE_ERROR would have to be changed as well. And consequently
UNREGISTER, UNREGISTERED and UNREGISTER_ERROR as well.

It would then depart of the common form for Request/Response message
exchanges which all are built around a request specific Request|Id.

But more importantly: what should be the Subscription|Id in an error for
a non-existing subscription? And what if more than 1 request to
UNSUBSCRIBE is outstanding? How do you correlate the responses (which
might differ in Error|Uri) with the original requests?

>
> 2) Unify some messages types:
> - CALL and INVOCATION (the INVOCATION.|REGISTERED.Registration||id
> parameter value could be send in the CALL.Options parameter)

CALL has Procedure|uri, whereas INVOCATION has REGISTERED.Registration
and (currently) no Procedure|uri. Probably INVOCATION should contain
Procedure|uri as well - that might allow pattern-based RPC endpoint
registrations.

> - CANCEL_CALL and CANCEL_INVOCATION
> - CALL_RESULT and INVOCATION_RESULT
> - CALL_ERROR and INVOCATION_ERROR
> - CALL_PROGRESS and INVOCATION_PROGRESS

The problem is: we want to allow a peer to both implement Callee and
Dealer roles. It would require the other peer to distinguish messages
between CALL_RESULT (as coming from Dealer) and INVOCATION_RESULT (as
coming from Callee) soley based on Request|Id. Again I think it makes
things more complicated and ambigious.

/Tobias

>
>
> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com.

par...@paroga.com

unread,
Dec 23, 2013, 8:15:59 PM12/23/13
to wam...@googlegroups.com, autob...@googlegroups.com
At first I must say that I like the improvements since v1.

Please think about a server who offers access to a filesystem (with only very small files): There is a method to read and write a file and every file change produces an event with the content of the file. Since I don't want to get an event for every file, I have to subscribe for all files I'm interested in. When I subscribe I send the filename in the Topic parameter (e.g. "com.paroga.filechange/folder/subfolder/file.ext"), which makes problems if the path contains a ".", so the path should be provided in an other parameter. In the read/write it's more or less clear that I'd use the argument for that [CALL, id, {}, "com.paroga.write", [], {"path":"/folder/subfolder/file.ext", "content":"new_content"}], but for the subscribe there the Options need to be abused for that. So I suggest adding arguments to the subscribe to allow [SUBSCRIBE, id, {}, "com.paroga.filechange", [], {"path":"/folder/subfolder/file.ext"}]. This would also avoid the "x_" prefix workaround for the Options.
If you think about a very frequently changing file I see an unnecessary overhead in sending the Topic all the time, since it's already clear from the Subscription. Since it's required for prefix/wildcard subscribe, I'd suggest to allow null or "" if it provides no additional information.
I somehow miss the option for PREFIX and suggest an ALIAS and maybe UNALIAS procedure as an replacement. I creates a simple mapping between the full URI and an usually shorter alias. This alias will then be valid everywhere for the type uri. This can also help for the previous point, since the server would be able to replace the long Topic in the EVENT message with the shorter alias.
Example:
C->S: [SUBSCRIBE, 1, {}, "com.paroga.filechange", [], {"path":"/folder/subfolder/file.ext"}]
S->C: [SUBSCRIBED, 1, 123]
C->S: [CALL, 2, {}, "com.paroga.write", [], {"path":"/folder/subfolder/file.ext", "content":"new_content"}]
S->C: [CALL_RESULT, 2, [], {}]
S->C: [EVENT, 123, 987, {}, "com.paroga.filechange", "new_content"]
becomes
C->S: [ALIAS, 1, "fc", "com.paroga.filechange"]
S->C: [ALIAS_RESULT, 1]
C->S: [ALIAS, 2, "w", "com.paroga.write"]
S->C: [ALIAS_RESULT, 2]
C->S: [SUBSCRIBE, 3, {}, "fc", [], {"path":"/folder/subfolder/file.ext"}]
S->C: [SUBSCRIBED, 3, 123]
C->S: [CALL, 4, {}, "w", [], {"path":"/folder/subfolder/file.ext", "content":"new_content"}]
S->C: [CALL_RESULT, 4, [], {}]
S->C: [EVENT, 123, 987, {}, "fc", "new_content"]

Are you happy with the current idea of meta events? Did you considered an META(UN)SUBSCRIBE instead? Or if it's only because of the list: What about a MULTI(UN)SUBSCRIBE, which is exactly like SUBSCRIBE, but allows a list for Topic.

Do you think PUBLISH.Options.rkey|string and PUBLISH.Options.nkey|string really belong into this spec? They seam quite specialized to me.

The names of the "good return" messages don't look very consistent to me. Some end to "ED" and some to "_RESULT". What about making all end to "_OK"? ("CALL_OK", "SUBSCRIBE_OK"). I'd also reorder some ids, so that the "good return" is always the id of the request incremented by one and the "bad return" incremented by two (CALL=70, CALL_OK=71,CALL_ERROR=72, CALL_PROGRESS=73, CANCEL_CALL=74)

Is there a good reason to use 2^53 instead of 2^32 for the upper bound of id? If the valid range would be 0 to 2^32-1 instead, CPUs could use simple registers to store the id.

Additionally I'd merge "Arguments|list, ArgumentsKw|dict" to "Arguments|any", because this looks very python-ish to me and WAMP is not python only. Providing a documentation for the WAMP->Python mapping would be better IMHO.

Do you accept pull request for changes in the spec?

Tobias Oberstein

unread,
Dec 24, 2013, 9:21:56 AM12/24/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi,

thanks for your detailed comments. Lets fold that into WAMPv2 ..

Am 24.12.2013 02:15, schrieb par...@paroga.com:
> At first I must say that I like the improvements since v1.
>
> Please think about a server who offers access to a filesystem (with only
> very small files): There is a method to read and write a file and every

Yep - thats good. Discuss a concrete scenario, and your's is certainly
one WAMPv2 should be able to cover.

> file change produces an event with the content of the file. Since I
> don't want to get an event for every file, I have to subscribe for all
> files I'm interested in. When I subscribe I send the filename in the
> Topic parameter (e.g.
> "com.paroga.filechange/folder/subfolder/file.ext"), which makes problems
> if the path contains a ".", so the path should be provided in an other

You might just replace '.' with '/' and '/' with '.' in filepaths. This
way you get topic URIs which conform to WAMPv2 and would be able to use
pattern-based subscriptions.

Original Filepath: /dir1/dir2/file1.ext
Topic: com.paroga.filesystem.onchange.dir1.dir2.file1/ext

Subscribe to: com.paroga.file.dir1 with Options.match == 'prefix' would
subscribe to any change beyond /dir1

[Ok, "/" as file extension separator doesn't look "pretty", but it
should work. And WAMP needs to have a special reserved character for
hierarchy. Using anything but '.' (or '/') would make WAMP URIs look exotic]

> parameter. In the read/write it's more or less clear that I'd use the
> argument for that [CALL, id, {}, "com.paroga.write", [],
> {"path":"/folder/subfolder/file.ext", "content":"new_content"}], but for

This is one option. What I find somewhat unsatisfying is that if you
would use above topic URIs, the URIs address application level objects
(resources = files/dirs). But with "com.paroga.write", the URI addresses
an operation, and the resource the operation works on is within the
CALL.Args.

An alternative would be

CALL.Procedure|uri == "com.paroga.filesystem.append.dir1.dir2.file1/ext" and
CALL.Args|list = ["new_content"]

Your filesystem implementation (the Callee that implements the append
operation) would then probably need to do pattern-based REGISTERs:

[REGISTER, Request|id, Options|dict, Procedure|uri]

Procedure == "com.paroga.filesystem.append"
Options == {"match": "prefix"}

The actual invocation of the "append" endpoint would then need to
provide the Procedure|uri

[INVOCATION, Request|id, REGISTERED.Registration|id, Options|dict,
CALL.Procedure|uri, CALL.Arguments|list, CALL.ArgumentsKw|dict]

where CALL.Procedure == "com.paroga.filesystem.append.dir1.dir2.file1/ext"

Your Callee implementation would then "cut out" everything that follows
the prefix "com.paroga.filesystem.append":

".dir1.dir2.file1/ext"

do the replacing '.' <=> '/':

"/dir1/dir2/file1.ext"

and perform the append with the supplied data in INVOCATION.Args to the
respective file.

I have added:
https://github.com/tavendo/WAMP/issues/28

Please leave your comments there. My vote is +1 for this.

> the subscribe there the Options need to be abused for that. So I suggest
> adding arguments to the subscribe to allow [SUBSCRIBE, id, {},
> "com.paroga.filechange", [], {"path":"/folder/subfolder/file.ext"}].

If you subscribe to "com.paroga.filechange", the Broker will dispatch
any file change event on any file to subscribers.

Brokers are unaware of a "path" attribute, whether that is in
SUBSCRIBE.Options or a new SUBSCRIBE.Args element.

If you come up with a Broker that processes events based on that custom
behavior, you won't be able to use any other WAMPv2 broker.

In general, the idea is to specify core routing behavior for WAMPv2 so
that Broker and Dealer implementations are interchangable.

> This would also avoid the "x_" prefix workaround for the Options.
> If you think about a very frequently changing file I see an unnecessary
> overhead in sending the Topic all the time, since it's already clear
> from the Subscription. Since it's required for prefix/wildcard
> subscribe, I'd suggest to allow null or "" if it provides no additional
> information.

Yes, we could do that optimization: in case the Subscription or
Registration is for "match" == "exact", transmit

EVENT.Topic == ""
INVOCATION.Procedure == ""

In case the Subscription/Registration is using "match" != "exact", the
fields would be filled.

My vote is +1 for this optimization (since it is also for a probably
very frequent case):
https://github.com/tavendo/WAMP/issues/29

> I somehow miss the option for PREFIX and suggest an ALIAS and maybe
> UNALIAS procedure as an replacement. I creates a simple mapping between
> the full URI and an usually shorter alias. This alias will then be valid
> everywhere for the type uri. This can also help for the previous point,
> since the server would be able to replace the long Topic in the EVENT
> message with the shorter alias.

There are multiple problems with PREFIX, one of which is: PREFIX in
WAMPv1 is per session. So a Broker cannot serialize an event once and
dispatch to all receivers. In fact, Autobahn will not use CURIEs in
EVENTs at all - even for WAMPv1.

There has been a longer discussion here:
https://github.com/tavendo/WAMP/issues/8

I don't think PREFIX or ALIAS brings any value, but introduces multiple
problems.

If you worry about wire level traffic volume: WebSocket has Deflate
compression. And this will compress not only URIs in WAMP messages, but
also URIs (and strings in general) within application payloads.

> Example:
> C->S: [SUBSCRIBE, 1, {}, "com.paroga.filechange", [],
> {"path":"/folder/subfolder/file.ext"}]
> S->C: [SUBSCRIBED, 1, 123]
> C->S: [CALL, 2, {}, "com.paroga.write", [],
> {"path":"/folder/subfolder/file.ext", "content":"new_content"}]
> S->C: [CALL_RESULT, 2, [], {}]
> S->C: [EVENT, 123, 987, {}, "com.paroga.filechange", "new_content"]
> becomes
> C->S: [ALIAS, 1, "fc", "com.paroga.filechange"]
> S->C: [ALIAS_RESULT, 1]
> C->S: [ALIAS, 2, "w", "com.paroga.write"]
> S->C: [ALIAS_RESULT, 2]
> C->S: [SUBSCRIBE, 3, {}, "fc", [], {"path":"/folder/subfolder/file.ext"}]
> S->C: [SUBSCRIBED, 3, 123]
> C->S: [CALL, 4, {}, "w", [], {"path":"/folder/subfolder/file.ext",
> "content":"new_content"}]
> S->C: [CALL_RESULT, 4, [], {}]
> S->C: [EVENT, 123, 987, {}, "fc", "new_content"]
>
> Are you happy with the current idea of meta events? Did you considered

Actually, METAEVENTs is an area we I'd very much welcome more input and
discussion.

> an META(UN)SUBSCRIBE instead? Or if it's only because of the list: What

My thinking is this: the subscription to a topic and associated
metatopics should happen in one request, and result in one subscription.

Say I have subscribed to "com.foo.bar" with "match" == "prefix".

How would I subscribe to metaevents for that subscription after the
initial "normal" subscription is already in place. That would need to
provide the SUBSCRIBED.Subscription|id. So it would need a new message type.

But I am open to have a discussion on METAEVENTs in general. We don't
have practical experience .. and people might have different
expectations. Up to now, I was only thinking of the

wamp.metatopic.subscriber.add
wamp.metatopic.subscriber.remove

metaevents. In fact, I can't come up with other useful "metaevents".
What do you think? What should be cover?

> about a MULTI(UN)SUBSCRIBE, which is exactly like SUBSCRIBE, but allows
> a list for Topic.

But what's the advantage?

Also: How would errors be handled? Say I multisubscribe 3 topics, and I
am not authorized for 1 of the 3.

How would Options be handled? Apply to all?

Since WAMPv2 request can be pipelined, you can send 100 subscribe
requests without waiting for each to return before sending the next.

>
> Do you think PUBLISH.Options.rkey|string and PUBLISH.Options.nkey|string
> really belong into this spec? They seam quite specialized to me.

They are part of the call/event routing flexibility that WAMP provides -
so yes, I think they are core.

>
> The names of the "good return" messages don't look very consistent to
> me. Some end to "ED" and some to "_RESULT". What about making all end to
> "_OK"? ("CALL_OK", "SUBSCRIBE_OK"). I'd also reorder some ids, so that
> the "good return" is always the id of the request incremented by one and
> the "bad return" incremented by two (CALL=70, CALL_OK=71,CALL_ERROR=72,
> CALL_PROGRESS=73, CANCEL_CALL=74)

If that systematics would make it easier for people to grasp the message
flows, then yes. Any opinions by others? I am mostly indifferent on this
one.

>
> Is there a good reason to use 2^53 instead of 2^32 for the upper bound
> of id? If the valid range would be 0 to 2^32-1 instead, CPUs could use
> simple registers to store the id.

Modern CPUs have 64 bit register banks. The upper bound is carefully
chosen to be representable in 64 bit integers and IEEE doubles, and at
the same time allow to make the probability of collisions practically zero.

>
> Additionally I'd merge "Arguments|list, ArgumentsKw|dict" to
> "Arguments|any", because this looks very python-ish to me and WAMP is
> not python only. Providing a documentation for the WAMP->Python mapping
> would be better IMHO.

No, please see here: https://github.com/tavendo/WAMP/issues/21

In fact, WAMPv2 allows positional and keyword based call results, which
isn't even in Python, but for example in PostgreSQL and Oracle.

And I was first wary of this, but beatgammit convinced me that we should
take the "superset of all" approach in WAMPv2. With WAMPv1, we tool the
"smallest common denominator" (positional args, 1 result) approach.

But, regarding arguments (and in general, all application payloads),
there is 1 major outstanding question:

All application payloads are now at the end within the lists forming
WAMP messages. The idea is, that Dealers and Brokers actually don't have
any need to parse or touch those app payloads, and might skip parsing
altogether.

Taking this idea further, I was thinking of making all app payloads just
strings (JSON) or binaries (MsgPack). So Dealers/Brokers can be even
more efficient and _probably_ we could even allow to transmit end-to-end
encrypted payloads.

Only the Subscriber/Publisher and Caller/Callee needs to perform a 2nd
parse on the WAMP level payload to get at the actual payload.

I would highly appreciate any input on this!

>
> Do you accept pull request for changes in the spec?

Yes. It would be good to have small PRs though (eg branch the spec per
issue/feature you would like to get merged). Git doesn't cope well with
cherry picking ..

Thanks!! And Merry Christmas;)

Patrick Gansterer

unread,
Dec 24, 2013, 9:08:57 PM12/24/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi,

On 24.12.2013, at 15:21, Tobias Oberstein <tobias.o...@gmail.com> wrote:
> thanks for your detailed comments. Lets fold that into WAMPv2 ..
more will follow - i just started ;-)

>> parameter. In the read/write it's more or less clear that I'd use the
>> argument for that [CALL, id, {}, "com.paroga.write", [],
>> {"path":"/folder/subfolder/file.ext", "content":"new_content"}], but for
>
> This is one option. What I find somewhat unsatisfying is that if you would use above topic URIs, the URIs address application level objects (resources = files/dirs). But with "com.paroga.write", the URI addresses an operation, and the resource the operation works on is within the CALL.Args.
>
> An alternative would be
>
> CALL.Procedure|uri == "com.paroga.filesystem.append.dir1.dir2.file1/ext" and
> CALL.Args|list = ["new_content"]
>
> Your filesystem implementation (the Callee that implements the append operation) would then probably need to do pattern-based REGISTERs:
>
> [REGISTER, Request|id, Options|dict, Procedure|uri]
>
> Procedure == "com.paroga.filesystem.append"
> Options == {"match": "prefix"}
>
> The actual invocation of the "append" endpoint would then need to provide the Procedure|uri
>
> [INVOCATION, Request|id, REGISTERED.Registration|id, Options|dict, CALL.Procedure|uri, CALL.Arguments|list, CALL.ArgumentsKw|dict]
>
> where CALL.Procedure == "com.paroga.filesystem.append.dir1.dir2.file1/ext"
>
> Your Callee implementation would then "cut out" everything that follows the prefix "com.paroga.filesystem.append":
>
> ".dir1.dir2.file1/ext"
>
> do the replacing '.' <=> '/':
>
> "/dir1/dir2/file1.ext"
>
> and perform the append with the supplied data in INVOCATION.Args to the respective file.
>
> I have added:
> https://github.com/tavendo/WAMP/issues/28
>
> Please leave your comments there. My vote is +1 for this.

Ok, pattern based registration is for sure an important point, but I have to solve an other problem. I must be able to set two files at the _same time_. If one fails, the whole call fails. Encoding a list of files then into the topic gets really ugly, so I still prefer this information in the payload.

>> the subscribe there the Options need to be abused for that. So I suggest
>> adding arguments to the subscribe to allow [SUBSCRIBE, id, {},
>> "com.paroga.filechange", [], {"path":"/folder/subfolder/file.ext"}].
>
> If you subscribe to "com.paroga.filechange", the Broker will dispatch any file change event on any file to subscribers.
>
> Brokers are unaware of a "path" attribute, whether that is in SUBSCRIBE.Options or a new SUBSCRIBE.Args element.
>
> If you come up with a Broker that processes events based on that custom behavior, you won't be able to use any other WAMPv2 broker.
>
> In general, the idea is to specify core routing behavior for WAMPv2 so that Broker and Dealer implementations are interchangable.

Ok, so I need to add additional requirements. ;-) I must be able to add additional "how to subscribe" stuff into the SUBSCRIBE. E.g. a maximum update rate, or filter for a specific user. Encoding this into the topic isn't a clean solution for me. Extending the options with all possible stuff, so that the broker is able to handle it, wouldn't work with very specific stuff.
This use-case will only work without a generic broker, so every subscriber has a publisher (which is then the WebSocket server directly). Do you have any idea for a clean solution where we can skip the broker and have a 1:1 matching for subscribers to be able to handle very specific subscribe requirements without violating the specification?

>> This would also avoid the "x_" prefix workaround for the Options.
>> If you think about a very frequently changing file I see an unnecessary
>> overhead in sending the Topic all the time, since it's already clear
>> from the Subscription. Since it's required for prefix/wildcard
>> subscribe, I'd suggest to allow null or "" if it provides no additional
>> information.
>
> Yes, we could do that optimization: in case the Subscription or Registration is for "match" == "exact", transmit
>
> EVENT.Topic == ""
> INVOCATION.Procedure == ""
>
> In case the Subscription/Registration is using "match" != "exact", the fields would be filled.
>
> My vote is +1 for this optimization (since it is also for a probably very frequent case):
> https://github.com/tavendo/WAMP/issues/29

Cool!

>> I somehow miss the option for PREFIX and suggest an ALIAS and maybe
>> UNALIAS procedure as an replacement. I creates a simple mapping between
>> the full URI and an usually shorter alias. This alias will then be valid
>> everywhere for the type uri. This can also help for the previous point,
>> since the server would be able to replace the long Topic in the EVENT
>> message with the shorter alias.
>
> There are multiple problems with PREFIX, one of which is: PREFIX in WAMPv1 is per session. So a Broker cannot serialize an event once and dispatch to all receivers. In fact, Autobahn will not use CURIEs in EVENTs at all - even for WAMPv1.
>
> There has been a longer discussion here:
> https://github.com/tavendo/WAMP/issues/8
>
> I don't think PREFIX or ALIAS brings any value, but introduces multiple problems.
>
> If you worry about wire level traffic volume: WebSocket has Deflate compression. And this will compress not only URIs in WAMP messages, but also URIs (and strings in general) within application payloads.

It's not only about the traffic, it's also about the CPU performance. Believe it or not, but I had an setup where the string lookup was an important factor to performance: Imaging the filesystem server from above, which sends an high amount of com.paroga.filechange events. The client needs to lookup the "client dispatch function" in a map (e.g. {"com.paroga.filechange":function(){}, "com.paroga.otherstuff":function(){}}). The length of the string made a noticeable difference.
If you now suggest to encode the path in the topic too, then the client dispatching gets even much more complicated.

>> an META(UN)SUBSCRIBE instead? Or if it's only because of the list: What
>
> My thinking is this: the subscription to a topic and associated metatopics should happen in one request, and result in one subscription.
>
> Say I have subscribed to "com.foo.bar" with "match" == "prefix".
>
> How would I subscribe to metaevents for that subscription after the initial "normal" subscription is already in place. That would need to provide the SUBSCRIBED.Subscription|id. So it would need a new message type.
>
> But I am open to have a discussion on METAEVENTs in general. We don't have practical experience .. and people might have different expectations. Up to now, I was only thinking of the
>
> wamp.metatopic.subscriber.add
> wamp.metatopic.subscriber.remove
>
> metaevents. In fact, I can't come up with other useful "metaevents". What do you think? What should be cover?

I think I understand now why you put them there: They will only create metaevents when "com.foo.bar" has been subscribed by some other peer. My initial understanding was that when I subscribe to the add metatopic I'll get metaevent when a peer subscribes to "com.foo" and "com.bar" and the "normal" topic is useless in the SUBSCRIBE message. Maybe that should be made more clear in the spec...

>> about a MULTI(UN)SUBSCRIBE, which is exactly like SUBSCRIBE, but allows
>> a list for Topic.
>
> But what's the advantage?

With your idea of metaevents: None :-)

> Also: How would errors be handled? Say I multisubscribe 3 topics, and I am not authorized for 1 of the 3.

Is subscribing to an metaevent always possible? Any security concerns about exposing this information?

>> Do you think PUBLISH.Options.rkey|string and PUBLISH.Options.nkey|string
>> really belong into this spec? They seam quite specialized to me.
>
> They are part of the call/event routing flexibility that WAMP provides - so yes, I think they are core.

Can you describe your use-case a little bit more? Routing flexibility is for sure something I agree with, but I didn't get the connection between EVENT and CALL. "nkey" is only mentioned for the EVENT message. What's the exact difference between a peer, node and resource?

>> The names of the "good return" messages don't look very consistent to
>> me. Some end to "ED" and some to "_RESULT". What about making all end to
>> "_OK"? ("CALL_OK", "SUBSCRIBE_OK"). I'd also reorder some ids, so that
>> the "good return" is always the id of the request incremented by one and
>> the "bad return" incremented by two (CALL=70, CALL_OK=71,CALL_ERROR=72,
>> CALL_PROGRESS=73, CANCEL_CALL=74)
>
> If that systematics would make it easier for people to grasp the message flows, then yes. Any opinions by others? I am mostly indifferent on this one.

I don't think that this is a big issue and have no strong opinion on it. Just wanted to mention it.

>> Is there a good reason to use 2^53 instead of 2^32 for the upper bound
>> of id? If the valid range would be 0 to 2^32-1 instead, CPUs could use
>> simple registers to store the id.
>
> Modern CPUs have 64 bit register banks. The upper bound is carefully chosen to be representable in 64 bit integers and IEEE doubles, and at the same time allow to make the probability of collisions practically zero.

As you might be able to imagine already I come from the embedded world, where 32bit is already a large register. For modern desktop PCs it's for sure nothing to think about. Do you think this additional 21bit will really reduce the probability of collision for real world applications? IMHO 32bit should be enough. If you create an event every millisecond, your application needs to run more than 3 years to reuse an id (if they are dumb incrementing numbers).

>> Additionally I'd merge "Arguments|list, ArgumentsKw|dict" to
>> "Arguments|any", because this looks very python-ish to me and WAMP is
>> not python only. Providing a documentation for the WAMP->Python mapping
>> would be better IMHO.
>
> No, please see here: https://github.com/tavendo/WAMP/issues/21
>
> In fact, WAMPv2 allows positional and keyword based call results, which isn't even in Python, but for example in PostgreSQL and Oracle.
>
> And I was first wary of this, but beatgammit convinced me that we should take the "superset of all" approach in WAMPv2. With WAMPv1, we tool the "smallest common denominator" (positional args, 1 result) approach.
>
> But, regarding arguments (and in general, all application payloads), there is 1 major outstanding question:
>
> All application payloads are now at the end within the lists forming WAMP messages. The idea is, that Dealers and Brokers actually don't have any need to parse or touch those app payloads, and might skip parsing altogether.
>
> Taking this idea further, I was thinking of making all app payloads just strings (JSON) or binaries (MsgPack). So Dealers/Brokers can be even more efficient and _probably_ we could even allow to transmit end-to-end encrypted payloads.
>
> Only the Subscriber/Publisher and Caller/Callee needs to perform a 2nd parse on the WAMP level payload to get at the actual payload.
>
> I would highly appreciate any input on this!

At which level do you see WAMP? Do you want to specify how functions have to look like in the protocol, or isn't it something for the application which builds on WAMP? If you go the first way then the spec should include the mapping for all programming languages, which might be out of scope.
Is there an advantage of having a defined number of elements in an message? If not, we can use
[CALL, Request|id, Options|dict, Procedure|uri, ...]
[CALL_RESULT, CALL.Request|id, ...]
with an variable length payload. This would reduce the size for simple calls and allow any combination for the implementation of applications. For the other case I'd suggest
[CALL, Request|id, Options|dict, Procedure|uri, Arguments|any]
[CALL_RESULT, CALL.Request|id, INVOCATION_RESULT.Result|any]
where the application has to encode everything into the any object/list/string.

What about allowing an optional payload (maybe with a maximum count?) for all messages? E.g.:
[SUBSCRIBE, Request|id, Options|dict, Topic|uri, ...]
[PUBLISH, Request|id, Options|dict, Topic|uri, ...]
[EVENT, SUBSCRIBED.Subscription|id, PUBLISHED.Publication|id, Details|dict, PUBLISH.Topic|uri, PUBLISH....]

Please don't remove the possibility for objects in the payload, since encoding everything as a string doesn't seam like a clean solution for me. If some application wants to do that an payload type of any would allow that anyway.

>> Do you accept pull request for changes in the spec?
>
> Yes. It would be good to have small PRs though (eg branch the spec per issue/feature you would like to get merged). Git doesn't cope well with cherry picking ..

I'll try my best. :-)

Two additional points I forgot in my last mail:
1) Can we introduce the requirement that all clients get messages in the exactly same order, when they sent the same messages to the broker:
E.g:
C->S: [SUBSCRIBE, 1, {}, "topic1"]
C->S: [SUBSCRIBE, 2, {}, "topic2"]
S->C: [SUBSCRIBED, 1]
S->C: [SUBSCRIBED, 2]
S->C: [EVENT, ..., "topic1", "A"]
S->C: [EVENT, ..., "topic2", null]
S->C: [EVENT, ..., "topic1", "B"]
can be also;
C->S: [SUBSCRIBE, 1, {}, "topic1"]
S->C: [SUBSCRIBED, 1]
C->S: [SUBSCRIBE, 2, {}, "topic2"]
S->C: [SUBSCRIBED, 2]
S->C: [EVENT, ..., "topic1", "A"]
S->C: [EVENT, ..., "topic2", null]
S->C: [EVENT, ..., "topic1", "B"]
but not:
C->S: [SUBSCRIBE, 1, {}, "topic1"]
C->S: [SUBSCRIBE, 2, {}, "topic2"]
S->C: [SUBSCRIBED, 2] <== Second subscribe returned first
S->C: [SUBSCRIBED, 1] <==
S->C: [EVENT, ..., "topic1", "A"]
S->C: [EVENT, ..., "topic2", null]
S->C: [EVENT, ..., "topic1", "B"]
and not:
C->S: [SUBSCRIBE, 1, {}, "topic1"]
C->S: [SUBSCRIBE, 2, {}, "topic2"]
S->C: [SUBSCRIBED, 1]
S->C: [SUBSCRIBED, 2]
S->C: [EVENT, ..., "topic1", "A"]
S->C: [EVENT, ..., "topic1", "B"] <== Second "topic1" before "topic2"
S->C: [EVENT, ..., "topic2", null] <==

2) Did you ever thought about high availability for WAMP? A client might want to connect to two broker at the same time to make sure it can get data even if one of them dies. I think about something like the following:
* Client connects to first broker B1 and does HELO
* Client connects to second broker B2 and does HELO
* Client tells B1 to "unify" with Session from B2 HELLO as active part
* Client tells B2 to "unify" with Session from B1 HELLO as passive part
* B1 und B2 are allowed to "unify" since they got matching Session ids from each other
* Client subscribes to "topic1" at B1 (since he is the active part)
* B1 tells B2 that "topic1" has been subscribed
* someone publishes "topic1"
* B1 sends "topic1" to client (since he is the active part)
* B2 ignores "topic1" (since he is the passive part)
* B1 dies
* Client recognizes dead B1
* Client switches B2 from passive to active
* someone publishes "topic1"
* B2 sends "topic1" to client (since he is the active part now)

Is that something you might consider to be specified directly at WAMP or is it overly application specific?

> Thanks!! And Merry Christmas;)

Thanks for the good basis to build upon. And Merry Christmas back, but maybe already too late. ;-)

-- Patrick

Tobias Oberstein

unread,
Dec 27, 2013, 6:50:25 AM12/27/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi Patrick,

I'll answer in 2 parts .. mail is long enough already;)

Am 25.12.2013 03:08, schrieb Patrick Gansterer:

>> Brokers are unaware of a "path" attribute, whether that is in SUBSCRIBE.Options or a new SUBSCRIBE.Args element.
>>
>> If you come up with a Broker that processes events based on that custom behavior, you won't be able to use any other WAMPv2 broker.
>>
>> In general, the idea is to specify core routing behavior for WAMPv2 so that Broker and Dealer implementations are interchangable.

I think we should make this more explicit in the spec, since it's quite
an important point: https://github.com/tavendo/WAMP/issues/32

>
> Ok, so I need to add additional requirements. ;-) I must be able to add additional "how to subscribe" stuff into the SUBSCRIBE. E.g. a maximum update rate, or filter for a specific user. Encoding this into the topic isn't a clean solution for me. Extending the options with all possible stuff, so that the broker is able to handle it, wouldn't work with very specific stuff.
> This use-case will only work without a generic broker, so every subscriber has a publisher (which is then the WebSocket server directly). Do you have any idea for a clean solution where we can skip the broker and have a 1:1 matching for subscribers to be able to handle very specific subscribe requirements without violating the specification?

This is quite application specific behavior. I think it would be crazy
to attempt to write a spec that defines generic Brokers which can cope
with above things and more.

For your specific scenario: why not have a _Callee_ endpoint

com.paroga.setnotify

where you can provide any search/filter conditions you like in Args.

Subscribers subscribe to the "generic" topic

com.paroga.onnotify

but your backend (the Publisher) can restrict publications using

PUBLISH.Options.eligible

to the exact set of desired recipients determined based on the
search/filter conditions originally supplied to "com.parago.setnotify"

The Callee behind "com.paroga.setnotify" would need to identify the
session ID of the Caller to manage the receivers. This is missing (we
only have it for PubSub), but is natural to add:

https://github.com/tavendo/WAMP/issues/31

Another approach would be to use "ephemeral topics":

com.paroga.setnotify

might return an ephemeral topic URI to the Caller based on the specific
search/filter conditions requested:

com.paroga.onnotify.e124

Two Callers wishing to get notified based on the same set of
search/filter conditions would get handed out the same ephemeral topic.

>> In case the Subscription/Registration is using "match" != "exact", the fields would be filled.
>>
>> My vote is +1 for this optimization (since it is also for a probably very frequent case):
>> https://github.com/tavendo/WAMP/issues/29
>
> Cool!

Note that there is a downside also:
https://github.com/tavendo/WAMP/issues/29#issuecomment-31256435

>> There has been a longer discussion here:
>> https://github.com/tavendo/WAMP/issues/8
>>
>> I don't think PREFIX or ALIAS brings any value, but introduces multiple problems.
>>
>> If you worry about wire level traffic volume: WebSocket has Deflate compression. And this will compress not only URIs in WAMP messages, but also URIs (and strings in general) within application payloads.
>
> It's not only about the traffic, it's also about the CPU performance. Believe it or not, but I had an setup where the string lookup was an important factor to performance: Imaging the filesystem server from above, which sends an high amount of com.paroga.filechange events. The client needs to lookup the "client dispatch function" in a map (e.g. {"com.paroga.filechange":function(){}, "com.paroga.otherstuff":function(){}}). The length of the string made a noticeable difference.

Yes, I agree, which is one reason why the new SUBSCRIBE/EVENT scheme
that is based on Subscription|id is better (the other major reason being
that it actually works with patterns also).

> If you now suggest to encode the path in the topic too, then the client dispatching gets even much more complicated.

The lookup for WAMPv2 within the Subscriber implementation is probably
more something like Subscription|id => Event Handler

Subscription|id is an integer that will might get hashed for the lookup
in a hash map. Since the integer is fixed length (64/53 bit), the
hashing is constant time, independent of Topic|uri length.

> Is subscribing to an metaevent always possible? Any security concerns about exposing this information?

The authorization scheme for metaevents is - like for "normal"
subcriptions - under Broker policy. This is one area where Broker
implementations might differ: one Broker might plainly allow everything,
another one might implement a fine-grained authorization regime for
subscribing (normal and meta) and publishing.

This should all be better explained / discussed in the spec
https://github.com/tavendo/WAMP/issues/30

END OF PART 1

Tobias

Patrick Gansterer

unread,
Dec 27, 2013, 7:27:35 AM12/27/13
to wam...@googlegroups.com, autob...@googlegroups.com
Hi Tobias,

On 27.12.2013, at 12:50, Tobias Oberstein <tobias.o...@gmail.com> wrote:
>>> Brokers are unaware of a "path" attribute, whether that is in SUBSCRIBE.Options or a new SUBSCRIBE.Args element.
>>>
>>> If you come up with a Broker that processes events based on that custom behavior, you won't be able to use any other WAMPv2 broker.
>>>
>>> In general, the idea is to specify core routing behavior for WAMPv2 so that Broker and Dealer implementations are interchangable.
>
> I think we should make this more explicit in the spec, since it's quite an important point: https://github.com/tavendo/WAMP/issues/32

What about adding an "broker/dealer-less mode"? Or is that you consider completely out of scope?

>> Ok, so I need to add additional requirements. ;-) I must be able to add additional "how to subscribe" stuff into the SUBSCRIBE. E.g. a maximum update rate, or filter for a specific user. Encoding this into the topic isn't a clean solution for me. Extending the options with all possible stuff, so that the broker is able to handle it, wouldn't work with very specific stuff.
>> This use-case will only work without a generic broker, so every subscriber has a publisher (which is then the WebSocket server directly). Do you have any idea for a clean solution where we can skip the broker and have a 1:1 matching for subscribers to be able to handle very specific subscribe requirements without violating the specification?
>
> This is quite application specific behavior. I think it would be crazy to attempt to write a spec that defines generic Brokers which can cope with above things and more.
>
> For your specific scenario: why not have a _Callee_ endpoint
>
> com.paroga.setnotify
>
> where you can provide any search/filter conditions you like in Args.
>
> Subscribers subscribe to the "generic" topic
>
> com.paroga.onnotify
>
> but your backend (the Publisher) can restrict publications using
>
> PUBLISH.Options.eligible
>
> to the exact set of desired recipients determined based on the search/filter conditions originally supplied to "com.parago.setnotify"
>
> The Callee behind "com.paroga.setnotify" would need to identify the session ID of the Caller to manage the receivers. This is missing (we only have it for PubSub), but is natural to add:
>
> https://github.com/tavendo/WAMP/issues/31
>
> Another approach would be to use "ephemeral topics":
>
> com.paroga.setnotify
>
> might return an ephemeral topic URI to the Caller based on the specific search/filter conditions requested:
>
> com.paroga.onnotify.e124
>
> Two Callers wishing to get notified based on the same set of search/filter conditions would get handed out the same ephemeral topic.

One problem with your solution is that i requires an additional ping-pong wit server. I have the requirement that subscribing to a few hundred event sources is fast (and easy). So I need something like the following:
[MULTI_SUBSCRIBE, id, {}, "com.paroga.filechage", [{path:"path1", delay:123}, {path:"path2"}, {path:"path3", cool:true}, ...]]
[MULTI_SUBSCRIBED, id, [{subscriptionid:11}, {subscriptionid:22}, {error:"invalid path"}, ...]],
[EVENT, 11, {content:"new value for path1"}]
[EVENT, 22, {content:"new value for path2"}]

>> Is subscribing to an metaevent always possible? Any security concerns about exposing this information?
>
> The authorization scheme for metaevents is - like for "normal" subcriptions - under Broker policy. This is one area where Broker implementations might differ: one Broker might plainly allow everything, another one might implement a fine-grained authorization regime for subscribing (normal and meta) and publishing.

My point was about the error handling: What if metaevent "meta1" is allowed and "meta2" is not? How you you report that to the user? Does the SUBSCRIBE fail with SUBSCRIBE_ERROR even if subscribing to the topic alone would work?

-- Patrick

Tobias Oberstein

unread,
Dec 27, 2013, 7:51:56 AM12/27/13
to wam...@googlegroups.com, autob...@googlegroups.com
>>> Do you think PUBLISH.Options.rkey|string and PUBLISH.Options.nkey|string
>>> really belong into this spec? They seam quite specialized to me.
>>
>> They are part of the call/event routing flexibility that WAMP provides - so yes, I think they are core.
>
> Can you describe your use-case a little bit more? Routing flexibility is for sure something I agree with, but I didn't get the connection between EVENT and CALL. "nkey" is only mentioned for the EVENT message. What's the exact difference between a peer, node and resource?
>


>> Modern CPUs have 64 bit register banks. The upper bound is carefully chosen to be representable in 64 bit integers and IEEE doubles, and at the same time allow to make the probability of collisions practically zero.
>
> As you might be able to imagine already I come from the embedded world, where 32bit is already a large register. For modern desktop PCs it's for sure nothing to think about. Do you think this additional 21bit will really reduce the probability of collision for real world applications? IMHO 32bit should be enough. If you create an event every millisecond, your application needs to run more than 3 years to reuse an id (if they are dumb incrementing numbers).
>

WAMP should work for distributed applications with large numbers of
devices (IoT/Mobile) and clustered/federated networks of Brokers and
Dealers. It is important that IDs can be generated locally, but
nevertheless have practically zero collision probability.

If you take 10 million peers, each drawing 1000 IDs/sec randomly, that's
8.6 x 10^14. On the other hand, 2^53 is roughly 9 x 10^15.

That being said, I think, given the overall processing needed for WAMP,
reducing IDs to 32 bit isn't really relevant. WAMPv1 is using long,
rnadom strings, and having short integer IDs is already a major
"optimization" over the current state.

> At which level do you see WAMP? Do you want to specify how functions have to look like in the protocol, or isn't it something for the application which builds on WAMP? If you go the first way then the spec should include the mapping for all programming languages, which might be out of scope.

WAMP on the wire follows a "superset of all" approach to call arguments
and results (and other application payloads).

It's the task of a WAMP implementation to map this ultimate flexility
into the host language.

E.g. JavaScript has no positional or keyword function returns.
AutobahnJS will probably then map results which are not strictly 1
positional result into a wrapper object.

> Is there an advantage of having a defined number of elements in an message? If not, we can use
> [CALL, Request|id, Options|dict, Procedure|uri, ...]
> [CALL_RESULT, CALL.Request|id, ...]

No, we had this in WAMPv1, and it proved to be problematic. WAMPv2 has
non-polymorphic messages by design. Every message type has a fixed
number of WAMP-level elements.

> with an variable length payload. This would reduce the size for simple calls and allow any combination for the implementation of applications. For the other case I'd suggest
> [CALL, Request|id, Options|dict, Procedure|uri, Arguments|any]
> [CALL_RESULT, CALL.Request|id, INVOCATION_RESULT.Result|any]
> where the application has to encode everything into the any object/list/string.
>
> What about allowing an optional payload (maybe with a maximum count?) for all messages? E.g.:
> [SUBSCRIBE, Request|id, Options|dict, Topic|uri, ...]
> [PUBLISH, Request|id, Options|dict, Topic|uri, ...]
> [EVENT, SUBSCRIBED.Subscription|id, PUBLISHED.Publication|id, Details|dict, PUBLISH.Topic|uri, PUBLISH....]

Again, no, I don't think that's an improvement. No polymorphic messages
please ..

>
> Please don't remove the possibility for objects in the payload, since encoding everything as a string doesn't seam like a clean solution for me. If some application wants to do that an payload type of any would allow that anyway.

What I meant is, instead of

CALL.Args|list = [1, "foo", {"a": 23}]

have

CALL.Args|string = JSON.stringify([1, "foo", {"a": 23}])

Since a Dealer does not need to analyze or change it's behavior based on
CALL.Args, it - in principle - has no need to parse CALL.Args.

Thinking about it, I forgot the "elephant in the room": WAMPv2 supports
different serializations, and a Broker/Dealer will hence need to
translate between different serializations. And thus _does_ have a need
to parse CALL.Args.
Sorry, I'm not sure I can follow above. What is "S" and "C"?

Can we use these shortcuts?

Bn: Broker n
Sn: Subscriber n
Pn: Publisher n
Dn: Dealer n
Cn: Caller n
En: Callee ("Endpoint") n

Regarding publishing, the order guarantee is as follows:

If S1 is subscribed to Topic 1 and Topic 2 and P1 publishes Event 1 to
Topic 1, and then Event 2 to Topic 2, then S1 will receive first Event 1
and then Event 2. This also holds for Topic 1 == Topic 2.

Further, if S1 subscribes to Topic 1, the SUBSCRIBED messages will be
sent by Broker before any EVENT for Topic 1.

But in general, SUBSCRIBE is asynchronous, and there is no guarantee on
order of return for multiple SUBSCRIBEs. The first SUBSCRIBE might
require the Broker to do a time-consuming lookup in some database,
whereas the second might be permissible immediately.

This clearly needs to be described in detail in the spec:
https://github.com/tavendo/WAMP/issues/33

> 2) Did you ever thought about high availability for WAMP? A client might want to connect to two broker at the same time to make sure it can get data even if one of them dies. I think about something like the following:
> * Client connects to first broker B1 and does HELO
> * Client connects to second broker B2 and does HELO
> * Client tells B1 to "unify" with Session from B2 HELLO as active part
> * Client tells B2 to "unify" with Session from B1 HELLO as passive part
> * B1 und B2 are allowed to "unify" since they got matching Session ids from each other
> * Client subscribes to "topic1" at B1 (since he is the active part)
> * B1 tells B2 that "topic1" has been subscribed
> * someone publishes "topic1"
> * B1 sends "topic1" to client (since he is the active part)
> * B2 ignores "topic1" (since he is the passive part)
> * B1 dies
> * Client recognizes dead B1
> * Client switches B2 from passive to active
> * someone publishes "topic1"
> * B2 sends "topic1" to client (since he is the active part now)
>
> Is that something you might consider to be specified directly at WAMP or is it overly application specific?

In fact, scale-out and high-availability scenarios is something that
should be covered by WAMP. But on a Broker-Dealer level. Mere
Callers/Callees/Subscribers/Publishers should not be concerned about that.

In fact, what is missing is a design for Broker-Broker and Dealer-Dealer
interactions.

Tobias Oberstein

unread,
Dec 27, 2013, 8:03:13 AM12/27/13
to autob...@googlegroups.com, wam...@googlegroups.com
Hi Patrick,

> One problem with your solution is that i requires an additional ping-pong wit server. I have the requirement that subscribing to a few hundred event sources is fast (and easy). So I need something like the following:
> [MULTI_SUBSCRIBE, id, {}, "com.paroga.filechage", [{path:"path1", delay:123}, {path:"path2"}, {path:"path3", cool:true}, ...]]
> [MULTI_SUBSCRIBED, id, [{subscriptionid:11}, {subscriptionid:22}, {error:"invalid path"}, ...]],
> [EVENT, 11, {content:"new value for path1"}]
> [EVENT, 22, {content:"new value for path2"}]

Not sure if I understand: WAMP requests can be pipelined .. you don't
need to wait for the n-th SUBSCRIBE to return befor you request the
n+1-th one.

You can send out 100 SUBSCRIBEs before even the first one returns ..
same with CALLs.

>>> Is subscribing to an metaevent always possible? Any security concerns about exposing this information?
>>
>> The authorization scheme for metaevents is - like for "normal" subcriptions - under Broker policy. This is one area where Broker implementations might differ: one Broker might plainly allow everything, another one might implement a fine-grained authorization regime for subscribing (normal and meta) and publishing.
>
> My point was about the error handling: What if metaevent "meta1" is allowed and "meta2" is not? How you you report that to the user? Does the SUBSCRIBE fail with SUBSCRIBE_ERROR even if subscribing to the topic alone would work?
>

The Broker must fail the request if it can't fulfill it completely, and
hence the SUBSCRIBE will fail if "meta2" is disallowed, even if the
request would have succeeded if only "topic1" and "meta1" would have
been requested.

/Tobias

Patrick Gansterer

unread,
Dec 27, 2013, 8:35:46 AM12/27/13
to wam...@googlegroups.com, autob...@googlegroups.com

On 27.12.2013, at 13:51, Tobias Oberstein <tobias.o...@gmail.com> wrote:

>>> Modern CPUs have 64 bit register banks. The upper bound is carefully chosen to be representable in 64 bit integers and IEEE doubles, and at the same time allow to make the probability of collisions practically zero.
>>
>> As you might be able to imagine already I come from the embedded world, where 32bit is already a large register. For modern desktop PCs it's for sure nothing to think about. Do you think this additional 21bit will really reduce the probability of collision for real world applications? IMHO 32bit should be enough. If you create an event every millisecond, your application needs to run more than 3 years to reuse an id (if they are dumb incrementing numbers).
>>
>
> WAMP should work for distributed applications with large numbers of devices (IoT/Mobile) and clustered/federated networks of Brokers and Dealers. It is important that IDs can be generated locally, but nevertheless have practically zero collision probability.
>
> If you take 10 million peers, each drawing 1000 IDs/sec randomly, that's 8.6 x 10^14. On the other hand, 2^53 is roughly 9 x 10^15.
>
> That being said, I think, given the overall processing needed for WAMP, reducing IDs to 32 bit isn't really relevant. WAMPv1 is using long, rnadom strings, and having short integer IDs is already a major "optimization" over the current state.

In an distributed environment this makes 100% sense. In a simple client/server environment with incrementing ids 32bit should be enough. But it's not a big issue anyway. Just thought.

>> At which level do you see WAMP? Do you want to specify how functions have to look like in the protocol, or isn't it something for the application which builds on WAMP? If you go the first way then the spec should include the mapping for all programming languages, which might be out of scope.
>
> WAMP on the wire follows a "superset of all" approach to call arguments and results (and other application payloads).
>
> It's the task of a WAMP implementation to map this ultimate flexility into the host language.

IMHO WAMP will be used to 95% with callers programmed in JS. There a CALL would perfectly match the behavior of Promise and is quite clear to understand, but I think you made your decision already...

> E.g. JavaScript has no positional or keyword function returns. AutobahnJS will probably then map results which are not strictly 1 positional result into a wrapper object.

To be clean you have to wrap it all the time. And

doWAMPstuff().then(function(res){
alert(res[0]);
})

to get the result for some node.js function like

function() {
return "message for alert()";
}

seams a little bit strange to me.

>
>> Is there an advantage of having a defined number of elements in an message? If not, we can use
>> [CALL, Request|id, Options|dict, Procedure|uri, ...]
>> [CALL_RESULT, CALL.Request|id, ...]
>
> No, we had this in WAMPv1, and it proved to be problematic. WAMPv2 has non-polymorphic messages by design. Every message type has a fixed number of WAMP-level elements.

What was problematic?

>> with an variable length payload. This would reduce the size for simple calls and allow any combination for the implementation of applications. For the other case I'd suggest
>> [CALL, Request|id, Options|dict, Procedure|uri, Arguments|any]
>> [CALL_RESULT, CALL.Request|id, INVOCATION_RESULT.Result|any]
>> where the application has to encode everything into the any object/list/string.
>>
>> What about allowing an optional payload (maybe with a maximum count?) for all messages? E.g.:
>> [SUBSCRIBE, Request|id, Options|dict, Topic|uri, ...]
>> [PUBLISH, Request|id, Options|dict, Topic|uri, ...]
>> [EVENT, SUBSCRIBED.Subscription|id, PUBLISHED.Publication|id, Details|dict, PUBLISH.Topic|uri, PUBLISH....]
>
> Again, no, I don't think that's an improvement. No polymorphic messages please ..
>
>>
>> Please don't remove the possibility for objects in the payload, since encoding everything as a string doesn't seam like a clean solution for me. If some application wants to do that an payload type of any would allow that anyway.
>
> What I meant is, instead of
>
> CALL.Args|list = [1, "foo", {"a": 23}]
>
> have
>
> CALL.Args|string = JSON.stringify([1, "foo", {"a": 23}])
>
> Since a Dealer does not need to analyze or change it's behavior based on CALL.Args, it - in principle - has no need to parse CALL.Args.
>
> Thinking about it, I forgot the "elephant in the room": WAMPv2 supports different serializations, and a Broker/Dealer will hence need to translate between different serializations. And thus _does_ have a need to parse CALL.Args.

So no option to send payload as MsgPack (at least without additional binary-to-string conversion like base64)?
> Sorry, I'm not sure I can follow above. What is "S" and "C"?

Server/Client, or better Broker/Subscriber

> Can we use these shortcuts?
>
> Bn: Broker n
> Sn: Subscriber n
> Pn: Publisher n
> Dn: Dealer n
> Cn: Caller n
> En: Callee ("Endpoint") n
>
> Regarding publishing, the order guarantee is as follows:
>
> If S1 is subscribed to Topic 1 and Topic 2 and P1 publishes Event 1 to Topic 1, and then Event 2 to Topic 2, then S1 will receive first Event 1 and then Event 2. This also holds for Topic 1 == Topic 2.
>
> Further, if S1 subscribes to Topic 1, the SUBSCRIBED messages will be sent by Broker before any EVENT for Topic 1.
>
> But in general, SUBSCRIBE is asynchronous, and there is no guarantee on order of return for multiple SUBSCRIBEs. The first SUBSCRIBE might require the Broker to do a time-consuming lookup in some database, whereas the second might be permissible immediately.

For one Subscriber it's clear that it can be completely asynchronous. I was thinking about two different subscriber sending the same SUBSCRIBE messages to the broker, but I don't think it's possible to guarantee in the distribute environment.


On 27.12.2013, at 14:03, Tobias Oberstein <tobias.o...@gmail.com> wrote:

>> One problem with your solution is that i requires an additional ping-pong wit server. I have the requirement that subscribing to a few hundred event sources is fast (and easy). So I need something like the following:
>> [MULTI_SUBSCRIBE, id, {}, "com.paroga.filechage", [{path:"path1", delay:123}, {path:"path2"}, {path:"path3", cool:true}, ...]]
>> [MULTI_SUBSCRIBED, id, [{subscriptionid:11}, {subscriptionid:22}, {error:"invalid path"}, ...]],
>> [EVENT, 11, {content:"new value for path1"}]
>> [EVENT, 22, {content:"new value for path2"}]
>
> Not sure if I understand: WAMP requests can be pipelined .. you don't need to wait for the n-th SUBSCRIBE to return befor you request the n+1-th one.
>
> You can send out 100 SUBSCRIBEs before even the first one returns .. same with CALLs.

I ment your "ephemeral topics", where com.paroga.setnotify returns the topic com.paroga.onnotify.e124. In that scenario SUBSCRIBE needs to wait for CALL_RESULT.


Did you thought about any flow control in WAMP?

-- Patrick

Tobias Oberstein

unread,
Dec 27, 2013, 12:26:49 PM12/27/13
to wam...@googlegroups.com, autob...@googlegroups.com
>>> At which level do you see WAMP? Do you want to specify how functions have to look like in the protocol, or isn't it something for the application which builds on WAMP? If you go the first way then the spec should include the mapping for all programming languages, which might be out of scope.
>>
>> WAMP on the wire follows a "superset of all" approach to call arguments and results (and other application payloads).
>>
>> It's the task of a WAMP implementation to map this ultimate flexility into the host language.
>
> IMHO WAMP will be used to 95% with callers programmed in JS. There a CALL would perfectly match the behavior of Promise and is quite clear to understand, but I think you made your decision already...

AutobahnJS does already use promises for RPCs.

Regarding decision: the thing is, this was discussed in length in
https://github.com/tavendo/WAMP/issues/21

and in fact WAMPv1 did follow the "minimal" approach of
positional-args-only and exactly 1 positional result. Sure, there is a
price for flexibility on wire-level: the need to map stuff into host
languages within WAMP implementations.

>
>> E.g. JavaScript has no positional or keyword function returns. AutobahnJS will probably then map results which are not strictly 1 positional result into a wrapper object.
>
> To be clean you have to wrap it all the time. And

Why? AutobahnJS might expose CALL results with only 1 positional result
directly ..

>> No, we had this in WAMPv1, and it proved to be problematic. WAMPv2 has non-polymorphic messages by design. Every message type has a fixed number of WAMP-level elements.
>
> What was problematic?

Things like non-extensibility, more complex parsing, no ability to have
kw args.

>> Thinking about it, I forgot the "elephant in the room": WAMPv2 supports different serializations, and a Broker/Dealer will hence need to translate between different serializations. And thus _does_ have a need to parse CALL.Args.
>
> So no option to send payload as MsgPack (at least without additional binary-to-string conversion like base64)?

No, what I meant is that Dealers/Brokers should translate between
different serialization formats, and hence do have a need for parsing
application payload also.

>> Regarding publishing, the order guarantee is as follows:
>>
>> If S1 is subscribed to Topic 1 and Topic 2 and P1 publishes Event 1 to Topic 1, and then Event 2 to Topic 2, then S1 will receive first Event 1 and then Event 2. This also holds for Topic 1 == Topic 2.
>>
>> Further, if S1 subscribes to Topic 1, the SUBSCRIBED messages will be sent by Broker before any EVENT for Topic 1.
>>
>> But in general, SUBSCRIBE is asynchronous, and there is no guarantee on order of return for multiple SUBSCRIBEs. The first SUBSCRIBE might require the Broker to do a time-consuming lookup in some database, whereas the second might be permissible immediately.
>
> For one Subscriber it's clear that it can be completely asynchronous. I was thinking about two different subscriber sending the same SUBSCRIBE messages to the broker, but I don't think it's possible to guarantee in the distribute environment.

Yep. Due to unknown network delays, order guarantees can only be made in
relation to what is received by peer A, given stuff sent by a peer B ..

> I ment your "ephemeral topics", where com.paroga.setnotify returns the topic com.paroga.onnotify.e124. In that scenario SUBSCRIBE needs to wait for CALL_RESULT.

Yes, but you can already send the 100 calls to "com.paroga.setnotify"
pipelined. Each SUBSCRIBE can then only be done if the respective call
returns (asynchronously).

> Did you thought about any flow control in WAMP?

Not sure what you mean. Flow-control as in "slow TCP receiver, and fast
send" or such?

/Tobias

Jordi Mariné Fort

unread,
Jan 6, 2014, 12:39:37 PM1/6/14
to wam...@googlegroups.com, autob...@googlegroups.com
Two more suggestions for WAMP v2:

1)  "wamp.cra.request" procedure could use the "ArgumentsKw" parameter of the CALL message to receive the "auth_extra|dict" information
     (instead of sending the "auth_extra|dict" as a second value in the "Arguments|list")

2) I think the METAEVENT messages also needs a "Details|dict" parameter, to inform the "topic|uri" in case of pattern-based subscriptions.

E.g.:

Client 1 subscribes to all application topics (for monitoring purposes): 
[10, 912873614, {"match": "prefix"
, "metaonly": 1, "metatopics": ["wamp.metatopic.subscriber.add", "wamp.metatopic.subscriber.remove"] }, "com.myapp"]

Client 2 (with sessionId=
71254637) subscribes to a specific topic:
[10, 834834342, {"match": "exact" }, "com.myapp.topic1"]

When client 1 receives the "metaevent" message of the client 2 subscription:
[41, 5512315355, 51415664, "wamp.metatopic.subscriber.add", 71254637]

It doesn't know in which topic the client 2 has been subscribed.

Tobias Oberstein

unread,
Jan 6, 2014, 4:11:28 PM1/6/14
to wam...@googlegroups.com, autob...@googlegroups.com
Am 06.01.2014 18:39, schrieb Jordi Marin� Fort:
> Two more suggestions for WAMP v2:
>
> 1) "wamp.cra.request" procedure could use the "ArgumentsKw" parameter
> of the CALL message to receive the "auth_extra|dict" information
> (instead of sending the "auth_extra|dict" as a second value in the
> "Arguments|list")

Thats a good catch. Now that we have kwargs for RPCs, we should use it.

https://github.com/tavendo/WAMP/issues/34

>
> 2) I think the METAEVENT messages also needs a "Details|dict" parameter,
> to inform the "topic|uri" in case of pattern-based subscriptions.

METAEVENT contains SUBSCRIBED.Subscription|id already.

The METAEVENT is only generated upon other peers
subscribing/unsubscribing to/from that _same_ SUBSCRIBED.Subscription|id.

>
> E.g.:
>
> |Client 1 subscribes to all application topics (for monitoring purposes):
> [10, 912873614, {"match": "prefix"|||, "metaonly": 1|,|||"metatopics": ["wamp.metatopic.subscriber.add", "wamp.metatopic.subscriber.remove"]|}, "com.myapp"]
>
> Client 2 (with sessionId=|||71254637)|subscribes to a specific topic:||
> [10, 834834342, {"match": "exact" }, "com.myapp.topic1"]

This will result in a _different_ subscription that is unrelated to the
first.

An event _published_ to topic "com.myapp.topic1" will be received under
both subscriptions, the subscriptions are still separate. There is no
set semantics involved.

>
> When client 1 receives the "metaevent" message of the client 2 subscription:
> ||[41, 5512315355, 51415664, "wamp.metatopic.subscriber.add", 71254637]|
>
> |It doesn't know in which topic the client 2 has been subscribed.|
>

Patrick Gansterer

unread,
Jan 8, 2014, 5:50:05 AM1/8/14
to wam...@googlegroups.com, autob...@googlegroups.com

On 27.12.2013, at 18:26, Tobias Oberstein <tobias.o...@gmail.com> wrote:

>> Did you thought about any flow control in WAMP?
>
> Not sure what you mean. Flow-control as in "slow TCP receiver, and fast send" or such?

Yes exactly. E.g. Browser does a heavy 3D-Rendering for every received event. Is this in the scope of the specification?


Was there a reason why you choose exactly MsgPack in favor of BSON, BJSON and UBJSON?
It might be the compactest but did you considered the complexity for parsing? UBJSON is the only one which provides a 1:1 mapping between binary and JSON.

-- Patrick

Tobias Oberstein

unread,
Jan 8, 2014, 8:08:40 AM1/8/14
to wam...@googlegroups.com, autob...@googlegroups.com
Am 08.01.2014 11:50, schrieb Patrick Gansterer:
>
> On 27.12.2013, at 18:26, Tobias Oberstein <tobias.o...@gmail.com> wrote:
>
>>> Did you thought about any flow control in WAMP?
>>
>> Not sure what you mean. Flow-control as in "slow TCP receiver, and fast send" or such?
>
> Yes exactly. E.g. Browser does a heavy 3D-Rendering for every received event. Is this in the scope of the specification?

TCP flow control is handled at the TCP level by the networking stack in
the OS kernel. This is unrelated to WAMP. But I guess this is not what
you mean.

If you receive events, and your app cannot keep up processing, simply
skip processing of events in your app. You will continue to receive all
events, but only process a subset in your app.

If your downlink network connection cannot keep up with the volume of
events sent by the WAMP broker, then it's an implementation detail of
the WAMP broker how it behaves. It might just start dropping events (to
that single slow consumer). Or it might start buffering, slowing down
_all_ receivers. Or it might deny further publications to the respective
topic.

All above is not relevant to the WAMP protocol specficiation however. We
should probably mention the stuff above that in the spec nevertheless.

https://github.com/tavendo/WAMP/issues/36

>
>
> Was there a reason why you choose exactly MsgPack in favor of BSON, BJSON and UBJSON?

MsgPack is widely used and implemented, and the others don't seem to
provide any compelling features.

> It might be the compactest but did you considered the complexity for parsing? UBJSON is the only one which provides a 1:1 mapping between binary and JSON.

I have added a section about the conversions done for byte arrays:

https://github.com/tavendo/WAMP/tree/master/spec#conversion

This is a simple scheme that is minimally intrusive and provides
transparency (a 1:1 mapping) for byte arrays between JSON and MsgPack.

You can find example code here:

https://github.com/tavendo/WAMP/tree/master/spec#byte-array-conversion

Note that performance-wise, JSON is the fastest in browsers anyway since
it is natively implemented.

Also note that compression-wise, WebSocket compression
("permessage-deflate") with JSON is very compact (usually compacter than
uncompressed MsgPack).

/Tobias

>
> -- Patrick
>

Jordi Mariné Fort

unread,
Feb 5, 2014, 12:19:24 PM2/5/14
to wam...@googlegroups.com, autob...@googlegroups.com
Hi,

I have another question:
Which RPC should be used to retrieve all topic messages?
(maybe "wamp.topic.history.last" with "limit" = 0?)

Also, I would like to suggest a "Publish.Options.persist|bool" parameter in case the message should actually be persisted/retrieved from an history.
(or discard the message once it has been sent to all subscribers).


Tobias Oberstein

unread,
Feb 7, 2014, 11:33:02 AM2/7/14
to wam...@googlegroups.com, autob...@googlegroups.com
Hi Jordi,

> Which RPC should be used to retrieve all topic messages?
> (maybe "|wamp.topic.history.last" with "limit" = 0|?)

https://github.com/tavendo/WAMP/issues/42

> Also, I would like to suggest a "Publish.Options.persist|bool" parameter
> in case the message should actually be persisted/retrieved from an history.
> (or discard the message once it has been sent to all subscribers).

https://github.com/tavendo/WAMP/issues/43

Please comment on the issues (if you have comments .. probably about the
last).

Note that both features would be optional features that advanced brokers
MAY provide.

Cheers,
/Tobias
Reply all
Reply to author
Forward
0 new messages