Decoupling Payloads

[Emile Cormier]

Because the application payload for certain message types are encoded as a positional argument of the message, this makes is difficult for router implementations to avoid needlessly serializing/deserializing the application payload.

For example, if two peers both use JSON encoding, there's no need for the router to parse the application payload: it can just forward it as-is.

For routers to avoid serializing/deserializing the payload, the implementors either have to write their own codecs from scratch, or hope that the codec has hooks that allow the router to skip parsing/encoding the payload.

Has there been any discussion in the evolution of WAMP v2 regarding the decoupling of application payload from the "header" portions of WAMP messages?

[Tobias Oberstein]

Hi Emile,

yeah, I've been thinking about this in the past, but discarded the
approach of doing that at the WAMP message structure level, for various
reasons.

However, there is a feature in Crossbar.io/Autobahn (not yet spec'ed
properly, but there are some bits in the WAMP proto repo):

payload transparency

If you are using this on a session that uses a serializer that actually
allow to skip-parse (json is not one), eg msgpack, you can reap such
benefits (skip parsing the actualy app payload altogether).

let me just quickly dig out some links, but there is more (no time right
now). you can ping meejah or argonholm - I dont think anyone else has
knowledge of the details right now.

"payload transparency" is the base used in 3 other features of CB:

mqtt (passthrough mode):

https://github.com/crossbario/crossbar-examples/tree/master/mqtt/basic

custom payload codecs:

https://github.com/crossbario/crossbar-examples/tree/master/payloadcodec

e2e encryption:

https://github.com/crossbario/crossbar-examples/tree/master/encryption/cryptobox

best is to dig from there .. docs are .. sketchy;) no time

--

the wamp repo should have some bits in the issues/comments .. cannot
find it right away.

hope this helps!

Cheers,
/Tobias

[Emile Cormier]

This topic has entered my thoughts again. I just realized that an easy way to achieve payload opaqueness with JSON would be using line-delimited JSON: https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON . Instead of a JSON array, text-based WAMP messages would be either two lines, or N lines, with the last line containing the JSON payload.

In the two-line scheme, the first line would contain a JSON array with the command string ("CALL", "PUBLISH", etc), as well as the command options. The second line would be the JSON payload.

In the N-line scheme, each line would be an element of the JSON array from the original scheme, with the last being the JSON payload.

Alternatively, the JSON payload could be encoded as Base64, but that would add both a computational and size cost.

With encodings supporting binary data (such as MsgPack or CBOR), making the payload opaque is of course much easier.

In hindsight, WAMP should not care about positional or keyword arguments. It should be up to the application and client libraries to interpret payloads as either multiple arguments, or as a single monolithic message.

BTW, the feature you call "payload transparency" might be better described as "payload opaqueness". :-)

[Tobias Oberstein]

Hi Emile,

aah, right, this old mailing list we never migrated to
https://forum.crossbar.io/ ;)

Am 03.11.21 um 21:08 schrieb Emile Cormier:

> This topic has entered my thoughts again. I just realized that an easy
> way to achieve payload opaqueness with JSON would be using
> line-delimited JSON:
> https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON .

"the JSON format does not allow return and newline characters within
primitive values (in strings those must be escaped as \r and \n,
respectively)"

ok. this sounds like a big source of potential non-conformant/weird
implementations in JSON parsers in the wield.

also: only within primitive values.

that is, a single string literal within a serialized JSON must not
contain non-escaped line breaks, right?

> Instead of a JSON array, text-based WAMP messages would be either two
> lines, or N lines, with the last line containing the JSON payload.
>
> In the two-line scheme, the first line would contain a JSON array with
> the command string ("CALL", "PUBLISH", etc), as well as the command
> options. The second line would be the JSON payload.
>
> In the N-line scheme, each line would be an element of the JSON array
> from the original scheme, with the last being the JSON payload.
>
> Alternatively, the JSON payload could be encoded as Base64, but that
> would add both a computational and size cost.

I'm not sure what the goal is? I mean, rather than using the existing
mechanism of using strings with a beginning \0 and non-text binary after
that?

>
> With encodings supporting binary data (such as MsgPack or CBOR), making
> the payload opaque is of course much easier.

you can looks a measured performance numbers here:

https://crossbario.com/docs/benchmarks/serialization/index.html

>
> In hindsight, WAMP should not care about positional or keyword
> arguments. It should be up to the application and client libraries to
> interpret payloads as either multiple arguments, or as a single
> monolithic message.

doing so removes the ability of a WAMP router to transparently translate
between different serializers used by different clients - router side.

but I would agree, the other mode ("payload opaqueness") should probably
be sth we should have in the WAMP BP. it's not sadly

>
> BTW, the feature you call "payload transparency" might be better
> described as "payload opaqueness". :-)

ok! thanks, I'm not a native speaker;)

> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com
> <mailto:wampws+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/wampws/0d3f8747-7a7e-4ac1-aa96-b4d94e328dcdn%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/0d3f8747-7a7e-4ac1-aa96-b4d94e328dcdn%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Emile Cormier]

On Thursday, November 4, 2021 at 12:41:14 AM UTC-3 tobias wrote:

Hi Emile,

aah, right, this old mailing list we never migrated to
https://forum.crossbar.io/ ;)

I didn't want to spam the WAMP protocol issue tracker with a half-baked idea, and I wasn't aware of any other place to discuss this idea.

 

"the JSON format does not allow return and newline characters within
primitive values (in strings those must be escaped as \r and \n,
respectively)"

ok. this sounds like a big source of potential non-conformant/weird
implementations in JSON parsers in the wield.

A JSON parser would not be used for interpreting my proposed newline-delimited WAMP message frame structure. A simple parser that extracts lines separated by newlines is all that would be needed, and is easy to write by hand. In C++, there wouldn't even be string copying involved, and I could just produce a cheap string_view for each line of the message frame.

For example:
[48, 7814135, {}, "com.myapp.echo"]

["Hello, world!"]

{}

where:

Line 1 is the command (in this case CALL), sequence ID, and options. This is parsed as JSON by the router for processing after stripping the trailing newline.

Line 2 contains the positional arguments which can be opaque to the router. That is, the router does not need to parse this as JSON and can pass it on wholesale to the callee after stripping the trailing newline.

Line 3 contains the keyword arguments which can also be opaque to the router.

If you meant that there are JSON encoders in the wild that don't escape newline characters, well then they are not JSON compliant, and I would not want to support them.

 

also: only within primitive values.

that is, a single string literal within a serialized JSON must not
contain non-escaped line breaks, right?

That's correct. Because the newline character can never appear within valid JSON, it can therefore be used as a delimiter to separate multiple JSON records. I've used this to great effect for my own hand-written key-value store, where records are JSON objects separated by newlines.

 

> Alternatively, the JSON payload could be encoded as Base64, but that
> would add both a computational and size cost.

I'm not sure what the goal is? I mean, rather than using the existing
mechanism of using strings with a beginning \0 and non-text binary after
that?

The idea is to skip parsing of the JSON payload by routers. I'd prefer the newline-delimited JSON scheme rather than Base64 encoding the JSON payload.

 

>
> With encodings supporting binary data (such as MsgPack or CBOR), making
> the payload opaque is of course much easier.

you can looks a measured performance numbers here:

https://crossbario.com/docs/benchmarks/serialization/index.html

>
> In hindsight, WAMP should not care about positional or keyword
> arguments. It should be up to the application and client libraries to
> interpret payloads as either multiple arguments, or as a single
> monolithic message.

doing so removes the ability of a WAMP router to transparently translate
between different serializers used by different clients - router side.

Treating the payload as a monolithic entity does not prevent routers from transcoding between different serialization formats. By "entity" I mean one of:

- boolean

- number

- text string

- byte array

- array

- object/map

 

For example, a router receives a JSON array from the caller as the monolithic CALL payload, and can convert it to the equivalent CBOR array to forward to the callee. The application protocol may decide that the array elements correspond to positional arguments, but this is a detail that the router doesn't need to care about. As a convenience to the application writer, the client libraries can take care of unpacking the positional arguments of the array before invoking the RPC handler.

but I would agree, the other mode ("payload opaqueness") should probably
be sth we should have in the WAMP BP. it's not sadly

Are there are Crossbar docs about this feature? The only thing I could find was https://crossbar.io/docs/Cryptobox-Payload-Encryption/

Cheers,

Emile

[Tobias Oberstein]

Hi Emile,

> I didn't want to spam the WAMP protocol issue tracker with a half-baked
> idea, and I wasn't aware of any other place to discuss this idea.

no worries filing issues, even to discuss "half baked stuff"! actually,
if you are serious about this, pls do file an issue, as there are more
eyes from other implementors, great to gather feedback.

it would be great if the issue actually had

1. perceived problems with existing protocol options
2. proposed new solution / protocol option
3. claimed advantages compared to status quo

from your point of view.

> but I would agree, the other mode ("payload opaqueness") should
> probably
> be sth we should have in the WAMP BP. it's not sadly
>
>
> Are there are Crossbar docs about this feature? The only thing I could
> find was https://crossbar.io/docs/Cryptobox-Payload-Encryption/

ok, I had a relook at the actual code.

in autobahn-python, if you set a "payload codec" using
set_payload_codec, this will make the client call into the codec to
serialize "args/kwargs" into "payload" - a bytes string.

if such a WAMP message with a non-empty payload (rather than
args/kwargs) is later touched in the client to serialize the
non-app-payload WAMP message

https://github.com/crossbario/autobahn-python/blob/a35f22eeaafca7568f1deb35c4a1b82ae78f77d4/autobahn/wamp/message.py#L4187

it insert payload into the rest of the wamp message - which is
serialized itself according to the serializer negotiated (which can be
json, cbor, ..)

so instead of serializing

[Call.MESSAGE_TYPE, self.request, options, self.procedure, self.args,
self.kwargs]

it serializes

[Call.MESSAGE_TYPE, self.request, options, self.procedure, self.payload]

and self.payload isn't touched by the negotiated serializer (since its
already bytes from codec.encode)

here are a couple of interesting locations:

https://github.com/crossbario/autobahn-python/blob/a35f22eeaafca7568f1deb35c4a1b82ae78f77d4/autobahn/wamp/message.py#L121

https://github.com/crossbario/autobahn-python/blob/a35f22eeaafca7568f1deb35c4a1b82ae78f77d4/autobahn/wamp/message.py#L3942

and here are 2 examples in crossbar which _use_ (build on) payload
transparency mode:

https://github.com/crossbario/crossbar-examples/tree/master/payloadcodec
https://github.com/crossbario/crossbar-examples/tree/master/mqtt/basic/passthrough

---

so given ^, I now think what you propose is a wamp serialization format,
and the payload-transparency cannot emulate that, as it only is
concerned with, well, the app payload.

Cheers,
/Tobias

[Emile Cormier]

Yes, what I propose would effectively be a new WAMP serialization format, say "wamp.2.ndjson", where NDJSON is according to this spec: http://ndjson.org/

I just remembered that newlines are permitted as white space characters within JSON outside of strings. But JSON encoders don't usually produce any whitespace when not in "prettify" mode.

[Tobias Oberstein]

Omer has setup a chat room for WAMP: https://discord.gg/ZfkUCxQB

> In the N-line scheme, each line would be an element of the JSON array
from the original scheme, with the last being the JSON payload.

how would the last array item (== line) in a WAMP msg be identified?

one option: end each WAMP msg with an empty line.


Am 04.11.21 um 19:42 schrieb Emile Cormier:

> <https://github.com/crossbario/autobahn-python/blob/a35f22eeaafca7568f1deb35c4a1b82ae78f77d4/autobahn/wamp/message.py#L121>
>
>
> https://github.com/crossbario/autobahn-python/blob/a35f22eeaafca7568f1deb35c4a1b82ae78f77d4/autobahn/wamp/message.py#L3942

> <https://github.com/crossbario/autobahn-python/blob/a35f22eeaafca7568f1deb35c4a1b82ae78f77d4/autobahn/wamp/message.py#L3942>
>
>
> and here are 2 examples in crossbar which _use_ (build on) payload
> transparency mode:
>
> https://github.com/crossbario/crossbar-examples/tree/master/payloadcodec

> <https://github.com/crossbario/crossbar-examples/tree/master/payloadcodec>
>
> https://github.com/crossbario/crossbar-examples/tree/master/mqtt/basic/passthrough

> <https://github.com/crossbario/crossbar-examples/tree/master/mqtt/basic/passthrough>
>
>
> ---
>
> so given ^, I now think what you propose is a wamp serialization
> format,
> and the payload-transparency cannot emulate that, as it only is
> concerned with, well, the app payload.
>
> Cheers,
> /Tobias
>

> --
> You received this message because you are subscribed to the Google
> Groups "WAMP" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to wampws+un...@googlegroups.com
> <mailto:wampws+un...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/wampws/052a2e2a-7cc5-4ecf-992d-8ef1644cde66n%40googlegroups.com
> <https://groups.google.com/d/msgid/wampws/052a2e2a-7cc5-4ecf-992d-8ef1644cde66n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Emile Cormier]

On Thursday, November 4, 2021 at 9:53:28 PM UTC-3 tobias wrote:

how would the last array item (== line) in a WAMP msg be identified?

one option: end each WAMP msg with an empty line.

The number of lines could be variable, much like there are variable number of array elements in the regular framing scheme.