Autobahn|Cpp serialization enhancements

[emile.co...@gmail.com]

I'm ready to seriously start evaluating WAMP for our IoT-related project. The backend software will be written in C++, with the UI frontend in HTML/Javascript. I plan on implementing a proof-of-concept demo before committing to use WAMP for the entire project.

Naturally, I would like to use Autobahn|Cpp, but it currently only supports MsgPack payloads. I would prefer to use JSON payloads instead. My motivation for using JSON payloads is due to the excellent Autojsoncxx library, which generates JSON serialization code from an IDL (similar to what Protobuf does).

I feel that the serialization in Autobahn|Cpp should belong to policy classes, instead of being baked into the session class. These serialization policy classes could be a template parameter of  session. For example:

template<typename IStream, typename OStream, typename Serializer = MsgPackSerializer> class session {

Then, instead of:

void publish(const std::string& topic, const anyvec& args, const anymap& kwargs);

we'd have something like:

template <typename TArgs, typename TKwargs>

void publish(const std::string& topic, const TArgs& args, const TKwargs& kwargs);

and then the Serializer policy class would take care of serializing args and kwargs from whatever to whatever. We would of course provide some pre-made serialization policies, but the user would be free to provide his own custom one as well.

I haven't yet figured out how to make the endpoint types more generic/customizable, but I've only recently started looking closely at Autobahn|Cpp.

I'd be willing to contribute my time to enhancing Autobahn|Cpp, but I must first know how attached Tavendo is to the current API. I feel that the way payloads are exchanged needs to be revised, which might break the current API.

Cheers,

Emile

[emile.co...@gmail.com]

For JSON serialization, I plan on using RapidJSON and possibly Autojsoncxx. I could make it so that the user is only bound to those dependencies if they choose the JSON serialization policy. How does Tavendo feel about adding those conditional dependencies to Autobahn|Cpp?

[Tobias Oberstein]

Hi Emile,

Am 28.11.2014 00:54, schrieb emile.co...@gmail.com:
> I'm ready to seriously start evaluating WAMP for our IoT-related
> project. The backend software will be written in C++, with the UI
> frontend in HTML/Javascript. I plan on implementing a proof-of-concept
> demo before committing to use WAMP for the entire project.

That's always a good idea. Verify that it actually works for you and
your requirements.

>
> Naturally, I would like to use Autobahn|Cpp, but it currently only
> supports MsgPack payloads. I would prefer to use JSON payloads instead.
> My motivation for using JSON payloads is due to the excellent

> Autojsoncxx <https://github.com/netheril96/autojsoncxx> library, which

> generates JSON serialization code from an IDL (similar to what Protobuf
> does).
>
> I feel that the serialization in Autobahn|Cpp should belong to policy
> classes, instead of being baked into the session class. These
> serialization policy classes could be a template parameter of session.
> For example:
>
> template<typename IStream, typename OStream, typenameSerializer =
> MsgPackSerializer> classsession{
>
>
> Then, instead of:
>
> voidpublish(conststd::string& topic, constanyvec& args, constanymap&
> kwargs);
>
>
> we'd have something like:
>
> template <typename TArgs, typename TKwargs>
> voidpublish(conststd::string& topic, constTArgs& args, constTKwargs&
> kwargs);
>
>
> and then the Serializer policy class would take care of serializing
> args and kwargs from whatever to whatever. We would of course provide
> some pre-made serialization policies, but the user would be free to
> provide his own custom one as well.

We can add JSON serialization support without affecting the user API.

In fact, the user API should not expose details such as the
serialization used at the transport level. The serialization is
negotiated for WebSocket (and also RawSocket in the latest revision).

>
> I haven't yet figured out how to make the endpoint types more
> generic/customizable, but I've only recently started looking closely at
> Autobahn|Cpp.
>
> I'd be willing to contribute my time to enhancing Autobahn|Cpp, but I
> must first know how attached Tavendo is to the current API. I feel that

Yes, the current API is what I'd call well designed, and we as said
above, we don't want to couple user code to details such as
serialization. Also, since WAMP is dynamically typed, the use of an IDL
to generate statically typed, rigid structures does not fit.

> the way payloads are exchanged needs to be revised, which might break
> the current API.

No, it doesn't. We can implement JSON serialization without changing the
API. If you want to follow that road, that would be of course a welcome
additition! But binding user code to implementation details is a no go
(as far as merging is meant of course .. you can fork and do whatever
you like obviously).

Cheers,
/Tobias

>
> Cheers,
> Emile
>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/autobahnws/b8bc9183-7af4-46db-a157-967c056fb330%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/b8bc9183-7af4-46db-a157-967c056fb330%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Tobias Oberstein]

Hi Emile,

Am 28.11.2014 18:59, schrieb emile.co...@gmail.com:
> For JSON serialization, I plan on using RapidJSON and possibly
> Autojsoncxx. I could make it so that the user is only bound to those

Both of above libs seem to follow the IDL / code generation road, which
does not fit WAMP.

> dependencies if they choose the JSON serialization policy. How does
> Tavendo feel about adding those conditional dependencies to Autobahn|Cpp?

The problem is the one above ...

/Tobias

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com
> <mailto:autobahnws+...@googlegroups.com>.
> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/241fbb65-e8b6-41c3-bf24-c2fbd9f5417b%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/241fbb65-e8b6-41c3-bf24-c2fbd9f5417b%40googlegroups.com?utm_medium=email&utm_source=footer>.

[emile.co...@gmail.com]

On Friday, November 28, 2014 7:01:02 PM UTC-4, Tobias Oberstein wrote:

In fact, the user API should not expose details such as the 
serialization used at the transport level. The serialization is 
negotiated for WebSocket (and also RawSocket in the latest revision). 

I was not aware that the type of serialization can be determined at runtime. That might make my serialization policy class idea unworkable.

Yes, the current API is what I'd call well designed, and we as said 

above, we don't want to couple user code to details such as
serialization. Also, since WAMP is dynamically typed, the use of an IDL
to generate statically typed, rigid structures does not fit.

While I agree that the WAMP messages themselves are dynamically typed, WAMP should not care if the application wants to deal with statically-typed payloads.

 

> the way payloads are exchanged needs to be revised, which might break
> the current API.

No, it doesn't. We can implement JSON serialization without changing the
API. If you want to follow that road, that would be of course a welcome
additition! But binding user code to implementation details is a no go
(as far as merging is meant of course .. you can fork and do whatever
you like obviously).

The reason I don't care much for the current API (based on boost::any), is that it forces me to convert all my statically-typed DTOs into dynamic types.

I had envisioned an API more like:

template <typename TArgs, typename TKwargs>

void publish(const std::string& topic, const TArgs& args, const TKwargs& kwargs);

where I could directly pass my DTOs, and the Serializer policy class would take care of serializing it into JSON or MsgPack. For example:

Payload p = {"John", 42, true};

session.publish("new employee", p);

I could make it so that the default Serializer policy takes anyvecs any anymaps; this would prevent breaking the current API and would allow users to still keep working with dynamic types if that's their preference.

But all this could be moot: I was not aware of the "dynamicness" of the serialization format during connection. I'll have to give it some more thought. For my application, I'm happy to restrict the serialization format to only JSON, but that restriction is obviously not something you want to impose in a library meant for general use!

[emile.co...@gmail.com]

On Friday, November 28, 2014 7:03:23 PM UTC-4, Tobias Oberstein wrote:

Both of above libs seem to follow the IDL / code generation road, which
does not fit WAMP.

Only Autojsoncxx follows the IDL approach. RapidJSON fully supports dynamic types via their Document class.

[Tobias Oberstein]

Just some short note as I am on mobile: I am ready to have a fresh look at all of this, and should we come up with something significantly better, we can even break user API. The hard part probably isnt the outgoing leg, but the incoming. Eg with registering procedures, how to transform the dynamically typed incoming call result into a static type, how to tell the lib that desired static type and how to handle situations where the incoming value cannot be transformed into the former ..

Sent from Mobile (Google Nexus 5)

--

You received this message because you are subscribed to the Google Groups "Autobahn" group.

To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+...@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/d1e2747e-043e-4c99-af13-07506d97102c%40googlegroups.com.

[Tobias Oberstein]

Ah, right. A SAX level approach like  https://github.com/miloyip/rapidjson/blob/master/example/simplereader/simplereader.cpp for transforming incoming WAMP JSON payloads to boost::any might work

Sent from Mobile (Google Nexus 5)

--

You received this message because you are subscribed to the Google Groups "Autobahn" group.

To unsubscribe from this group and stop receiving emails from it, send an email to autobahnws+...@googlegroups.com.
To post to this group, send email to autob...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/autobahnws/7e95c604-fac9-413d-8f72-0d45567a78bc%40googlegroups.com.

[emile.co...@gmail.com]

On Saturday, November 29, 2014 2:30:23 AM UTC-4, Tobias Oberstein wrote:

The hard part probably isnt the outgoing leg, but the incoming. Eg with registering procedures, how to transform the dynamically typed incoming call result into a static type, how to tell the lib that desired static type and how to handle situations where the incoming value cannot be transformed into the former .

I had similar concerns as well. I think the solution might lie in registering a serializer along with the statically-typed callback.

[emile.co...@gmail.com]

Tobias, what was the rationale for having separate IStream and OStream template parameters? Is it even possible for WAMP to run on different transports for input and output?

[Tobias Oberstein]

In principle, this is possible. You could have one leg over a
unidirectional pipe, and the other over a shared-memory queue.

Not that this makes a lot of sense;)

Basically, I needed the types for the 2 args (ingress/egrees leg).

[David Chappelle]

You should take a look at msgpack-rpc-cpp for examples of how to do the call marshalling in a way that is fast and does not require every single parameter to incur the overhead of a malloc call as is the case with boost::any. In particular refer to caller.h. It utilizes msgpack::type::tuple to bundle together the parameters. They didn't so such a great job with unmarshalling but I am sure a similar semantic could be used. If you could get a similar tuple class written in json then the two should be interchangeable perhaps as a plugin.

[emile.co...@gmail.com]

Thank you for your comments and rationale behind some of the design decisions behind Autobahn|Cpp.

I'm getting the impression that we have different design philosophies, so I think I'll go develop my own WAMP C++ library. I'm under a bit of time pressure to move forward in our project, so I don't have time to "negotiate" the features that I would like to see implemented.

If I go ahead with this library, I plan on making it open-source. I'll share the link to the repo once I have something semi-stable.

Again, thank you for your comments and good luck with WAMP! :-)

[Tobias Oberstein]

David,

Am 30.11.2014 17:41, schrieb David Chappelle:
> You should take a look at msgpack-rpc-cpp for examples of how to do the
> call marshalling in a way that is fast and does not require every single
> parameter to incur the overhead of a malloc call as is the case with

Faster than what? How much faster? Do I care?

Given octets will walk through a complete TCP/IP stack and then over
some wire, micro optimizations simply might get dwarfed by the former.

Also: small object allocation can be done "efficiently" using

http://www.boost.org/doc/libs/1_57_0/libs/pool/doc/html/boost/pool_allocator.html
http://www.boost.org/doc/libs/1_57_0/libs/pool/doc/html/boost/fast_pool_allocator.html

doing away with actual malloc syscalls altogether.

I don't know if that can be plugged into boost::any though ..

> boost::any. In particular refer to caller.h. It utilizes
> msgpack::type::tuple to bundle together the parameters. They didn't so
> such a great job with unmarshalling but I am sure a similar semantic
> could be used. If you could get a similar tuple class written in json
> then the two should be interchangeable perhaps as a plugin.

MsgPack uses length prefixing for strings, whereas JSON does not. Which
means, when you start to receive a string, you don't know it's size.
Sooner or later you _will_ need dynamic memory when parsing JSON.

Also: what exactly is the goal here? If it's about performance, we
should _first_ define a benchmark, _second_ measure existing code, and
only _third_ implement and test alternatives.

If it's about user API and convenience, this is something I do care a
_lot_ about. Do you think (like Emile seems) that boost::any is bad as a
parameter/result container in the user API of AutobahnCpp?

Cheers,
/Tobias

>
> On Sunday, 30 November 2014 10:35:10 UTC-5, Tobias Oberstein wrote:
>

> Am 29.11.2014 18:32, schrieb emile.co...@gmail.com <javascript:>:
> > Tobias, what was the rationale for having separate IStream and
> OStream
> > template parameters? Is it even possible for WAMP to run on
> different
> > transports for input and output?
>
> In principle, this is possible. You could have one leg over a
> unidirectional pipe, and the other over a shared-memory queue.
>
> Not that this makes a lot of sense;)
>
> Basically, I needed the types for the 2 args (ingress/egrees leg).
>

> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+...@googlegroups.com>.

> To post to this group, send email to autob...@googlegroups.com

> <mailto:autob...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/9b3dd6e9-c111-418c-a8e3-9d00586e24bd%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/9b3dd6e9-c111-418c-a8e3-9d00586e24bd%40googlegroups.com?utm_medium=email&utm_source=footer>.

[David Chappelle]

The malloc and copy of every parameter is pretty wasteful. What if a parameter is a large container or large string? I declare a string, populate it (i.e. several KB of json), then make a session call which allocates a new piece of memory and copies the data over. Then we get to the transport layer and probably copy it again into a temporary buffer and then finally call write() to pass it off to the kernel and out the network stack. What if we have to process 10,000s of these per second? I come from a background in very high performance embedded development primarily building software/hardware systems deployed into Tier 1 carrier grade networks so these things are somewhat ingrained in my philosophies. There is a concerted focus these days on zero copy semantics and minimizing wasteful calls to allocate memory. Cap-n-proto and nanomsg are very good examples.

I agree with you completely though. User API is always first and foremost. I have seen far too many complex and confusing APIs that were built for performance and not usability.

In my particular use case, CPU cycles are important because I need to be very conservative when it comes to power consumption. Plus, everything will be running locally over unix domain sockets. So in my case, the network overhead isn't really an issue at all but minimizing cpu cycles is. I also need a service oriented architecture and not just a classic monolithic one so I need to strike a balance.

In general the usage of boost::any isn't a big deal for an api used for things like web based systems where messaging rates are low volume and system resources are more plentiful. However, for embedded systems where constraints on CPU utilization are a bit tighter these sorts of things tend to be points of contention. Eliminating a few copies and memory allocations in operations that happen 10,000s of times per second can be very important even though they are perceived as micro optimizations.

> <mailto:autobahnws+unsub...@googlegroups.com>.

[emile.co...@gmail.com]

In case anyone's interested, I just thought I'd announce that I've begun working on my own C++ WAMP library. This way, I can proceed full speed without being a drain on Tavendo's time. Personally, I would much rather that Tavendo focuses its time on Crossbar.io and Javascript implementations, where I have vastly less expertise than in C++.

Here's a summary of my progress. The C++11 library currently supports both JSON and Msgpack, via a recursive Variant object that I wrote from scratch. The transport layer is handled by an extensible class hierarchy, and currently supports TCP and Unix domain sockets via Boost.Asio. Except for calling ioservice.run(), the user of the library will not be exposed to any details from Boost. It should be possible to later add support for Websockets by extending the transport layer classes via inheritance and making use of zaphoyd/websocketpp for the actual implementation.

Both client and router roles are currently supported for raw sockets at the transport layer. At the WAMP sesssion layer above, I only plan on supporting the client role for now. I might end up implementing a portion of the router role at the WAMP session layer if it simplifies unit testing.

I later plan on adding support for my Variant class to Autojsoncxx. This would allow the user to pass DTOs (i.e. structs) as WAMP call parameters, and Autojsoncxx would convert them into Variant objects. I also plan on supporting variadic templates for CALL and PUBLISH requests, which will hopefully make the API look very similar to regular function calls.

I will host the project publicity on Github once it reaches a minimal usable state (with some minimal documentation). Even though this work is part of a commercial endeavor, our company will open-source this library because we have an interest in seeing the WAMP ecosystem thrive. We will also benefit by having others find defects and suggest improvements for our library.

Cheers,

Emile

[Tobias Oberstein]

Hi Emile,

> In case anyone's interested, I just thought I'd announce that I've

Yes, absolutely! Please keep us up to date with your efforts.

begun
> working on my own C++ WAMP library. This way, I can proceed full speed
> without being a drain on Tavendo's time. Personally, I would much rather
> that Tavendo focuses its time on Crossbar.io and Javascript

We will;) I really appreciate your attitude: ask, have an opinion,
disagree, write code, thinking the "bigger picture" (WAMP ecosystem), ..

>
> Here's a summary of my progress. The C++11 library currently supports
> both JSON and Msgpack, via a recursive Variant object that I wrote from
> scratch. The transport layer is handled by an extensible class
> hierarchy, and currently supports TCP and Unix domain sockets via
> Boost.Asio. Except for calling ioservice.run(), the user of the library
> will not be exposed to any details from Boost. It should be possible to
> later add support for Websockets by extending the transport layer
> classes via inheritance and making use of zaphoyd/websocketpp for the
> actual implementation.

So you are using RawSocket framing currently, with either JSON or MsgPack?

One note rgd RawSocket / Crossbar.io: we still need to fix:

https://github.com/tavendo/AutobahnPython/issues/291

Currently, we still use the "old" (pre-revision) RawSocket framing.

>
> Both client and router roles are currently supported for raw sockets at
> the transport layer. At the WAMP sesssion layer above, I only plan on
> supporting the client role for now. I might end up implementing a
> portion of the router role at the WAMP session layer if it simplifies
> unit testing.
>
> I later plan on adding support for my Variant class to Autojsoncxx. This
> would allow the user to pass DTOs (i.e. structs) as WAMP call
> parameters, and Autojsoncxx would convert them into Variant objects. I
> also plan on supporting variadic templates for CALL and PUBLISH
> requests, which will hopefully make the API look very similar to regular
> function calls.

This sounds cool. Personally, I think the user experience of an API is
critically important .. and with C++ in particular to keep the syntax
clean/sane.

>
> I will host the project publicity on Github once it reaches a minimal
> usable state (with some minimal documentation). Even though this work is
> part of a commercial endeavor, our company will open-source this library
> because we have an interest in seeing the WAMP ecosystem thrive. We will

Awesome! I highly appreciate the ecosystem thinking!

> also benefit by having others find defects and suggest improvements for
> our library.

Yes. That's open-source. Btw: please consider chosing a "liberal"
license as this is a WAMP client library directly linking with user code.

Cheers,
/Tobias


>
> Cheers,
> Emile

>
> --
> You received this message because you are subscribed to the Google
> Groups "Autobahn" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to autobahnws+...@googlegroups.com

> <mailto:autobahnws+...@googlegroups.com>.

> To post to this group, send email to autob...@googlegroups.com
> <mailto:autob...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/autobahnws/3ed8e792-70c5-4aef-8f2b-accb52b292e0%40googlegroups.com
> <https://groups.google.com/d/msgid/autobahnws/3ed8e792-70c5-4aef-8f2b-accb52b292e0%40googlegroups.com?utm_medium=email&utm_source=footer>.

[emile.co...@gmail.com]

Thanks for the encouragement, Tobias. Since there will be a heavy dependency on Boost.Asio, my library will adopt the Boost license as well.

[emile.co...@gmail.com]

To those following this thread, I have made an announcement on the WAMP group regarding the initial release of my library.