Thoughts on implementing a zeromq pub/sub broker

[Jun Zhang]

Hi there,

I'm very excited to find that CAF is the one framework that I want and fills the gap in C++ with very high quality implementation. After I find CAF I decide to use it as the backbone of my project.

I am mostly working on the financial services area, the market data population requires high-throughput and low-latency transport. Zeromq pub/sub is a quite popular choice for this task (yes there are messages lost, but as stock market data is always updating so the lost is tolerable, like in streaming video).

So I find bridging the zeromq and CAF world would be beneficial. What I am thinking of now is this (correct me if there are simpler ways):

add something like:

    void publish(caf::actor whom, std::string const& endpoint)

which basically extends the current actor publish to a zeromq (pub) endpoint, like "tcp://*5555".

also at the other end, add a 

    actor remote_actor(const std::string& endpoint)

which spawns a broker to a zeromq (sub) endpoint, like "tcp://127.0.0.1:5555", 

I am still struggling to understand how to fill the gap between the CAF and zeromq world. So the solution can be totally open, maybe even just use simple bridge actors to do message transfer.

Any thoughts/ideas would be very appreciated.

Thanks

[Dominik Charousset]

Hi Jun,

> I'm very excited to find that CAF is the one framework that I want and fills the gap in C++ with very high quality implementation. After I find CAF I decide to use it as the backbone of my project.

just out of curiosity: how did you find out about CAF? Where you explicitly searching for "actor model c++"?

> I am mostly working on the financial services area, the market data population requires high-throughput and low-latency transport. Zeromq pub/sub is a quite popular choice for this task (yes there are messages lost, but as stock market data is always updating so the lost is tolerable, like in streaming video).
>
> So I find bridging the zeromq and CAF world would be beneficial. What I am thinking of now is this (correct me if there are simpler ways):
>
> add something like:
>
> void publish(caf::actor whom, std::string const& endpoint)
>
> which basically extends the current actor publish to a zeromq (pub) endpoint, like "tcp://*5555".
>
> also at the other end, add a
>
> actor remote_actor(const std::string& endpoint)
>
> which spawns a broker to a zeromq (sub) endpoint, like "tcp://127.0.0.1:5555",
>
> I am still struggling to understand how to fill the gap between the CAF and zeromq world. So the solution can be totally open, maybe even just use simple bridge actors to do message transfer.
>
> Any thoughts/ideas would be very appreciated.

CAF has groups to deal with pub/sub communication patterns. So my first suggestion would be to have a look at group modules. An API for your actors would look something along the lines of this: `self->join(group::get("0mq", "tcp://*5555"));`.

You need to implement the interface abstract_group::module. The three member functions you need to worry about are "subscribe", "unsubscribe", and "enqueue". The "enqueue" function is called whenever an actors sends something to the group. The other two functions should be self-explanatory.

I should be rather straightforward to wrap 0mq in a group module. There's one caveat, though: if you receive a message via 0mq, you'll most likely have to map the sender of each message to invalid_actor, unless you couple 0mq with some on-the-fly connection broker to be able to create valid proxies (at which point you would have to deal with CAF's middleman and basp_broker). As long as you don't need to reply to published messages, you'll be fine. :)

Dominik

[Jun Zhang]

Thank you very much for the reply, I'll have a try :)

Btw, I actually move to CAF from scala/akka. A simple search on actor model lead me to akka, but "c++ actor" seems to point to CAF. The reasons that I move are:

1. I'm quite experienced in C++ but Scala is a whole new beast to tame

2. CAF has far exceeded my expectation on a C++ library and I love its coding style

3. I test the performance myself, which is around 3 million msg/sec on my mac book, which is amazing. (but group performance is not suitable for market data)

4. It builds and starts working almost out-of-the-box, compares to akka, which takes me weeks to even get started.

And I'm also interested in adding msgpack as a wire format, as basp_broker currently do. Besides its performance and compactness, msgpack has a wide language support, which would greatly help cross language communication (mainly Python in my area). Has there been any thoughts about that?

在 2015年2月6日星期五 UTC+8下午11:11:18，Dominik Charousset写道：

[Dominik Charousset]

> Thank you very much for the reply, I'll have a try :)

Good luck. :)

> 1. I'm quite experienced in C++ but Scala is a whole new beast to tame
> 2. CAF has far exceeded my expectation on a C++ library and I love its coding style
> 3. I test the performance myself, which is around 3 million msg/sec on my mac book, which is amazing. (but group performance is not suitable for market data)
> 4. It builds and starts working almost out-of-the-box, compares to akka, which takes me weeks to even get started.

Music to our ears.

> And I'm also interested in adding msgpack as a wire format, as basp_broker currently do. Besides its performance and compactness, msgpack has a wide language support, which would greatly help cross language communication (mainly Python in my area). Has there been any thoughts about that?

There were some thoughts on using something like google protobuf/msgpack in the past. The main reason we go with a custom binary format for now is to keep CAF dependency-free. Squeezing out the last bit of performance by packing the data more efficiently is something we might still want to look into in the future. It doesn't have a high priority, though.

As far as cross-language communication is concerned, I would suggest to either (1) run a full CAF node firing selected messages through a scripting engine (and read the result back) or (2) specify an API using whatever protocol you prefer and use a broker to translate between "the outside" and your actor system. Otherwise, you would have to "act" as a proper CAF node, meaning you would essentially have to re-implement BASP (which you should consider to be a moving target, see basp.hpp/basp_broker.hpp) in Python.

Dominik

[Jun Zhang]

One thing I find is that a new wire format is not only about serialize/deserialize, it has to work with CAF's uniform_type system.

As msgpack is only a specification, not truly a library, so using msgpack can also be dependency-free.

I'll try to implement one msgpack for CAF with no extra dependencies apart from c++11, and see how it goes...

在 2015年2月7日星期六 UTC+8上午12:41:16，Dominik Charousset写道：

[Jun Zhang]

One thing I find is that a new wire format is not only about serialize/deserialize, it has to work with CAF's uniform_type system.

As msgpack is only a specification, not truly a library, so using msgpack can also be dependency-free.

I'll try to implement one msgpack for CAF with no extra dependencies apart from c++11, and see how it goes...

在 2015年2月7日星期六 UTC+8上午12:41:16，Dominik Charousset写道：

> Thank you very much for the reply, I'll have a try :)

[Peter Kümmel]

Have you also checked
http://nanomsg.org
a successor of zmg, a more lightwight lib by
the founder of zmq.
Should be simpler to be integrated in caf.

> --
> You received this message because you are subscribed to the
> Google Groups "actor-framework" group.
> To unsubscribe from this group and stop receiving emails
> from it, send an email to
> actor-framewo...@googlegroups.com
> <mailto:actor-framewo...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Jun Zhang]

Thanks Peter, I know that one, and I see a great potential in this project. But seems the development going on this project is not very active?

在 2015年2月10日星期二 UTC+8上午2:33:10，Peter Kümmel写道：

> <mailto:actor-framework+unsub...@googlegroups.com>.