actor_pool performance

[gry....@gmail.com]

I have some problems with the behavior of the actor pool. I’am using it for accessing a key/value store (based on LMDB). For increasing the read performance I ‘am using a set of workers. But dispatching the read requests properly seems to be a problem with the current implementation. Not all the requests are satisfied in the same time. The fastest are around 2 usec while the slowest (disk access) could be 1000 times slower!  Using a round robin strategy for example is not optimal, since the requests waiting behind a slow request will get delayed while other workers probably have nothing to do.

This problem can’t be solved by a modified dispatching strategy (just improved).

 

Ideally there should be a single central queue where all the workers are consuming the next message.

Are there plans to implement such a pool? Or are the available MPMC queues not fast enough?

[Dominik Charousset]

Hi,

the problem is that actor pools do not assume to have all workers run in the same process. Also, users would no longer be able to add workers at runtime with MPMC queues in the mix (because actors cannot change the mailbox dynamically).

Using streaming instead of a pool could offset this problem, since the backpressure allows for more intelligent work assignment than round-robin. However, our streaming implementation is still in early development. A concurrency-only pool could fix your performance problem in the short term but is not on our list.

Instead of using a shared MPMC mailbox, you could also implement a task pool with a max. number of concurrently running tasks. You can simply implement lightweight tasks by inheriting from `resumable`. This interface is used by the scheduler of CAF and allows you to schedule arbitrary tasks alongside event-based actors. I think the task model is easier to implement if you want to give it a try (i.e., don't have the time to wait a few months for the streaming engine).

    Dominik

[gry....@gmail.com]

Hi Dominik,

Thank you for the explanation. At the moment we are working with the current actor pool. As long as all the data does fit in the memory of our servers it’s fine. Most of the clients of our PushServers are connected via WWAN, so small variations in speed do not count.

You are right, a local concurrency pool would be the best performing solution for our problem.

The streaming engine does sound interesting. Does this mean actors would be connected by something like a pipe with flow control?

-Gerry

[Dominik Charousset]

Hi,

> The streaming engine does sound interesting. Does this mean actors would be connected by something like a pipe with flow control?

yes. We are also thinking about adding robust clustering options with automatic re-deployment of failed workers etc. Don't get too excited yet, though. We're still in early development. Expect more news on this in the next couple of months.

Dominik

[gry....@gmail.com]

Hi, 

I’am looking forward to the new CAF features. Clustering is something we need too. One part of our server is using the RAFT protocol to synchronize the key/value store inside a cluster. Unfortunately I haven’t found the time to rewrite the RAFT part to use the CAF.

-Gerry

[Matthias Vallentin]

> Clustering is something we need too. One part of our server is using
> the RAFT protocol to synchronize the key/value store inside a cluster.
> Unfortunately I haven’t found the time to rewrite the RAFT part to use
> the CAF.

I've started a very rough Raft implementation. Ultimately, we'll have
a first-class consensus module available. It might be a Zookeeper
wrapper at first, but native CAF support will follow suit.

The abstraction will be similar to etcd: a key-value store sitting on
top of a (Raft) consensus module.

Matthias

[gry....@gmail.com]

This sounds extremely interesting. I will keep your great blog in mind.

-Gerry

[Gerry Meyer]

What happened to the RAFT consensus module? 

I saw it once in VAST, but now it did disappear now and it never appeared in CAF.

Will there be ever a consensus module in CAF?

Does CAF has a future still (outside Tenzir)? 

[Matthias Vallentin]

Hi Gerry,

We recently stopped using the consensus module in VAST because a simple KV-store did the trick as well. The implementation was always 95% there, and worked for our small use case, but it still needs that extra 5% of intensive testing in order to become useful for a variety of different use cases. It's a text-book implementation based on the paper, so everything is there in principle.

At Tenzir, we currently lack the cycles to bring it back, but would be very happy to revive it as an open-source project in itself, as long as there is interest and time-investment from the community.

CAF is surely used outside Tenzir and thriving on its own, and I'll let Dominik chime in on this. :-)

--
You received this message because you are subscribed to the Google Groups "actor-framework" group.
To unsubscribe from this group and stop receiving emails from it, send an email to actor-framewo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/actor-framework/f4871a11-0cdd-46cf-8e10-afc49ca74783o%40googlegroups.com.

[Dominik Charousset]

Hi Gerry,

> What happened to the RAFT consensus module?
> I saw it once in VAST, but now it did disappear now and it never appeared in CAF.
> Will there be ever a consensus module in CAF?

We are currently focusing our efforts on other aspects, but I could see a RAFT implementation as part of a clustering module. That won’t happen before our redesigned network layer [1] is ready to fly, though.

> Does CAF has a future still (outside Tenzir)?

Absolutely! To sort this out first: you are probably referring to personal ties that existed between Tenzir and CAF in the past. I can assure you that CAF stands on its own.

Now, regarding the future of CAF. The upcoming 0.18 release is going to be a major milestone in the development of CAF and towards a 1.0 release. We have applied many lessons learned, reviewed almost all aspects of the framework, benchmarked (and optimized) CAF in a variety of work loads and use cases, and—last but not least—worked closely with a handful of companies that already use CAF in production today (one exciting new feature that came out of such a collaboration is the new Metrics module [2]).

If you feel like there could be more activity outside of the GitHub repository: I hear you. But don’t worry. This is the calm before the storm. :)

Dominik

[1] https://github.com/actor-framework/incubator/tree/master/libcaf_net
[2] https://actor-framework.readthedocs.io/en/latest/Metrics.html