Actor Framework for distributed System

[Ahsan Ahmad]

Hi all,
I have been entrusted with the task of ascertaining if Actor Framework suits our needs. Ours is a distributed Systems Project in which nodes would be interconnected using different possible configurations e.g. Ring topology (no central/ master node), Star topology (one master) and Ip network (no master).

The nodes might select a control master themselves later for other functions but it wont be defined at the start. Any node may go off  or new may arrive anytime. If control master goes off, the remaining elect a new one. Uptill now we have been working with ZeroMQ for internode communication.

I have read the whole CAF documentation.
1) My First Qs is, would you recommend CAF for such a project. The nodes use embedded ARM processors which performs only control tasks. The major donkey work is done by FPGA also in the node.   
2) Suppose all nodes are connected with Ip protocol. Suppose there are 5 nodes working properly. They are using ptp messgages as well as pool::broadcast. 6th one arrives. How does it tell the others about its existence and that it too should be added to the broadcast list. One solution i could think of is that each node, in parallel to CAF, should monitor its network interface and when "discovery packet" arrives, it should somehow be able to tell the CAF Framework that the new node is there and the appropriate actions be taken.

I hope I was able to clearly explain the problem. Thanks in advance for your help. 

[Ahsan Ahmad]

I read the Docu again. Does the answer lies in using Broker ?

[Dominik Charousset]

Hi,

I have been entrusted with the task of ascertaining if Actor Framework suits our needs. Ours is a distributed Systems Project in which nodes would be interconnected using different possible configurations e.g. Ring topology (no central/ master node), Star topology (one master) and Ip network (no master). 

whether CAF is a good fit depends on your operating system and hardware platform. Are you using a Linux variant? Our default network backend uses sockets and our scheduler uses threads from the C++ standard library.

The nodes might select a control master themselves later for other functions but it wont be defined at the start. Any node may go off  or new may arrive anytime. If control master goes off, the remaining elect a new one. Uptill now we have been working with ZeroMQ for internode communication.

I have read the whole CAF documentation. 
1) My First Qs is, would you recommend CAF for such a project. The nodes use embedded ARM processors which performs only control tasks. The major donkey work is done by FPGA also in the node.    

Given that you are using (or have used) ZeroMQ, I assume your nodes are powerful enough to run a "full fledged" OS with sockets, etc. That makes CAF a viable option. One advice I can give you in regards to power consumption is to switch from the work-stealing scheduler implementation to the work-sharing one. In a low-concurrency setting, the latter one should put less burden on the CPU.

2) Suppose all nodes are connected with Ip protocol. Suppose there are 5 nodes working properly. They are using ptp messgages as well as pool::broadcast. 6th one arrives. How does it tell the others about its existence and that it too should be added to the broadcast list. One solution i could think of is that each node, in parallel to CAF, should monitor its network interface and when "discovery packet" arrives, it should somehow be able to tell the CAF Framework that the new node is there and the appropriate actions be taken.

I read the Docu again. Does the answer lies in using Broker ?

Yes, brokers are our abstraction for networking. However, they use TCP sockets underneath. Support for UDP-based brokers is planned [1], but it seems no one is currently working on this at the moment.

    Dominik

[1] https://github.com/actor-framework/actor-framework/issues/145

[Ahsan Ahmad]

Thanks a lot for your reply.

Yes, we are using linux OS. Processors are very powerful so their overburdening is not a problem. When I said they are embedded ARM processors, i meant to imply that they wont be required to perform heavy duty computations or simulations.

The reason I aksed if CAF would be good for such a system is because that Docu on most part seems to be referring to the sort of problems where a pool of nodes collectively carry out intensive computation the sort of which you require in Finite Element Methods Analysis. My project on the other hand implements an audio system. All nodes are audio sources and sinks. The major work (audio processing) is done by FPGAs. The processors perform control and switching tasks. With CAF, we just want to simplify tasks such as control data exchange and replication.

We are good with TCP. We need reliable communication.

Could you specify resources where I could learn more about Brokers. I drew this figure for you. Should the new node send simply an IP broadcast and it will be picked up by brokers in the already existing nodes ? Is there any example that addresses the same problem ?

[Ahsan Ahmad]

'The actors in the figure are A, B, C and D from left to right. I forgot to label them.  :)

[Dominik Charousset]

The reason I aksed if CAF would be good for such a system is because that Docu on most part seems to be referring to the sort of problems where a pool of nodes collectively carry out intensive computation the sort of which you require in Finite Element Methods Analysis. My project on the other hand implements an audio system. All nodes are audio sources and sinks. The major work (audio processing) is done by FPGAs. The processors perform control and switching tasks. With CAF, we just want to simplify tasks such as control data exchange and replication.

We are good with TCP. We need reliable communication.

Then CAF seems like a good fit.

Could you specify resources where I could learn more about Brokers. I drew this figure for you. Should the new node send simply an IP broadcast and it will be picked up by brokers in the already existing nodes ? Is there any example that addresses the same problem ?

Broadcasting is usually done via UDP. If you can do the bootstrapping via a well-known host, than you could use CAF's group communication. One server would publish its groups at a well-known port and each node would then connect to this server. After that step, all nodes are in the same group and already can talk to each other (via the server). It is also possible to implement an UDP-based group module that allows you to subscribe to multicast addresses directly and to broadcast your credentials (but would require to implement the "low-level" UDP handling in the module).

I think the broker and group server/chat examples might be of some help to you: https://github.com/actor-framework/actor-framework/tree/master/examples

    Dominik