AROM: Processing Big Data with Data Flow Graphs and Functional Programming

[Nam-Luc Tran]

Dear BeScala members,

I am Tran Nam-Luc and I work for the R&D department of Euranova (http://www.euranova.eu), a Belgian company focused on innovation in IT.

In the scope of one of our research project we have recently developped AROM, a distributed processing framework which main intention is to provide a playground for research and developing on distributed processing.

Indeed AROM is based on the general Data Flow Graph processing model which we believe is more general than the MapReduce model proposed by Google. This not only allows to design distributed algorithms in a more natural way but also leaves more opening for optimizations. Examples of such algorithms comprise relational operations, iterations and, especially, jobs with pipelined topologies. 

Based on the DFG processing model, AROM jobs are defined as Directed Ayclic Graphs (DAG) where each vertex is a program processing data, also called operator, and edges represent communication channels between vertices. We provide a comprehensive API for the definition of the job graph and the connections between the operators.

AROM is written in Scala. Using functional programming paradigms to express the jobs permits us to enforce generic and reusable operators (this is also the point-of-view of many projects involved in data processing, like Scalding for example). At its basis it uses the Akka actors framework. This architecture potentially allows for deployment on very large clusters and on Cloud infrastructures.

Very recently, we have released AROM as an Open Source project. We are interested in exposing the ideas and concepts of AROM to a community of Scala enthusiasts and in developing a community around the project. Would you be interested in a presentation and a talk around AROM?

Looking forward to hearing from you,

Nam-Luc Tran <namluc DOT tran AT euranova DOT eu>

AROM on github: https://github.com/nltran/arom

[Robert Kirkpatrick]

Very interesting!

Robert.

[Renato Guerra Cavalcanti]

Hi Nam,

Thanks for sharing that with us. 

And yes, we can certainly organise a meet-up where you can present the project. I'm sure the whole Belgian Scala community will be interested in hearing more about it.

We are now in the middle of the holidays and many of us are traveling. I think we can start looking into that on the second half of August. What do you think?

Regards,

Renato

[andy petrella]

+1 

Looking forward for this meetup...

Looking great stuffs Nam (kind of thing that shake the old mathematician that is lost somewhere my mind)

Thanks Renato

(don't forget the tool that should poke me some days before ^^)

--

Andy Petrella

Belgium (Liège)
********

 IT Consultant for NextLab sprl (co-founder)
********
Mobile: +32 495 99 11 04

Mails:  

• andy.p...@nextlab.be
• andy.p...@gmail.com

Socials:

• Twitter: https://twitter.com/#!/noootsab
  
• LinkedIn: http://be.linkedin.com/in/andypetrella
• Blogger: http://ska-la.blogspot.com/
• GitHub:  https://github.com/andypetrella

[Francis De Brabandere]

+1 certainly interested!

Cheers,
F

On 7 August 2012 22:06, BeScala on behalf of andy petrella

[Nam-Luc Tran]

Hello everybody,

Thank you all for your replies, I am glad that the idea has drawn much interest  :)

How do you usually organize your meet-up dates, do you plan in advance a schedule for the whole year, or can I propose you a date for the presentation for the first or second week of September?

Cheers,

Nam 

[Renato Guerra Cavalcanti]

Hi Nam,

We don't have a year schedule. 

We are a small community and we do it ad-hoc. When we have someone willing to give a talk, we look for a location and date and announce it on our different channels (website, mailing list, LinkedIn, twitter, etc).

So basically, if Euronova can host the event, we only need to find a date and start to announce it. 

Cheers,

Renato