Julia and Spark

[Tanmay K. Mohapatra]

This thread is to discuss Julia - Spark integration further.

This is a continuation of discussions from https://groups.google.com/forum/#!topic/julia-users/LeCnTmOvUbw (the thread topic was misleading and we could not change it).

To summarize briefly, here are a few interesting packages:

- https://github.com/d9w/Spark.jl

- https://github.com/jey/Spock.jl

- https://github.com/benhamner/MachineLearning.jl

- packages at https://github.com/JuliaParallel

We can discuss approaches and coordinate efforts towards whichever looks promising.

[Steven Sagaert]

I've been comtemplating writing a high level wrapper to Spark myself since I'm interested in both Julia & Spark but I was waiting for Julia 0.4 to finalize before even starting.

One can do the integration on several levels:

1) simply wrap the Spark java API via JavaCall. This is the low level approach. BTW I've experimented with javaCall and found it was unstable & also lacking functionality (e.g. there's no way to shutdown the jvm or create a pool of JVM analogous to DB connections) so that might need some work before trying the Spark integration.

2) Spark 1.3 has now new and high level interfaces: dataframe API for accessing data in the form of distributed dataframes & pipeline API to compose algo via pipeline framework. By wrapping the spark dataframe with julia dataframe you would quickly have a high level (data scientist level) interface to Spark. BTW Spark dataframes are actually also FASTER than the more low level approaches like java/scala methods calls or Spark SQL (intermediate level) because Spark itself can do more optimizations (this is similar to how PyData Blaze works). By wrapping the pipeline API one could quickly compose Spark algos to create new algos.

3) for an intermediate approach : wrap the Spark SQL API and use SQL to query the system.

Personally I would start with dataframe & pipeline API. Maybe later on if needed add Spark SQL API and only do the low level stuff last if needed. But before interfacing Spark dataframes with julia ones the julia dataframe should become more powerful: at least && and || should be allowed in indexing for richer "querying" like in R dataframes.

[wil...@gmail.com]

1) simply wrap the Spark java API via JavaCall. This is the low level approach. BTW I've experimented with javaCall and found it was unstable & also lacking functionality (e.g. there's no way to shutdown the jvm or create a pool of JVM analogous to DB connections) so that might need some work before trying the Spark integration.

Using JavaCall is not an option, especially when JVM became close-sourced, see https://github.com/aviks/JavaCall.jl/issues/7.

Python bindings are done through Py4J, which is RPC to JVM. If you look at the sparkR, it is done in a same way. sparkR uses a RPC interface to communicate with a Netty-based Spark JVM backend that translates R calls into JVM calls, keeps SparkContext on a JVM side, and ships serialized data to/from R.

So it is just a matter of writing Julia RPC to JVM and wrapping necessary Spark methods in a Julia friendly way. 

[Steven Sagaert]

yes that's a solid approach. For my personal julia - java integrations I also run the JVM in a separate process.

[wil...@gmail.com]

However, I wonder, how hard it would be to implement RDD in Julia? It looks straight forward from a RDD paper how to implement it. It is a robust abstraction that can be used in any parallel computation.

[Andrei Zh]

Julia bindings for Spark would provide much more than just RDD, they will give us access to multiple big data components for streaming, machine learning, SQL capabilities and much more. 

[wil...@gmail.com]

Of course, a Spark data access infrastructure is unbeatable, due to mature JVM-based libraries for accessing various data sources and formats (avro, parquet, hdfs). That includes SQL support as well. But, look at Python and R bindings, these are just facades for JVM calls. MLLib is written in Scala, Streaming API as well, and then all this called from Python or R, all data transformations happen on JVM level. It would be more efficient write code in Scala then use any non-JVM bindings. Think of overhead for RPC and data serialization over huge volumes of data needed to be processed and you'll understand why Dpark exists. BTW, machine learning libraries in JVM, good luck. It only works because of large computational resources used, but even that has its limits.

[Tanmay K. Mohapatra]

There was some attempt made towards a pure Julia RDD in Spark.jl (https://github.com/d9w/Spark.jl).

We also have DistributedArrays (https://github.com/JuliaParallel/DistributedArrays.jl), Blocks (https://github.com/JuliaParallel/Blocks.jl) and (https://github.com/JuliaStats/DataFrames.jl).

I wonder if it is possible to leverage any of these for a pure Julia RDD.

And MachineLearning.jl or something similar could probably be the equivalent of MLib.

[wil...@gmail.com]

Unfortunately, Spark.jl is an incorrect RDD implementation. Instead of creating transformations as independent abstraction operations with a lazy evaluation, the package has all transformations immediately executed upon their call. This is completely undermines whole purpose of RDD as fault-tolerant parallel data structure.

[ssarkaray...@gmail.com]

Is there any implementation with streams of RDDs for Julia ? 

[Jey Kottalam]

Could you please define "streams of RDDs"?

[Sisyphuss]

http://dl.acm.org/citation.cfm?id=2228301

[Jey Kottalam]

Are you asking about Spark Streaming support?

[ssarkaray...@gmail.com]

Yes.

[Frank]

Hi,

I would have expected more interest in a Spark & Julia integration. Is the lack of interest due to
a) missing use cases

b) fact that both Spark and Julia are very new - relatively speaking

What do you think?

Thanks

Frank

On Wednesday, April 15, 2015 at 11:37:50 AM UTC+2, Tanmay K. Mohapatra wrote:

[Christof Stocker]

Personally, I think the most progress is made if some person has a huge interest in doing it. I for one have a big interest in using Julia for ML, but I myself am not particularly interested in using Spark from Julia. I just don't feel like it would be useful to me for anything. In the situations that I do use spark I don't feel like I would gain anything from using it from Julia. That of course doesn't mean that it wouldn't be very useful to others, but it does mean that it is unlikely that I will spend any of my time on it in the near future. Maybe other people are in similar situations.

What I can leave you with is this: I think open source is a place in which one person can make all the difference in the world, if he/she sets his/her mind to it. So if someone is interested in doing it, go for it. I don't think it's to far fetched to assume that once the functionality is available (and reasonable mature) that people will gravitate towards it.

[Andrei]

Small number of use cases is an important reason. I see many people interested in Julia & Spark integration, but almost nobody interested enough to invest time into its development. 

Another reason is that Julia infrastructure (and especially Julia-Java integration) is not mature enough to make integrations of such level. Instability of JNI, inconsistency between Java and Scala, serialization issues in Julia - these are just few difficulties I faced while working on Sparta.jl. Many people do great work to fix such issues, but at the moment Julia is far behind, say, Python. 

Finally, it's just huge amount of work. I don't mean basic functionality like map and reduce operations over text file, but the whole variety of supported data formats, DataFrames, subprojects like Spark Streaming and MLlib, etc. And without these features we get back to paragraph 1 - nobody is interested enough to invest time when there's already PySpark and SparkR. 

All of these makes me think that similar framework for big data analytics written in pure Julia could bypass many of these issues and generate more interest in Julia community. I wonder if somebody would want to take part in such a challenge. 

[Christof Stocker]

A pure Julia ML ecosystem is something that is actively being worked
on/towards. Naturally large problem sizes are one big reason that people
are interested in working on this. It just takes time to flesh it out to
something that people are used to from languages like python or R. As I
see it, Julia is different and offers unique opportunities for
connecting research, education, and application. It is an interesting
journey to figure out how to get the most out of what the language has
to offer.

[Andrei]

I wouldn't restrict big data analytics to machine learning - it also includes SQL queries and visualization, real-time data enrichment, ETL, etc. I don't think Julia ML ecosystem by itself will ever expand to these areas. 

[Christof Stocker]

You are right. My tunnel vision strikes again. That's what happens when one works with a hammer all day; all I see is nails :-)