Help for starting writing code for spark..

[Harshit]

Hi,

I am new to spark and scala, and have gone through the spark programming guide, but i still need help writing jobs in spark.

As i have worked in hadoop map reduce, i understand the philosophy of writing code in map reduce.

I have searched the web but have not found enough examples.

from what i read i am under the impression that it will be hard to transform the existing map reduce code (which is in java) to fit for spark.

 

I also need help regarding the usage of Transformations and Actions mentioned in the Spark Programming Guide.

Please help out with some examples or study material, also let me know that is there any way that the existing hadoop map reduce code can be reused with spark.

[Matei Zaharia]

Hi Harshit,

You can find a bunch of sample programs in the examples/ folder of Spark. They include things like word count, K-means, PageRank, etc. Other than that, the best way to learn Spark is to play with it interactively inside spark-shell.

In terms of learning Scala, there's a great book available free online at http://www.artima.com/pins1ed/.

Right now it's not possible to just use a Hadoop mapper or reducer in Spark, but because you can call into Java code, it shouldn't be hard to factor out the part of the code that's doing the work into a separate class and call it from Scala in a map function.

Matei

[Sotiris Orfanou]

Hi,

I am also new and I am trying to start writing spark programs.I have seen the examples and I understand most of them.

Where I need help is with spark. I was using eclipse in the past to write scala and java programs.How can I use eclipse to write spark.Do you have any suggestions?

Sotiris

[Matei Zaharia]

Sotiris,

You can add Spark as a Maven dependency to your Eclipse project. (You might need a Maven plugin for Eclipse to do this) . Or, try the SBT Eclipse plugin (https://github.com/typesafehub/sbteclipse).

Matei

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.