Spark - help in writing code

[Sotiris Orfanou]

Hi,

Is there anything that can help me create my own spark program with its dependencies. 

What I mean: I create a file with a text editor and called it "test".In the test file I have my spark code but I don't know how to get it running with 

spark command : ./run myfile MASTER-IP

What do I have to do?

Is there anything like a guide or tutorial.I have seen many tutorials about spark code, but none of them show you how to add the needed dependencies or anything else in a jar file and where to put it in order to run it in spark

Sotiris

[Mark Hamstra]

The Quick Start Guide shows how to build and run stand alone Spark programs using the Scala, Java and Python APIs.  You should not be using the 'run' script to launch your jobs -- that is not it's intended purpose.

[Sotiris Orfanou]

So if I understand correct, I just write my code in an myfile.scala create .sbt file and it should work with sbt run on the cluster?

[Mark Hamstra]

No, the Quick Start Guide shows you how to run the job locally.  From that Guide:

This example only runs the job locally; for a tutorial on running jobs across several machines, see the Standalone Mode documentation, and consider using a distributed input source, such as HDFS.

[Sotiris Orfanou]

Yes, sorry I meant the pc not cluster. In order to use my program on a cluster I must first set up slaves and master and write my code so that it can work in parallel.
Then, I run my program with the master_ip as an input so that the job will be send on the master.
At least that is what I understood from what I have read. 

Thanks for your help Mark.