How to hand a file to Main OR where does Main run

[m.neuma...@gmail.com]

Hej,

I run Stratosphere on Yarn starting jobs from an edge node outside of the actual compute cluster.
The program I wrote has lots of parameter and I don't want to hand them over as command line parameter. My current idea is to just put them in a java properties file and hand that over when starting the job.
Before I start programming and testing things I need to know if this could even work.

The config file is local on the machine where I start the job its only read by the Main thread. I guess the Main thread runs on the master node that Yarn assigns and therefor running on a different machine. Is that true?
If yes how could I pass the parameter file in a way that I can access in the Main thread? It cannot be part of the Jar since it needs to be easy to change.

cheers Martin

[Fabian Hueske]

Hej,

the main() method of the program is invoked by the client (CLI or Web). The client assembles and configures the program (by calling main()), calls the optimizer, and submits the resulting execution plan to the master (JobManager). So the program configuration happens on the machine where you run the client.

If you read the config file only in the main() method (and not in the constructors of your functions), I think it should work.

Cheers, Fabian

--
You received this message because you are subscribed to the Google Groups "stratosphere-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stratosphere-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/stratosphere-dev.
For more options, visit https://groups.google.com/d/optout.

[Stephan Ewen]

You can also run main() locally when you use a RemoteEnvironment. But if you use the command line, then the main runs where the command line runs, as Fabian said.