could python program in spark run on hadoop-yarn?

[Xicheng Dong]

hi, all

   could spark python-program run on hadoop-yarn? I read running-on-yarn.html and python-programming-guide.html, but do not find any useful infomation.

[Josh Rosen]

It might work using the `yarn-client` mode (https://spark.incubator.apache.org/docs/latest/running-on-yarn.html#launch-spark-application-with-yarn-client-mode), but I haven't tried it yet. PySpark has a dependency on Py4J, but only in the driver JVM, so I don't know that we need to have anything in SPARK_YARN_APP_JAR since all of the dependencies should be included in SPARK_JAR anyways.

I've opened a JIRA ticket to track progress on Yarn support for PySpark, whether it ends up simply requiring better documentation or new code: https://spark-project.atlassian.net/browse/SPARK-1004

On Tue, Dec 17, 2013 at 1:13 AM, Xicheng Dong <cls...@gmail.com> wrote:

hi, all

   could spark python-program run on hadoop-yarn? I read running-on-yarn.html and python-programming-guide.html, but do not find any useful infomation.

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.