Oozie and pyspark

[Buntu Dev]

I'm trying to use the Oozie Spark action and running into issue where the dependent .py files are expected to be local. I do not see any 'FILES' option for the spark action, how does one go about adding the .py files to the action?

Thanks!

[Romain Rigaux]

Could you specify them in the 'Jars' section?

https://oozie.apache.org/docs/4.2.0/DG_SparkActionExtension.html

On Fri, Apr 22, 2016 at 8:00 AM, Buntu Dev <bunt...@gmail.com> wrote:

I'm trying to use the Oozie Spark action and running into issue where the dependent .py files are expected to be local. I do not see any 'FILES' option for the spark action, how does one go about adding the .py files to the action?

Thanks!

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+u...@cloudera.org.

[Buntu Dev]

I tried that and still get "Error: Only local python files are supported". I even added the files to the spark opts "--py-files" and "--files" but didn't help. Is there a working example of the workflow definition?

Thanks!

[Romain Rigaux]

This is what we could make work so far: http://gethue.com/use-the-spark-action-in-oozie/

I would recommend to ask on the Oozie list as the Spark action is pretty brittle until it is ported to work with Livy