run with virtual environement in yarn

[Vendaim Part2]

Hi there,

I'm trying to run a python program(lenet.py) using bigdl on yarn cluster mode, when I run https://bigdl-project.github.io/master/#PythonUserGuide/run-without-pip/#yarn.example, it gives the following error:

 ERROR yarn.Client: Application diagnostics message: User application exited with status 1
Exception in thread "main" org.apache.spark.SparkException: Application application_1611155136694_0008 finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:1149)
    at org.apache.spark.deploy.yarn.YarnClusterApplication.start(Client.scala:1526)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala

  I have two nodes. 1 of these is master and both are workers,for this makes the following changes:

    BIGDL_HOME=/home/spark/bigdl
    SPARK_HOME=/home/spark/spark
    PYTHON_API_PATH=${BIGDL_HOME}/lib/bigdl-0.12.1-python-api.zip
    BigDL_JAR_PATH=${BIGDL_HOME}/lib/bigdl-0.12.1-jar-with-dependencies.jar
    PYTHONPATH=${PYTHON_API_PATH}:$PYTHONPATH
    VENV_HOME=/home/spark/bigdl

    PYSPARK_PYTHON=./venv.zip/venv/bin/python3 ${SPARK_HOME}/bin/spark-submit \
    --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./venv.zip/venv/bin/python3 \
    --master yarn-cluster \
    --executor-memory 6g \
    --driver-memory 6g \
    --executor-cores 4 \
    --num-executors 2 \
    --properties-file ${BIGDL_HOME}/conf/spark-bigdl.conf \
    --jars ${BigDL_JAR_PATH} \
    --py-files ${PYTHON_API_PATH} \
    --archives ${VENV_HOME}/venv.zip \
    --conf spark.driver.extraClassPath=bigdl-0.12.1-jar-with-dependencies.jar \
    --conf spark.executor.extraClassPath=bigdl-0.12.1-jar-with-dependencies.jar \
    ${BIGDL_HOME}/examples/lenet/lenet.py

Thanks

Vendaim

[Qiyuan Gong]

Hi Vendaim,

    YarnClusterApplication.start(Client.scala:1526) means your application (BigDL job) crash after submitting to Yarn. 

    Can you share detailed logs of application_1611155136694_0008? 

Have a nice day!

Qiyuan

[Vendaim Part2]

Hi Qiyuan,

Sorry for late reply, I just solved the problem by passing mine images path through --dataPath.

Thanks alot!

Vendaim

[Nagarajan duraipandi]

I am also getting same error :   ERROR yarn.Client: Application diagnostics message: User application exited with status 1

Could pls provide solution fo this ? --dataPath means ? where i can give ?

[Jennie Wang]

Hi, 

--dataPath option means the directory where you put downloaded mnist dataset. If you didn't specify this option, it will download mnist dataset to  "/tmp/mnist". Can you provide complete log for this error?

Thanks,

Jennie