"/bin/bash: java: command not found" problem:spark run on yarn

906 views
Skip to first unread message

kunp...@gmail.com

unread,
Aug 8, 2013, 1:57:24 AM8/8/13
to spark...@googlegroups.com
Hi,

   Based on the guide of " launching Spark on YARN",I  build the spark(0.6.0) and run the example of SparkPi on hadoop-2.0.5-alpha,then I got the "/bin/bash: java: command not found" error in stderr file.
Info got from console:

SPARK_JAR=./core/target/spark-core-assembly-0.6.0.jar ./run spark.deploy.yarn.Client  --jar examples/target/scala-2.9.2/spark-examples_2.9.2-0.6.0.jar  --class spark.examples.SparkPi  --args yarn-standalone  --num-workers 3   --worker-cores 1
13/08/08 05:44:49 INFO yarn.Client: Connecting to ResourceManager atnamemaster/10.0.0.106:9080
13/08/08 05:44:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/08/08 05:44:50 INFO yarn.Client: Got Cluster metric info from ASM, numNodeManagers=3
13/08/08 05:44:50 INFO yarn.Client: Queue info .. queueName=default, queueCurrentCapacity=0.0, queueMaxCapacity=1.0, queueApplicationCount=10, queueChildQueueCount=0
13/08/08 05:44:50 INFO yarn.Client: Requesting new Application
13/08/08 05:44:50 INFO yarn.Client: Got new ApplicationId: application_1375929175938_0011
13/08/08 05:44:50 INFO yarn.Client: Max mem capabililty of resources in this cluster 8192
13/08/08 05:44:50 INFO yarn.Client: Setting up application submission context for ASM
13/08/08 05:44:50 INFO yarn.Client: Preparing Local resources
13/08/08 05:44:50 INFO yarn.Client: Uploading core/target/spark-core-assembly-0.6.0.jar to hdfs://namemaster:9000/user/root/spark/11spark.jar
13/08/08 05:44:56 INFO yarn.Client: Uploading examples/target/scala-2.9.2/spark-examples_2.9.2-0.6.0.jar to hdfs://namemaster:9000/user/root/spark/11app.jar
13/08/08 05:44:56 INFO yarn.Client: Setting up the launch environment
13/08/08 05:44:56 INFO yarn.Client: Setting up container launch context
13/08/08 05:44:56 INFO yarn.Client: Command for the ApplicationMaster: java  -server -Xmx640m  spark.deploy.yarn.ApplicationMaster --class spark.examples.SparkPi --jar examples/target/scala-2.9.2/spark-examples_2.9.2-0.6.0.jar --args  'yarn-standalone'  --worker-memory 1024 --worker-cores 1 --num-workers 3 1> <LOG_DIR>/stdout 2> <LOG_DIR>/stderr
13/08/08 05:44:56 INFO yarn.Client: Submitting application to ASM
13/08/08 05:44:57 INFO yarn.Client: Application report from ASM:
         application identifier: application_1375929175938_0011
         appId: 11
         clientToken: null
         appDiagnostics:
         appMasterHost: N/A
         appQueue: default
         appMasterRpcPort: 0
         appStartTime: 1375940696450
         yarnAppState: ACCEPTED
         distributedFinalState: UNDEFINED
         appTrackingUrl: namemaster:8088/proxy/application_1375929175938_0011/
         appUser: root
13/08/08 05:44:58 INFO yarn.Client: Application report from ASM:
         application identifier: application_1375929175938_0011
         appId: 11
         clientToken: null
         appDiagnostics: Application application_1375929175938_0011 failed 1 times due to AM Container for appattempt_1375929175938_0011_000001 exited with  exitCode: 127 due to: .Failing this attempt.. Failing the application.
         appMasterHost: N/A
         appQueue: default
         appMasterRpcPort: 0
         appStartTime: 1375940696450
         yarnAppState: FAILED
         distributedFinalState: FAILED
         appTrackingUrl: namemaster:8088/proxy/application_1375929175938_0011/
         appUser: root

I checked the yarn logs and got the "/bin/bash: java: command not found" error in stderr file.  
But I've already set the JAVA_HOME both in the .bashrc and hadoop-env.sh,and checked the map/reduce examples with no problems,and even in the spark-env.sh.



kunp...@gmail.com

unread,
Aug 8, 2013, 4:17:25 AM8/8/13
to spark...@googlegroups.com
I have already solved the problem by add PATH=$PATH:$JAVA_HOME/bin in the hadoop-env.sh.
   Pretty Cool!
Reply all
Reply to author
Forward
0 new messages