Execution of the example SparkPi in the standalone mode in Eclipse hang

[Matthew Huang]

Hi, I set up a Spark (0.8) standalone cluster (one master and one slave node) in my local server where from 

localhost:8080 

I see master node: 

• URL: spark://127.0.0.1:7077
• Workers: 0
• Cores: 0 Total, 0 Used
• Memory: 0.0 B Total, 0.0 B Used
• Applications: 0 Running, 0 Completed

......

and in the worker node web GUI http://Iocalhost:8081 see:

• ID: worker-20130925193637-localhost-49302
• Master URL: spark://127.0.0.1:7077
• Cores: 4 (0 Used)
• Memory: 3.0 GB (0.0 B Used)

Back to Master

......

I then built the all in one jar spark-assembly-0.8.0-incubating-hadoop1.0.4.jar and put that in my project's Java build path so that I can get the SparkPi.scala compiled.

I used 

val spark = new SparkContext("spark://127.0.0.1:7077", "SparkPi",

System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_EXAMPLES_JAR")))

(if I change from spark://127.0.0.1:7077 to local the code works), when I run this, the program just hang there and give me no result.

The only thing I see is:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

So  I have two problems.

1. I need to get log information so that I can get better understanding of what's going on. I see the StaticLoggerBinder class is indeed in the all-in-one spark jar file too. Regardless, to try, I also added SLF4J implementation (slf4j-simple-1.7.5.jar file) in the build path and get same warning message. Why is this?

2. why the code hang without giving me the PI result? Also, why I cannot see anything happen in the web page for both the master and slave node (I suppose to see some result there as well, right)? I also checked the logs and work folder in my Spark home and didn't see anything too......

Can you help me solve this issue?

Thanks,

Xiaobing

[Matthew Huang]

Ok, I think previously I didn't config the work node correctly. Now with the worker node there, I got the log in Eclipse:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Exception in thread "main" org.apache.spark.SparkException: Job failed: Task 0.0:1 failed more than 4 times

at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:760)

at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:758)

at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)

at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)

at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:758)

at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:379)

at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:441)

at org.apache.spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:149)

And then

in the webui for worker node, I received:

Spark Executor Command: "/Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/bin/java" "-cp" ":/Users/xiaobinghuang/spark-0.8.0-incubating/conf:/Users/xiaobinghuang/spark-0.8.0-incubating/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop1.0.4.jar" "-Xms512M" "-Xmx512M" "org.apache.spark.executor.StandaloneExecutorBackend" "akka://spark@localhost:55641/user/StandaloneScheduler" "0" "localhost" "4"
========================================

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Since SLF4J is still an issue, how do I know what happened?

Thanks,

Matt

[Matthew Huang]

For the SLF impl missing thing, I cornered the issue to Spark folder. Basically, when I run the java execution directly as below:

java -cp /Users/myname/spark-0.8.0-incubating/examples/target/scala-2.9.3/spark-examples-assembly-0.8.0-incubating.jar:/Users/myname/spark-0.8.0-incubating/conf:/Users/myname/spark-0.8.0-incubating/assembly/target/scala-2.9.3/spark-assembly-0.8.0-incubating-hadoop1.0.4.jar:/Users/myname/spark-0.8.0-incubating/lib_managed/slf4j-simple-1.7.5.jar org.apache.spark.examples.SparkPi local[2]

I received the following:

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".

SLF4J: Defaulting to no-operation (NOP) logger implementation

SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.

Pi is roughly 3.14198

Which means I can receive the result but SLF impl is still not in place. I am so confused as slf4j-simple-1.7.5.jar is right at my classpath. 

Can anyone help me with this puzzle? 

Thanks,

Matt

On Friday, September 27, 2013 12:15:30 AM UTC-5, Matthew Huang wrote: