[JIRA] (SPARKR-222) SparkR wordcount example is not working on EMR cluster

126 views
Skip to first unread message

suresh babu lankipalli (JIRA)

unread,
Mar 8, 2015, 11:42:01 AM3/8/15
to spark...@googlegroups.com
suresh babu lankipalli created an issue
 
SparkR / Bug SPARKR-222
SparkR wordcount example is not working on EMR cluster
Issue Type: Bug Bug
Assignee: Unassigned
Created: 08/Mar/15 8:41 AM
Environment:

Hadoop EMR + spark + SparkR

Hadoop version : 2.4.0
Spark version :1.2.1
SparkR build with below command:
USE_YARN=1 SPARK_YARN_VERSION=2.4.0 SPARK_HADOOP_VERSION=2.4.0 ./install-dev.sh

Priority: Major Major
Reporter: suresh babu lankipalli

WARN DAGScheduler: Creating new stage failed due to exception - job: 0
java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:184)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:197)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.getPartitions(RRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.getPartitions(RRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:79)
at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:80)
at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:209)
at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:207)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.dependencies(RDD.scala:207)
at org.apache.spark.scheduler.DAGScheduler.visit$1(DAGScheduler.scala:301)
at org.apache.spark.scheduler.DAGScheduler.getParentStages(DAGScheduler.scala:313)
at org.apache.spark.scheduler.DAGScheduler.newStage(DAGScheduler.scala:247)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:734)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1389)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 45 more
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 50 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
... 52 more
collect on 16 failed with java.lang.reflect.InvocationTargetException
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:107)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:60)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:22)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Error in configuring object
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at org.apache.spark.rdd.HadoopRDD.getInputFormat(HadoopRDD.scala:184)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:197)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.getPartitions(RRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.getPartitions(RRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:222)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:220)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:220)
at org.apache.spark.ShuffleDependency.<init>(Dependency.scala:79)
at org.apache.spark.rdd.ShuffledRDD.getDependencies(ShuffledRDD.scala:80)
at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:209)
at org.apache.spark.rdd.RDD$$anonfun$dependencies$2.apply(RDD.scala:207)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.dependencies(RDD.scala:207)
at org.apache.spark.scheduler.DAGScheduler.visit$1(DAGScheduler.scala:301)
at org.apache.spark.scheduler.DAGScheduler.getParentStages(DAGScheduler.scala:313)
at org.apache.spark.scheduler.DAGScheduler.newStage(DAGScheduler.scala:247)
at org.apache.spark.scheduler.DAGScheduler.handleJobSubmitted(DAGScheduler.scala:734)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1389)
at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
... 45 more
Caused by: java.lang.IllegalArgumentException: Compression codec com.hadoop.compression.lzo.LzoCodec not found.
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:135)
at org.apache.hadoop.io.compress.CompressionCodecFactory.<init>(CompressionCodecFactory.java:175)
at org.apache.hadoop.mapred.TextInputFormat.configure(TextInputFormat.java:45)
... 50 more
Caused by: java.lang.ClassNotFoundException: Class com.hadoop.compression.lzo.LzoCodec not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.io.compress.CompressionCodecFactory.getCodecClasses(CompressionCodecFactory.java:128)
... 52 more
Error: returnStatus == 0 is not TRUE
Execution halted
Stopping SparkR

Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4-OD-15-055#64014-sha1:797c660)
Atlassian logo

Shivaram Venkataraman (JIRA)

unread,
Mar 8, 2015, 4:08:03 PM3/8/15
to spark...@googlegroups.com
Shivaram Venkataraman commented on Bug SPARKR-222
 
Re: SparkR wordcount example is not working on EMR cluster

I don't know much about EMR, but the problems seems to come "Compression codec com.hadoop.compression.lzo.LzoCodec not found". AFAIK Lzo is not distributed with Hadoop but requires import as a native library. Would you happen to know if any flags are required to do this when using Hadoop / Spark in Scala or Python

suresh babu lankipalli (JIRA)

unread,
Mar 8, 2015, 4:14:01 PM3/8/15
to spark...@googlegroups.com

The below command I run on SparkR
YARN_CONF_DIR=/home/hadoop/conf/ ./sparkR examples/wordcount.R yarn-client s3n://opt.agero/data/file.txt

My hadoop Core-site.xml has below configuration, I am still not understand why SparkR can't able to find Lzo compression

<property><name>io.compression.codecs</name><value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec,org.apache.hadoop.io.compress.BZip2Codec,org.apache.hadoop.io.compress.SnappyCodec</value></property>

suresh babu lankipalli (JIRA)

unread,
Mar 8, 2015, 4:17:01 PM3/8/15
to spark...@googlegroups.com

My Spark-env.sh file configuraion has this : In this configuration I am pointing hadoop-lzo.jar file, but still throwing error.

xport HADOOP_HOME="/home/hadoop"
export SPARK_DAEMON_JAVA_OPTS="-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70"
export SPARK_LOCAL_DIRS=/mnt/spark
export SPARK_LOG_DIR=/mnt/var/log/apps
export SPARK_CLASSPATH="/home/hadoop/spark/conf:/home/hadoop/conf:/home/hadoop/spark/classpath/emr/:/home/hadoop/spark/classpath/emrfs/:/home/hadoop/share/hadoop/common/lib/*:/home/hadoop/share/hadoop/common/lib/hadoop-lzo.jar"

Shivaram Venkataraman (JIRA)

unread,
Mar 8, 2015, 6:33:02 PM3/8/15
to spark...@googlegroups.com

I think the problem is that the core-site.xm and spark-env.sh aren't being picked up. Can you try to use the sparkR-submit script as described in https://github.com/amplab-extras/SparkR-pkg/blob/master/README.md#running-on-a-cluster-using-sparkr-submit ? It has support to pick up the same arguments as spark-submit and you can also pass in extra jars using --jars etc.

suresh babu lankipalli (JIRA)

unread,
Mar 8, 2015, 8:05:02 PM3/8/15
to spark...@googlegroups.com

I followed steps from the link you provided: getting the below exception.

export SPARK_HOME=/home/hadoop/spark
export YARN_CONF_DIR=/home/hadoop/conf
export JAVA_HOME=export JAVA_HOME=/usr/java/latest
/home/hadoop/SparkR-pkg/lib/SparkR/sparkR-submit --master yarn-client /home/hadoop/SparkR-pkg/examples/wordcount.R yarn-client s3n://bucketname/data/file.txt

23:54:12 INFO metrics.MetricsSaver: Saved 4:17 records to /mnt/var/em/raw/i-2c81f6d6_20150308_SparkSubmit_29443_raw.bin
15/03/08 23:54:13 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, ip-10-168-80-20-.ec2.internal): java.net.SocketTimeoutException: Accept timed out
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:50)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/03/08 23:54:13 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 2, ip-10-168-80-20-.ec2.internal, PROCESS_LOCAL, 1346 bytes)
15/03/08 23:54:13 INFO scheduler.TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on executor ip-10-168-114-96.ec2.internal: java.net.SocketTimeoutException (Accept timed out) [duplicate 1]
15/03/08 23:54:13 INFO scheduler.TaskSetManager: Starting task 1.1 in stage 0.0 (TID 3, ip-10-168-114-96.ec2.internal, PROCESS_LOCAL, 1346 bytes)
15/03/08 23:54:24 WARN scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 2, ip-10-168-80-20-.ec2.internal): java.net.SocketTimeoutException: Accept timed out
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:50)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/03/08 23:54:24 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 0.0 (TID 4, ip-10-168-80-20-.ec2.internal, PROCESS_LOCAL, 1346 bytes)
15/03/08 23:54:24 INFO scheduler.TaskSetManager: Lost task 1.1 in stage 0.0 (TID 3) on executor ip-10-168-114-96.ec2.internal: java.net.SocketTimeoutException (Accept timed out) [duplicate 1]
15/03/08 23:54:24 INFO scheduler.TaskSetManager: Starting task 1.2 in stage 0.0 (TID 5, ip-10-168-114-96.ec2.internal, PROCESS_LOCAL, 1346 bytes)
15/03/08 23:54:34 WARN scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 4, ip-10-168-80-20-.ec2.internal): java.net.SocketTimeoutException: Accept timed out
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:50)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/03/08 23:54:34 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 0.0 (TID 6, ip-10-168-80-20-.ec2.internal, PROCESS_LOCAL, 1346 bytes)
15/03/08 23:54:34 INFO scheduler.TaskSetManager: Lost task 1.2 in stage 0.0 (TID 5) on executor ip-10-168-114-96.ec2.internal: java.net.SocketTimeoutException (Accept timed out) [duplicate 1]
15/03/08 23:54:34 INFO scheduler.TaskSetManager: Starting task 1.3 in stage 0.0 (TID 7, ip-10-168-114-96.ec2.internal, PROCESS_LOCAL, 1346 bytes)
15/03/08 23:54:42 INFO metrics.MetricsSaver: Saved 8:15 records to /mnt/var/em/raw/i-2c81f6d6_20150308_SparkSubmit_29443_raw.bin
15/03/08 23:54:44 WARN scheduler.TaskSetManager: Lost task 0.3 in stage 0.0 (TID 6, ip-10-168-80-20-.ec2.internal): java.net.SocketTimeoutException: Accept timed out
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:50)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

15/03/08 23:54:44 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
15/03/08 23:54:44 INFO cluster.YarnClientClusterScheduler: Cancelling stage 0
15/03/08 23:54:44 INFO cluster.YarnClientClusterScheduler: Stage 0 was cancelled
15/03/08 23:54:44 INFO scheduler.DAGScheduler: Job 0 failed: collect at NativeMethodAccessorImpl.java:-2, took 44.762929 s

Caused by: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, ip-10-168-80-20-.ec2.internal): java.net.SocketTimeoutException: Accept timed out
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:50)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at edu.berkeley.cs.amplab.sparkr.BaseRRDD.compute(RRDD.scala:33)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:280)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:247)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:56)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:200)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1214)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1203)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1202)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1202)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:696)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:696)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1420)


at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
at org.apache.spark.scheduler.DAGSchedulerEventProcessActor.aroundReceive(DAGScheduler.scala:1375)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
at akka.actor.ActorCell.invoke(ActorCell.scala:487)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
at akka.dispatch.Mailbox.run(Mailbox.scala:220)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

Error: returnStatus == 0 is not TRUE
Execution halted

Shivaram Venkataraman (JIRA)

unread,
Mar 9, 2015, 12:08:02 AM3/9/15
to spark...@googlegroups.com

Do you have the Spark executor logs for this run ? From the error message it looks like the workers couldn't start R, but one will need to see the executor logs to make sure.

suresh babu lankipalli (JIRA)

unread,
Mar 9, 2015, 4:25:03 PM3/9/15
to spark...@googlegroups.com

I didn't found anything in logs, can you please guide me for Spark logs path, where I can found these logs on cluster.

suresh babu lankipalli (JIRA)

unread,
Mar 9, 2015, 4:26:03 PM3/9/15
to spark...@googlegroups.com
suresh babu lankipalli edited a comment on Bug SPARKR-222
I didn't found anything in logs, can you please guide me for Spark logs path, where I can found these logs on cluster.

Path I checked : /mnt/var/log/hadoop

suresh babu lankipalli (JIRA)

unread,
Mar 9, 2015, 4:58:02 PM3/9/15
to spark...@googlegroups.com

stderr log not showing any exception's :

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0-amzn-3/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/46/spark-assembly-1.2.1-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/09 20:33:32 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/03/09 20:33:33 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1425166635254_0645_000001
15/03/09 20:33:34 INFO spark.SecurityManager: Changing view acls to: hadoop
15/03/09 20:33:34 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/03/09 20:33:34 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/03/09 20:33:35 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/03/09 20:33:35 INFO Remoting: Starting remoting
15/03/09 20:33:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark...@ip-10-166-125-72.ec2.internal:40282]
15/03/09 20:33:35 INFO util.Utils: Successfully started service 'sparkYarnAM' on port 40282.
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Driver now available: ip-10-37-194-88.ec2.internal:41514
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://spark...@ip-10-37-194-88.ec2.internal:41514/user/YarnScheduler
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> 10.37.194.88, PROXY_URI_BASES -> http://10.37.194.88:9046/proxy/application_1425166635254_0645),/proxy/application_1425166635254_0645)
15/03/09 20:33:35 INFO client.RMProxy: Connecting to ResourceManager at /10.37.194.88:9024
15/03/09 20:33:35 INFO yarn.YarnRMClientImpl: Registering the ApplicationMaster
15/03/09 20:33:35 INFO yarn.YarnAllocationHandler: Will allocate 2 executor containers, each with 1408 MB memory including 384 MB overhead
15/03/09 20:33:35 INFO yarn.YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:1408, vCores:1>
15/03/09 20:33:35 INFO yarn.YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:1408, vCores:1>
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/03/09 20:33:40 INFO impl.AMRMClientImpl: Received new token for : ip-10-166-125-72.ec2.internal:9103
15/03/09 20:33:40 INFO impl.AMRMClientImpl: Received new token for : ip-10-168-114-96.ec2.internal:9103
15/03/09 20:33:40 INFO util.RackResolver: Resolved ip-10-166-125-72.ec2.internal to /default-rack
15/03/09 20:33:40 INFO util.RackResolver: Resolved ip-10-168-114-96.ec2.internal to /default-rack
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching container container_1425166635254_0645_01_000002 for on host ip-10-166-125-72.ec2.internal
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark...@ip-10-37-194-88.ec2.internal:41514/user/CoarseGrainedScheduler, executorHostname: ip-10-166-125-72.ec2.internal
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching container container_1425166635254_0645_01_000003 for on host ip-10-168-114-96.ec2.internal
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://spark...@ip-10-37-194-88.ec2.internal:41514/user/CoarseGrainedScheduler, executorHostname: ip-10-168-114-96.ec2.internal
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Starting Executor Container
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Starting Executor Container
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Preparing Local resources
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Preparing Local resources
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Prepared Local resources Map(_spark_.jar -> resource

{ scheme: "hdfs" host: "ec2-54-221-177-108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar" }

size: 127193956 timestamp: 1425933210291 type: FILE visibility: PRIVATE, wordcount.R -> resource

{ scheme: "hdfs" host: "ec2-54-221-177-108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R" }

size: 575 timestamp: 1425933210363 type: FILE visibility: PRIVATE)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Prepared Local resources Map(_spark_.jar -> resource

{ scheme: "hdfs" host: "ec2-54-221-177-108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar" }

size: 127193956 timestamp: 1425933210291 type: FILE visibility: PRIVATE, wordcount.R -> resource

{ scheme: "hdfs" host: "ec2-54-221-177-108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R" }

size: 575 timestamp: 1425933210363 type: FILE visibility: PRIVATE)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> /home/hadoop/spark/conf:/home/hadoop/conf:/home/hadoop/spark/classpath/emr/:/home/hadoop/spark/classpath/emrfs/:/home/hadoop/share/hadoop/common/lib/:/home/hadoop/share/hadoop/common/lib/hadoop-lzo.jar:$PWD:$PWD/_spark.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/:$HADOOP_COMMON_HOME/share/hadoop/common/lib/:$HADOOP_HDFS_HOME/share/hadoop/hdfs/:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/:$HADOOP_YARN_HOME/share/hadoop/yarn/:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/:/usr/share/aws/emr/emrfs/lib/:/usr/share/aws/emr/lib/:/usr/share/aws/emr/auxlib/:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/:/usr/share/aws/emr/emrfs/lib/:/usr/share/aws/emr/lib/:/usr/share/aws/emr/auxlib/:$PWD/app.jar:$PWD/, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1425166635254_0645/, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 127193956,575, SPARK_USER -> hadoop, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1425933210291,1425933210363, LD_LIBRARY_PATH -> $LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/local/lib64:/usr/lib64:/home/hadoop/lib/native:/usr/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib:, SPARK_YARN_CACHE_FILES -> hdfs://ec2-54-221-177-108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar#spark_.jar,hdfs://ec2-54-221-177-108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R#wordcount.R)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> /home/hadoop/spark/conf:/home/hadoop/conf:/home/hadoop/spark/classpath/emr/:/home/hadoop/spark/classpath/emrfs/:/home/hadoop/share/hadoop/common/lib/:/home/hadoop/share/hadoop/common/lib/hadoop-lzo.jar:$PWD:$PWD/_spark.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/:$HADOOP_COMMON_HOME/share/hadoop/common/lib/:$HADOOP_HDFS_HOME/share/hadoop/hdfs/:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/:$HADOOP_YARN_HOME/share/hadoop/yarn/:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/:/usr/share/aws/emr/emrfs/lib/:/usr/share/aws/emr/lib/:/usr/share/aws/emr/auxlib/:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/:/usr/share/aws/emr/emrfs/lib/:/usr/share/aws/emr/lib/:/usr/share/aws/emr/auxlib/:$PWD/app.jar:$PWD/, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1425166635254_0645/, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 127193956,575, SPARK_USER -> hadoop, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1425933210291,1425933210363, LD_LIBRARY_PATH -> $LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/local/lib64:/usr/lib64:/home/hadoop/lib/native:/usr/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib:, SPARK_YARN_CACHE_FILES -> hdfs://ec2-54-221-177-108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar#spark_.jar,hdfs://ec2-54-221-177-108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R#wordcount.R)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m -Xmx1024m , -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70, -Djava.io.tmpdir=$PWD/tmp, '-Dspark.driver.port=41514', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark...@ip-10-37-194-88.ec2.internal:41514/user/CoarseGrainedScheduler, 2, ip-10-168-114-96.ec2.internal, 1, application_1425166635254_0645, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m -Xmx1024m , -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70, -Djava.io.tmpdir=$PWD/tmp, '-Dspark.driver.port=41514', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://spark...@ip-10-37-194-88.ec2.internal:41514/user/CoarseGrainedScheduler, 1, ip-10-166-125-72.ec2.internal, 1, application_1425166635254_0645, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-10-168-114-96.ec2.internal:9103
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-10-166-125-72.ec2.internal:9103
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://spark...@ip-10-166-125-72.ec2.internal:40282] -> [akka.tcp://spark...@ip-10-37-194-88.ec2.internal:41514]
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
15/03/09 20:34:42 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1425166635254_0645

suresh babu lankipalli (JIRA)

unread,
Mar 9, 2015, 5:01:04 PM3/9/15
to spark...@googlegroups.com
suresh babu lankipalli edited a comment on Bug SPARKR-222
stderr log not showing any exception's : 


SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/home/hadoop/.versions/2.4.0-amzn-3/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/mnt/var/lib/hadoop/tmp/nm-local-dir/usercache/hadoop/filecache/46/spark-assembly-1.2.1-hadoop2.4.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
15/03/09 20:33:32 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT]
15/03/09 20:33:33 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1425166635254_0645_000001
15/03/09 20:33:34 INFO spark.SecurityManager: Changing view acls to: hadoop
15/03/09 20:33:34 INFO spark.SecurityManager: Changing modify acls to: hadoop
15/03/09 20:33:34 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify permissions: Set(hadoop)
15/03/09 20:33:35 INFO slf4j.Slf4jLogger: Slf4jLogger started
15/03/09 20:33:35 INFO Remoting: Starting remoting
15/03/09 20:33:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkYarnAM@ip- 10 11 - 166-125- 72.ec2.internal:40282]

15/03/09 20:33:35 INFO util.Utils: Successfully started service 'sparkYarnAM' on port 40282.
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Waiting for Spark driver to be reachable.
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Driver now available: ip- 10 11 - 37-194- 88.ec2.internal:41514
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Listen to driver: akka.tcp://sparkDriver@ip-
10 11 - 37-194- 88.ec2.internal:41514/user/YarnScheduler

15/03/09 20:33:35 INFO yarn.ApplicationMaster: Add WebUI Filter. AddWebUIFilter(org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter,Map(PROXY_HOSTS -> 10.37.194.88, PROXY_URI_BASES -> http://10.37.194.88:9046/proxy/application_1425166635254_0645),/proxy/application_1425166635254_0645)
15/03/09 20:33:35 INFO client.RMProxy: Connecting to ResourceManager at /10.37.194.88:9024
15/03/09 20:33:35 INFO yarn.YarnRMClientImpl: Registering the ApplicationMaster
15/03/09 20:33:35 INFO yarn.YarnAllocationHandler: Will allocate 2 executor containers, each with 1408 MB memory including 384 MB overhead
15/03/09 20:33:35 INFO yarn.YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:1408, vCores:1>
15/03/09 20:33:35 INFO yarn.YarnAllocationHandler: Container request (host: Any, priority: 1, capability: <memory:1408, vCores:1>
15/03/09 20:33:35 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000
15/03/09 20:33:40 INFO impl.AMRMClientImpl: Received new token for : ip- 10 11 - 166-125- 72.ec2.internal:9103
15/03/09 20:33:40 INFO impl.AMRMClientImpl: Received new token for : ip-
10 12 - 168- 114-96.ec2.internal:9103
15/03/09 20:33:40 INFO util.RackResolver: Resolved ip-
10 11 - 166-125- 72.ec2.internal to /default-rack
15/03/09 20:33:40 INFO util.RackResolver: Resolved ip-
10 12 - 168- 114-96.ec2.internal to /default-rack
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching container container_1425166635254_0645_01_000002 for on host ip-
10 11 - 166-125- 72.ec2.internal
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@ip-
10 11 - 37-194- 88.ec2.internal:41514/user/CoarseGrainedScheduler,  executorHostname: ip- 10 11 - 166-125- 72.ec2.internal
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching container container_1425166635254_0645_01_000003 for on host ip-
10 12 - 168- 114-96.ec2.internal
15/03/09 20:33:40 INFO yarn.YarnAllocationHandler: Launching ExecutorRunnable. driverUrl: akka.tcp://sparkDriver@ip-
10 11 - 37-194- 88.ec2.internal:41514/user/CoarseGrainedScheduler,  executorHostname: ip- 10 12 - 168- 114-96.ec2.internal

15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Starting Executor Container
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Starting Executor Container
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: yarn.client.max-nodemanagers-proxies : 500
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up ContainerLaunchContext
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Preparing Local resources
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Preparing Local resources
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "ec2- 54 12 - 221-177- 108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar" } size: 127193956 timestamp: 1425933210291 type: FILE visibility: PRIVATE, wordcount.R -> resource { scheme: "hdfs" host: "ec2- 54 12 - 221-177- 108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R" } size: 575 timestamp: 1425933210363 type: FILE visibility: PRIVATE)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Prepared Local resources Map(__spark__.jar -> resource { scheme: "hdfs" host: "ec2-
54 12 - 221-177- 108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar" } size: 127193956 timestamp: 1425933210291 type: FILE visibility: PRIVATE, wordcount.R -> resource { scheme: "hdfs" host: "ec2- 54 12 - 221-177- 108.compute-1.amazonaws.com" port: 9000 file: "/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R" } size: 575 timestamp: 1425933210363 type: FILE visibility: PRIVATE)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> /home/hadoop/spark/conf:/home/hadoop/conf:/home/hadoop/spark/classpath/emr/*:/home/hadoop/spark/classpath/emrfs/*:/home/hadoop/share/hadoop/common/lib/*:/home/hadoop/share/hadoop/common/lib/hadoop-lzo.jar:$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$PWD/__app__.jar:$PWD/*, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1425166635254_0645/, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 127193956,575, SPARK_USER -> hadoop, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1425933210291,1425933210363, LD_LIBRARY_PATH -> $LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/local/lib64:/usr/lib64:/home/hadoop/lib/native:/usr/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib:, SPARK_YARN_CACHE_FILES -> hdfs://ec2-
54 12 - 221-177- 108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar#__spark__.jar,hdfs://ec2- 54 12 - 221-177- 108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R#wordcount.R)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with environment: Map(CLASSPATH -> /home/hadoop/spark/conf:/home/hadoop/conf:/home/hadoop/spark/classpath/emr/*:/home/hadoop/spark/classpath/emrfs/*:/home/hadoop/share/hadoop/common/lib/*:/home/hadoop/share/hadoop/common/lib/hadoop-lzo.jar:$PWD:$PWD/__spark__.jar:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/lib/*:/usr/share/aws/emr/auxlib/*:$PWD/__app__.jar:$PWD/*, SPARK_YARN_STAGING_DIR -> .sparkStaging/application_1425166635254_0645/, SPARK_YARN_CACHE_FILES_FILE_SIZES -> 127193956,575, SPARK_USER -> hadoop, SPARK_YARN_CACHE_FILES_VISIBILITIES -> PRIVATE,PRIVATE, SPARK_YARN_MODE -> true, SPARK_YARN_CACHE_FILES_TIME_STAMPS -> 1425933210291,1425933210363, LD_LIBRARY_PATH -> $LD_LIBRARY_PATH:/usr/lib64/R/lib:/usr/local/lib64:/usr/lib64:/home/hadoop/lib/native:/usr/lib64:/usr/local/cuda/lib64:/usr/local/cuda/lib:, SPARK_YARN_CACHE_FILES -> hdfs://ec2-
54 12 - 221-177- 108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/spark-assembly-1.2.1-hadoop2.4.0.jar#__spark__.jar,hdfs://ec2- 54 12 - 221-177- 108.compute-1.amazonaws.com:9000/user/hadoop/.sparkStaging/application_1425166635254_0645/wordcount.R#wordcount.R)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m -Xmx1024m , -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70, -Djava.io.tmpdir=$PWD/tmp, '-Dspark.driver.port=41514', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://sparkDriver@ip-
10 11 - 37-194- 88.ec2.internal:41514/user/CoarseGrainedScheduler, 2, ip- 10 12 - 168- 114-96.ec2.internal, 1, application_1425166635254_0645, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/03/09 20:33:40 INFO yarn.ExecutorRunnable: Setting up executor with commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill %p', -Xms1024m -Xmx1024m , -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70, -Djava.io.tmpdir=$PWD/tmp, '-Dspark.driver.port=41514', -Dspark.yarn.app.container.log.dir=<LOG_DIR>, org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://sparkDriver@ip-
10 11 - 37-194- 88.ec2.internal:41514/user/CoarseGrainedScheduler, 1, ip- 10 11 - 166-125- 72.ec2.internal, 1, application_1425166635254_0645, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-
10 12 - 168- 114-96.ec2.internal:9103
15/03/09 20:33:40 INFO impl.ContainerManagementProtocolProxy: Opening proxy : ip-
10 11 - 166-125- 72.ec2.internal:9103
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Driver terminated or disconnected! Shutting down. Disassociated [akka.tcp://sparkYarnAM@ip-
10 11 - 166-125- 72.ec2.internal:40282] -> [akka.tcp://sparkDriver@ip- 10 11 - 37-194- 88.ec2.internal:41514]

15/03/09 20:34:42 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
15/03/09 20:34:42 INFO impl.AMRMClientImpl: Waiting for application to be successfully unregistered.
15/03/09 20:34:42 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1425166635254_0645

Shivaram Venkataraman (JIRA)

unread,
Mar 9, 2015, 5:01:10 PM3/9/15
to spark...@googlegroups.com

These seem to be the YARN logs – I was trying to look for the err logs from the `CoarseGrainedExecutorBackend`

suresh babu lankipalli (JIRA)

unread,
Mar 11, 2015, 10:28:03 AM3/11/15
to spark...@googlegroups.com

My WordCount running successfully without any exception:
INFO scheduler.DAGScheduler: Job 0 finished: collect at NativeMethodAccessorImpl.java:-2, took 7.322869 s
exposed : 1
dispatching, : 1
machines. : 1
referencing : 1
collection : 1
coarse-grained : 1
because : 1
transformations : 1
complexity : 1
abstraction : 2
provides : 1
created : 1
applying : 1
manipulate : 1
Python, : 1
scheduling, : 1
language-integrated : 1
systems, : 1
This : 1
data : 1
is : 4
API : 1
Java, : 1
Scala : 1
RDD : 1
basic : 1
data. : 1
can : 1
overall : 1
local, : 1
Distributed : 1
through : 1
called : 1
across : 1
simplifies : 1
Resilient : 1
external : 1
storage : 1
foundation : 1
existing : 1
manipulating : 1
functionalities. : 1
project. : 1
programming : 2
local : 1
the : 3
way : 1
partitioned : 1
distributed : 1
collections. : 1
Datasets, : 1
datasets : 1
collections : 1
similar : 2
Core : 1
fundamental : 1
It : 1
RDDs : 2
be : 1
by : 2
in : 2
logical : 1
of : 3
on : 1
task : 1
or : 1
to : 2
in-process : 1
I/O : 1
RDDs. : 1
Spark : 1
The : 2
applications : 1
and : 1
a : 2
Stopping SparkR

suresh babu lankipalli (JIRA)

unread,
Mar 11, 2015, 10:32:01 AM3/11/15
to spark...@googlegroups.com

I Build another EMR Cluster (2.4.0) + spark (1.2.1) +SparkR (latest code from github), I build this cluster same as previous one.
when I try to run Wordcount example getting Below error No clue whats causing the problem.

[nioEventLoopGroup-2-2] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_0_piece0
2015-03-11 14:19:21,562 INFO [nioEventLoopGroup-2-2] spark.SparkContext (Logging.scala:logInfo(59)) - Created broadcast 0 from textFile at NativeMethodAccessorImpl.java:-2
<init> on edu.berkeley.cs.amplab.sparkr.RRDD failed with java.util.NoSuchElementException: next on empty iterator
java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
at scala.collection.IterableLike$class.head(IterableLike.scala:91)
at scala.collection.mutable.ArrayOps$ofRef.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:108)
at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
at scala.collection.mutable.ArrayOps$ofRef.head(ArrayOps.scala:108)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.handleMethodCall(SparkRBackendHandler.scala:116)


at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:60)
at edu.berkeley.cs.amplab.sparkr.SparkRBackendHandler.channelRead0(SparkRBackendHandler.scala:22)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:163)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:333)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:319)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:787)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:130)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137)
at java.lang.Thread.run(Thread.java:745)

Error: returnStatus == 0 is not TRUE
Execution halted

Shivaram Venkataraman (JIRA)

unread,
Mar 13, 2015, 2:42:01 AM3/13/15
to spark...@googlegroups.com

suresh babu lankipalli Its hard to make out what the error could be from the stack trace, but my guess is that there might be some Spark version mismatch (i.e. Spark version installed on cluster vs. version linked to SparkR).

Prateek Baranwal (JIRA)

unread,
Apr 7, 2015, 9:32:02 PM4/7/15
to spark...@googlegroups.com

@shivaram
I am hitting the same issue. It's a yarn grid for me. Can you elaborate spark version installed on cluster there is none hence the assembly jar. What else can I do for diagnostic. Executor logs has R cycling again and again

This message was sent by Atlassian JIRA (v6.4-OD-16-006#64014-sha1:e47e3fa)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages