On iteration 1
13/06/17 15:20:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
13/06/17 15:20:58 WARN snappy.LoadSnappy: Snappy native library not loaded
13/06/17 15:20:58 INFO mapred.FileInputFormat: Total input paths to process : 1
13/06/17 15:20:58 INFO spark.SparkContext: Starting job: reduce at SparkLRhdfs2.scala:46
13/06/17 15:20:58 INFO scheduler.DAGScheduler: Got job 0 (reduce at SparkLRhdfs2.scala:46) with 12 output partitions (allowLocal=false)
13/06/17 15:20:58 INFO scheduler.DAGScheduler: Final stage: Stage 0 (map at SparkLRhdfs2.scala:44)
13/06/17 15:20:58 INFO scheduler.DAGScheduler: Parents of final stage: List()
13/06/17 15:20:58 INFO scheduler.DAGScheduler: Missing parents: List()
13/06/17 15:20:58 INFO scheduler.DAGScheduler: Submitting Stage 0 (MappedRDD[3] at map at SparkLRhdfs2.scala:44), which has no missing parents
13/06/17 15:20:58 INFO scheduler.DAGScheduler: Submitting 12 missing tasks from Stage 0 (MappedRDD[3] at map at SparkLRhdfs2.scala:44)
13/06/17 15:20:58 INFO local.LocalScheduler: Running ResultTask(0, 0)
13/06/17 15:20:58 INFO local.LocalScheduler: Size of task 0 is 1871 bytes
13/06/17 15:20:58 INFO local.LocalScheduler: Fetching http://10.139.8.186:38584/jars/spark-tutorial_2.9.2-0.1.jar with timestamp 1371482457534
13/06/17 15:20:58 INFO spark.Utils: Fetching http://10.139.8.186:38584/jars/spark-tutorial_2.9.2-0.1.jar to /tmp/fetchFileTemp1929838124673535594.tmp
13/06/17 15:20:58 INFO local.LocalScheduler: Adding file:/tmp/spark-7ba1d208-42e5-4c0f-aebc-d86687f5e83c/spark-tutorial_2.9.2-0.1.jar to class loader
13/06/17 15:20:58 INFO spark.CacheManager: Cache key is rdd_2_0
13/06/17 15:20:58 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@691
13/06/17 15:21:04 INFO storage.MemoryStore: ensureFreeSpace(120176024) called with curMem=33704, maxMem=994317434
13/06/17 15:21:04 INFO storage.MemoryStore: Block rdd_2_0 stored as values to memory (estimated size 114.6 MB, free 833.6 MB)
13/06/17 15:21:04 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 833.6 MB)
13/06/17 15:21:04 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:21:05 INFO local.LocalScheduler: Finished ResultTask(0, 0)
13/06/17 15:21:05 INFO local.LocalScheduler: Running ResultTask(0, 1)
13/06/17 15:21:05 INFO local.LocalScheduler: Size of task 1 is 1871 bytes
13/06/17 15:21:05 INFO scheduler.DAGScheduler: Completed ResultTask(0, 0)
13/06/17 15:21:05 INFO spark.CacheManager: Cache key is rdd_2_1
13/06/17 15:21:05 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@692
13/06/17 15:21:11 INFO storage.MemoryStore: ensureFreeSpace(120176024) called with curMem=120209728, maxMem=994317434
13/06/17 15:21:11 INFO storage.MemoryStore: Block rdd_2_1 stored as values to memory (estimated size 114.6 MB, free 719.0 MB)
13/06/17 15:21:11 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 719.0 MB)
13/06/17 15:21:11 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:21:11 INFO local.LocalScheduler: Finished ResultTask(0, 1)
13/06/17 15:21:11 INFO scheduler.DAGScheduler: Completed ResultTask(0, 1)
13/06/17 15:21:11 INFO local.LocalScheduler: Running ResultTask(0, 2)
13/06/17 15:21:11 INFO local.LocalScheduler: Size of task 2 is 1871 bytes
13/06/17 15:21:11 INFO spark.CacheManager: Cache key is rdd_2_2
13/06/17 15:21:11 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@693
13/06/17 15:21:19 INFO storage.MemoryStore: ensureFreeSpace(120176024) called with curMem=240385752, maxMem=994317434
13/06/17 15:21:19 INFO storage.MemoryStore: Block rdd_2_2 stored as values to memory (estimated size 114.6 MB, free 604.4 MB)
13/06/17 15:21:19 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 604.4 MB)
13/06/17 15:21:19 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:21:19 INFO local.LocalScheduler: Finished ResultTask(0, 2)
13/06/17 15:21:19 INFO scheduler.DAGScheduler: Completed ResultTask(0, 2)
13/06/17 15:21:19 INFO local.LocalScheduler: Running ResultTask(0, 3)
13/06/17 15:21:19 INFO local.LocalScheduler: Size of task 3 is 1871 bytes
13/06/17 15:21:19 INFO spark.CacheManager: Cache key is rdd_2_3
13/06/17 15:21:19 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@694
13/06/17 15:21:30 INFO storage.MemoryStore: ensureFreeSpace(120176024) called with curMem=360561776, maxMem=994317434
13/06/17 15:21:30 INFO storage.MemoryStore: Block rdd_2_3 stored as values to memory (estimated size 114.6 MB, free 489.8 MB)
13/06/17 15:21:30 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:21:30 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:21:31 INFO local.LocalScheduler: Finished ResultTask(0, 3)
13/06/17 15:21:31 INFO scheduler.DAGScheduler: Completed ResultTask(0, 3)
13/06/17 15:21:31 INFO local.LocalScheduler: Running ResultTask(0, 4)
13/06/17 15:21:31 INFO local.LocalScheduler: Size of task 4 is 1871 bytes
13/06/17 15:21:31 INFO spark.CacheManager: Cache key is rdd_2_4
13/06/17 15:21:31 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@695
13/06/17 15:23:10 WARN storage.BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, ip-10-139-8-186.ec2.internal, 47180) with no recent heart beats
13/06/17 15:23:16 WARN storage.BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, ip-10-139-8-186.ec2.internal, 34113) with no recent heart beats
13/06/17 15:23:31 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:23:32 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:23:33 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager ip-10-139-8-186.ec2.internal:47180 with 901.1 MB RAM
13/06/17 15:23:36 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:23:36 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Trying to register BlockManager
Uncaught error from thread [spark-akka.actor.default-dispatcher-65] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
java.lang.OutOfMemoryError: Java heap space
        at akka.jsr166y.ForkJoinPool$WorkQueue.growArray(ForkJoinPool.java:846)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)

Exception: sbt.TrapExitSecurityException thrown from the UncaughtExceptionHandler in thread "spark-akka.actor.default-dispatcher-65"
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:23:36 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 1 attempts
java.util.concurrent.TimeoutException: Futures timed out after [10000] milliseconds
        at akka.dispatch.DefaultPromise.ready(Future.scala:870)
        at akka.dispatch.DefaultPromise.result(Future.scala:874)
        at akka.dispatch.Await$.result(Future.scala:74)
        at spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:154)
        at spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:57)
        at spark.storage.BlockManager.spark$storage$BlockManager$$heartBeat(BlockManager.scala:100)
        at spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:127)
        at akka.actor.DefaultScheduler$$anon$1.run(Scheduler.scala:142)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94)
        at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
13/06/17 15:23:36 INFO storage.BlockManager: Reporting 8 blocks to the master.
13/06/17 15:23:36 INFO storage.BlockManager: Reporting 8 blocks to the master.
13/06/17 15:23:36 INFO storage.BlockManager: Reporting 8 blocks to the master.
13/06/17 15:23:36 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 1 attempts
java.util.concurrent.TimeoutException: Futures timed out after [10000] milliseconds
        at akka.dispatch.DefaultPromise.ready(Future.scala:870)
        at akka.dispatch.DefaultPromise.result(Future.scala:874)
        at akka.dispatch.Await$.result(Future.scala:74)
        at spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:154)
        at spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:57)
        at spark.storage.BlockManager.spark$storage$BlockManager$$heartBeat(BlockManager.scala:100)
        at spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:127)
        at akka.actor.DefaultScheduler$$anon$1.run(Scheduler.scala:142)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94)
        at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
13/06/17 15:23:36 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 1 attempts
java.util.concurrent.TimeoutException: Futures timed out after [10000] milliseconds
        at akka.dispatch.DefaultPromise.ready(Future.scala:870)
        at akka.dispatch.DefaultPromise.result(Future.scala:874)
        at akka.dispatch.Await$.result(Future.scala:74)
        at spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:154)
        at spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:57)
        at spark.storage.BlockManager.spark$storage$BlockManager$$heartBeat(BlockManager.scala:100)
        at spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:127)
        at akka.actor.DefaultScheduler$$anon$1.run(Scheduler.scala:142)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94)
        at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
13/06/17 15:23:36 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 1 attempts
java.util.concurrent.TimeoutException: Futures timed out after [10000] milliseconds
        at akka.dispatch.DefaultPromise.ready(Future.scala:870)
        at akka.dispatch.DefaultPromise.result(Future.scala:874)
        at akka.dispatch.Await$.result(Future.scala:74)
        at spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:154)
        at spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:57)
        at spark.storage.BlockManager.spark$storage$BlockManager$$heartBeat(BlockManager.scala:100)
        at spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:127)
        at akka.actor.DefaultScheduler$$anon$1.run(Scheduler.scala:142)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94)
        at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
13/06/17 15:23:36 ERROR local.LocalScheduler: Exception in task 4
java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapCharBuffer.<init>(HeapCharBuffer.java:57)
        at java.nio.CharBuffer.allocate(CharBuffer.java:331)
        at java.nio.charset.CharsetDecoder.decode(CharsetDecoder.java:777)
        at org.apache.hadoop.io.Text.decode(Text.java:350)
        at org.apache.hadoop.io.Text.decode(Text.java:327)
        at org.apache.hadoop.io.Text.toString(Text.java:254)
        at spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:241)
        at spark.SparkContext$$anonfun$textFile$1.apply(SparkContext.scala:241)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:401)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:401)
        at scala.collection.Iterator$class.foreach(Iterator.scala:772)
        at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
        at spark.CacheManager.getOrCompute(CacheManager.scala:53)
        at spark.RDD.iterator(RDD.scala:193)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:206)
        at spark.RDD.iterator(RDD.scala:195)
        at spark.scheduler.ResultTask.run(ResultTask.scala:76)
        at spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:74)
        at spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
13/06/17 15:23:36 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:23:36 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager ip-10-139-8-186.ec2.internal:34113 with 948.3 MB RAM
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:23:36 INFO local.LocalScheduler: Running ResultTask(0, 5)
13/06/17 15:23:36 INFO storage.BlockManager: Reporting 5 blocks to the master.
13/06/17 15:23:36 INFO storage.BlockManager: Reporting 5 blocks to the master.
13/06/17 15:23:36 ERROR actor.ActorSystemImpl: Uncaught error from thread [spark-akka.actor.default-dispatcher-65] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space
        at akka.jsr166y.ForkJoinPool$WorkQueue.growArray(ForkJoinPool.java:846)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1478)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
13/06/17 15:23:36 INFO scheduler.DAGScheduler: Failed to run reduce at SparkLRhdfs2.scala:46
13/06/17 15:23:36 INFO local.LocalScheduler: Size of task 5 is 1871 bytes
[error] (run-main) spark.SparkException: Job failed: ResultTask(0, 4) failed: ExceptionFailure(java.lang.OutOfMemoryError: Java heap space)
spark.SparkException: Job failed: ResultTask(0, 4) failed: ExceptionFailure(java.lang.OutOfMemoryError: Java heap space)
        at spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:629)
        at spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:627)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:627)
        at spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:588)
        at spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:294)
        at spark.scheduler.DAGScheduler.spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:358)
        at spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:102)
[trace] Stack trace suppressed: run last compile:run-main for the full output.
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 833.6 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_6 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 786.5 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 719.0 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_6 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 786.5 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_6
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 604.4 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_6
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 671.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 671.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 557.3 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 557.3 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_6 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 557.3 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_6
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 442.7 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 442.7 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 442.7 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_5 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 328.1 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_5
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_5 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 328.1 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_5
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 328.1 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:23:36 INFO network.ConnectionManager: Selector thread was interrupted!
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_4 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 213.5 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_4
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_4 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 213.5 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_4
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 213.5 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 98.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 98.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_5 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 98.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_5
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_4 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 98.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_4
13/06/17 15:23:36 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:47180 (size: 114.6 MB, free: 98.9 MB)
13/06/17 15:23:36 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
java.lang.RuntimeException: Nonzero exit code: 1
        at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run-main for the full output.
[error] (compile:run-main) Nonzero exit code: 1
[error] Total time: 161 s, completed Jun 17, 2013 3:23:36 PM
> 13/06/17 15:23:36 INFO spark.CacheManager: Cache key is rdd_2_5
13/06/17 15:23:36 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@696
13/06/17 15:25:03 WARN storage.BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, ip-10-139-8-186.ec2.internal, 34113) with no recent heart beats
13/06/17 15:25:04 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:25:04 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:25:04 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager ip-10-139-8-186.ec2.internal:34113 with 948.3 MB RAM
13/06/17 15:25:09 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:25:09 INFO storage.BlockManager: Reporting 5 blocks to the master.
13/06/17 15:25:09 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 833.6 MB)
13/06/17 15:25:09 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:25:09 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:25:09 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:25:09 INFO storage.BlockManager: Reporting 5 blocks to the master.
13/06/17 15:25:12 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 833.6 MB)
13/06/17 15:25:12 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:25:12 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 719.0 MB)
13/06/17 15:25:12 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:25:12 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:25:13 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:25:13 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 604.4 MB)
13/06/17 15:25:13 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:25:13 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:25:13 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:13 INFO storage.BlockManager: Reporting 5 blocks to the master.
13/06/17 15:25:14 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:25:14 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:14 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:25:14 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:25:17 ERROR local.LocalScheduler: Exception in task 5
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:2694)
        at java.lang.String.<init>(String.java:203)
        at java.lang.String.substring(String.java:1913)
        at java.lang.String.split(String.java:2288)
        at java.lang.String.split(String.java:2355)
        at sparktutorial.SparkLRhdfs2$.readPoint(SparkLRhdfs2.scala:20)
        at sparktutorial.SparkLRhdfs2$$anonfun$3.apply(SparkLRhdfs2.scala:35)
        at sparktutorial.SparkLRhdfs2$$anonfun$3.apply(SparkLRhdfs2.scala:35)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:401)
        at scala.collection.Iterator$class.foreach(Iterator.scala:772)
        at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
        at spark.CacheManager.getOrCompute(CacheManager.scala:53)
        at spark.RDD.iterator(RDD.scala:193)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:206)
        at spark.RDD.iterator(RDD.scala:195)
        at spark.scheduler.ResultTask.run(ResultTask.scala:76)
        at spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:74)
        at spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
13/06/17 15:25:17 INFO local.LocalScheduler: Running ResultTask(0, 6)
13/06/17 15:25:17 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:17 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:25:17 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:17 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:25:17 INFO local.LocalScheduler: Size of task 6 is 1871 bytes
13/06/17 15:25:17 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:17 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:25:17 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:17 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:17 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:25:17 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:25:17 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:25:17 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:25:17 INFO spark.CacheManager: Cache key is rdd_2_6
13/06/17 15:25:17 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@697
13/06/17 15:27:20 ERROR local.LocalScheduler: Exception in task 6
java.lang.OutOfMemoryError: Java heap space
        at java.lang.String.substring(String.java:1913)
        at java.lang.String.split(String.java:2288)
        at java.lang.String.split(String.java:2355)
        at sparktutorial.SparkLRhdfs2$.readPoint(SparkLRhdfs2.scala:20)
        at sparktutorial.SparkLRhdfs2$$anonfun$3.apply(SparkLRhdfs2.scala:35)
        at sparktutorial.SparkLRhdfs2$$anonfun$3.apply(SparkLRhdfs2.scala:35)
        at scala.collection.Iterator$$anon$19.next(Iterator.scala:401)
        at scala.collection.Iterator$class.foreach(Iterator.scala:772)
        at scala.collection.Iterator$$anon$19.foreach(Iterator.scala:399)
        at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
        at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:102)
        at spark.CacheManager.getOrCompute(CacheManager.scala:53)
        at spark.RDD.iterator(RDD.scala:193)
        at spark.rdd.MappedRDD.compute(MappedRDD.scala:12)
        at spark.RDD.computeOrReadCheckpoint(RDD.scala:206)
        at spark.RDD.iterator(RDD.scala:195)
        at spark.scheduler.ResultTask.run(ResultTask.scala:76)
        at spark.scheduler.local.LocalScheduler.runTask$1(LocalScheduler.scala:74)
        at spark.scheduler.local.LocalScheduler$$anon$1.run(LocalScheduler.scala:50)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:722)
13/06/17 15:27:20 INFO local.LocalScheduler: Running ResultTask(0, 7)
13/06/17 15:27:20 INFO local.LocalScheduler: Size of task 7 is 1871 bytes
13/06/17 15:27:20 WARN storage.BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, ip-10-139-8-186.ec2.internal, 34113) with no recent heart beats
13/06/17 15:27:20 INFO storage.BlockManager: BlockManager reregistering with master
13/06/17 15:27:20 INFO storage.BlockManagerMaster: Trying to register BlockManager
13/06/17 15:27:20 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Registering block manager ip-10-139-8-186.ec2.internal:34113 with 948.3 MB RAM
13/06/17 15:27:20 INFO storage.BlockManagerMaster: Registered BlockManager
13/06/17 15:27:20 INFO storage.BlockManager: Reporting 5 blocks to the master.
13/06/17 15:27:20 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_1 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 833.6 MB)
13/06/17 15:27:20 INFO storage.BlockManagerMaster: Updated info of block rdd_2_1
13/06/17 15:27:20 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_3 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 719.0 MB)
13/06/17 15:27:20 INFO storage.BlockManagerMaster: Updated info of block rdd_2_3
13/06/17 15:27:20 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_2 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 604.4 MB)
13/06/17 15:27:20 INFO storage.BlockManagerMaster: Updated info of block rdd_2_2
13/06/17 15:27:20 INFO storage.BlockManagerMasterActor$BlockManagerInfo: Added rdd_2_0 in memory on ip-10-139-8-186.ec2.internal:34113 (size: 114.6 MB, free: 489.8 MB)
13/06/17 15:27:20 WARN actor.ActorSystemImpl: An exception was thrown by TimerTask. WARNING arguments left: 1
13/06/17 15:27:20 INFO storage.BlockManagerMaster: Updated info of block rdd_2_0
13/06/17 15:27:20 INFO spark.CacheManager: Cache key is rdd_2_7
13/06/17 15:27:20 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@698
13/06/17 15:27:25 WARN storage.BlockManagerMasterActor: Removing BlockManager BlockManagerId(<driver>, ip-10-139-8-186.ec2.internal, 47180) with no recent heart beats
13/06/17 15:29:08 ERROR local.LocalScheduler: Exception in task 7
java.lang.OutOfMemoryError: Java heap space
13/06/17 15:29:08 INFO local.LocalScheduler: Running ResultTask(0, 8)
13/06/17 15:29:08 INFO local.LocalScheduler: Size of task 8 is 1871 bytes
13/06/17 15:29:08 WARN storage.BlockManagerMaster: Error sending message to BlockManagerMaster in 1 attempts
akka.pattern.AskTimeoutException: Timed out
        at akka.dispatch.DefaultPromise.result(Future.scala:875)
        at akka.dispatch.Await$.result(Future.scala:74)
        at spark.storage.BlockManagerMaster.askDriverWithReply(BlockManagerMaster.scala:154)
        at spark.storage.BlockManagerMaster.sendHeartBeat(BlockManagerMaster.scala:57)
        at spark.storage.BlockManager.spark$storage$BlockManager$$heartBeat(BlockManager.scala:100)
        at spark.storage.BlockManager$$anonfun$initialize$1.apply$mcV$sp(BlockManager.scala:127)
        at akka.actor.DefaultScheduler$$anon$1.run(Scheduler.scala:142)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:94)
        at akka.jsr166y.ForkJoinTask$AdaptedRunnableAction.exec(ForkJoinTask.java:1381)
        at akka.jsr166y.ForkJoinTask.doExec(ForkJoinTask.java:259)
        at akka.jsr166y.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:975)
        at akka.jsr166y.ForkJoinPool.runWorker(ForkJoinPool.java:1479)
        at akka.jsr166y.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:104)
Caused by: akka.pattern.AskTimeoutException: Timed out
        at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:274)
        at akka.actor.DefaultScheduler$$anon$6$$anon$7.run(Scheduler.scala:183)
        ... 6 more
13/06/17 15:29:09 ERROR actor.ActorSystemImpl: Uncaught error from thread [spark-akka.actor.default-dispatcher-31] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled
java.lang.OutOfMemoryError: Java heap space
Uncaught error from thread [spark-akka.actor.default-dispatcher-31] shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[spark]
java.lang.OutOfMemoryError: Java heap space
13/06/17 15:29:09 INFO spark.CacheManager: Cache key is rdd_2_8
13/06/17 15:29:09 INFO spark.CacheManager: Computing partition spark.rdd.HadoopPartition@699