Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/03/01 08:36:15 INFO SparkContext: Running Spark version 1.4.1
16/03/01 08:36:16 INFO SecurityManager: Changing view acls to: ubuntu
16/03/01 08:36:16 INFO SecurityManager: Changing modify acls to: ubuntu
16/03/01 08:36:16 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ubuntu); users with modify permissions: Set(ubuntu)
16/03/01 08:36:16 INFO Slf4jLogger: Slf4jLogger started
16/03/01 08:36:16 INFO Remoting: Starting remoting
16/03/01 08:36:16 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://
spark...@192.168.56.4:46236]
16/03/01 08:36:16 INFO Utils: Successfully started service 'sparkDriver' on port 46236.
16/03/01 08:36:16 INFO SparkEnv: Registering MapOutputTracker
16/03/01 08:36:16 INFO SparkEnv: Registering BlockManagerMaster
16/03/01 08:36:16 INFO DiskBlockManager: Created local directory at /tmp/spark-d59ce13b-ce92-4a6b-9f48-309abf18c6a4/blockmgr-448725ff-1b71-4d13-afa2-91df5e250370
16/03/01 08:36:16 INFO MemoryStore: MemoryStore started with capacity 530.3 MB
16/03/01 08:36:17 INFO HttpFileServer: HTTP File server directory is /tmp/spark-d59ce13b-ce92-4a6b-9f48-309abf18c6a4/httpd-a1120858-fef8-47db-8db7-91b723a1ca82
16/03/01 08:36:17 INFO HttpServer: Starting HTTP Server
16/03/01 08:36:17 INFO Utils: Successfully started service 'HTTP file server' on port 55073.
16/03/01 08:36:17 INFO SparkEnv: Registering OutputCommitCoordinator
16/03/01 08:36:17 INFO Utils: Successfully started service 'SparkUI' on port 4040.
16/03/01 08:36:17 INFO SparkUI: Started SparkUI at
http://192.168.56.4:404016/03/01 08:36:17 INFO Executor: Starting executor ID driver on host localhost
16/03/01 08:36:17 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41014.
16/03/01 08:36:17 INFO NettyBlockTransferService: Server created on 41014
16/03/01 08:36:17 INFO BlockManagerMaster: Trying to register BlockManager
16/03/01 08:36:17 INFO BlockManagerMasterEndpoint: Registering block manager localhost:41014 with 530.3 MB RAM, BlockManagerId(driver, localhost, 41014)
16/03/01 08:36:17 INFO BlockManagerMaster: Registered BlockManager
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:34
16/03/01 08:36:18 INFO DAGScheduler: Got job 0 (take at PreprocessorSpec.scala:34) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 0(take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at take at PreprocessorSpec.scala:34), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=0, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1703) called with curMem=2984, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 1703.0 B, free 530.3 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on localhost:41014 (size: 1703.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, PROCESS_LOCAL, 1579 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/03/01 08:36:18 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1761 bytes result sent to driver
16/03/01 08:36:18 INFO DAGScheduler: ResultStage 0 (take at PreprocessorSpec.scala:34) finished in 0,062 s
16/03/01 08:36:18 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 61 ms on localhost (1/1)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
16/03/01 08:36:18 INFO DAGScheduler: Job 0 finished: take at PreprocessorSpec.scala:34, took 0,269164 s
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:34
16/03/01 08:36:18 INFO DAGScheduler: Got job 1 (take at PreprocessorSpec.scala:34) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 1(take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 1 (MapPartitionsRDD[5] at take at PreprocessorSpec.scala:34), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=4687, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1685) called with curMem=7671, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1685.0 B, free 530.3 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on localhost:41014 (size: 1685.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 1 (MapPartitionsRDD[5] at take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 1.0 with 1 tasks
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 1, localhost, PROCESS_LOCAL, 1577 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
16/03/01 08:36:18 INFO Executor: Finished task 0.0 in stage 1.0 (TID 1). 1757 bytes result sent to driver
16/03/01 08:36:18 INFO DAGScheduler: ResultStage 1 (take at PreprocessorSpec.scala:34) finished in 0,001 s
16/03/01 08:36:18 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 1) in 12 ms on localhost (1/1)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Removed TaskSet 1.0, whose tasks have all completed, from pool
16/03/01 08:36:18 INFO DAGScheduler: Job 1 finished: take at PreprocessorSpec.scala:34, took 0,022106 s
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:34
16/03/01 08:36:18 INFO DAGScheduler: Got job 2 (take at PreprocessorSpec.scala:34) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 2(take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[8] at take at PreprocessorSpec.scala:34), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=9356, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1691) called with curMem=12340, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 1691.0 B, free 530.3 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on localhost:41014 (size: 1691.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 2 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 2 (MapPartitionsRDD[8] at take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 2.0 with 1 tasks
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 2, localhost, PROCESS_LOCAL, 1582 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 2.0 (TID 2)
16/03/01 08:36:18 INFO Executor: Finished task 0.0 in stage 2.0 (TID 2). 1763 bytes result sent to driver
16/03/01 08:36:18 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 2) in 16 ms on localhost (1/1)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Removed TaskSet 2.0, whose tasks have all completed, from pool
16/03/01 08:36:18 INFO DAGScheduler: ResultStage 2 (take at PreprocessorSpec.scala:34) finished in 0,002 s
16/03/01 08:36:18 INFO DAGScheduler: Job 2 finished: take at PreprocessorSpec.scala:34, took 0,027984 s
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:34
16/03/01 08:36:18 INFO DAGScheduler: Got job 3 (take at PreprocessorSpec.scala:34) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 3(take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 3 (MapPartitionsRDD[11] at take at PreprocessorSpec.scala:34), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=14031, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1690) called with curMem=17015, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 1690.0 B, free 530.3 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on localhost:41014 (size: 1690.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 3 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 3 (MapPartitionsRDD[11] at take at PreprocessorSpec.scala:34)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 3.0 with 1 tasks
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 3, localhost, PROCESS_LOCAL, 1580 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 3.0 (TID 3)
16/03/01 08:36:18 INFO Executor: Finished task 0.0 in stage 3.0 (TID 3). 1759 bytes result sent to driver
16/03/01 08:36:18 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 3) in 7 ms on localhost (1/1)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Removed TaskSet 3.0, whose tasks have all completed, from pool
16/03/01 08:36:18 INFO DAGScheduler: ResultStage 3 (take at PreprocessorSpec.scala:34) finished in 0,008 s
16/03/01 08:36:18 INFO DAGScheduler: Job 3 finished: take at PreprocessorSpec.scala:34, took 0,023639 s
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:50
16/03/01 08:36:18 INFO DAGScheduler: Got job 4 (take at PreprocessorSpec.scala:50) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 4(take at PreprocessorSpec.scala:50)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[14] at take at PreprocessorSpec.scala:50), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(3208) called with curMem=18705, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 3.1 KB, free 530.3 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1770) called with curMem=21913, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 1770.0 B, free 530.3 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on localhost:41014 (size: 1770.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4 (MapPartitionsRDD[14] at take at PreprocessorSpec.scala:50)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 4, localhost, PROCESS_LOCAL, 1537 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 4.0 (TID 4)
16/03/01 08:36:18 INFO Executor: Finished task 0.0 in stage 4.0 (TID 4). 1974 bytes result sent to driver
16/03/01 08:36:18 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 4) in 8 ms on localhost (1/1)
16/03/01 08:36:18 INFO DAGScheduler: ResultStage 4 (take at PreprocessorSpec.scala:50) finished in 0,009 s
16/03/01 08:36:18 INFO TaskSchedulerImpl: Removed TaskSet 4.0, whose tasks have all completed, from pool
16/03/01 08:36:18 INFO DAGScheduler: Job 4 finished: take at PreprocessorSpec.scala:50, took 0,020782 s
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:50
16/03/01 08:36:18 INFO DAGScheduler: Got job 5 (take at PreprocessorSpec.scala:50) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 5(take at PreprocessorSpec.scala:50)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[17] at take at PreprocessorSpec.scala:50), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=23683, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1687) called with curMem=26667, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 1687.0 B, free 530.3 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on localhost:41014 (size: 1687.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 5 (MapPartitionsRDD[17] at take at PreprocessorSpec.scala:50)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 5.0 with 1 tasks
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 5, localhost, PROCESS_LOCAL, 1515 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 5.0 (TID 5)
16/03/01 08:36:18 INFO Executor: Finished task 0.0 in stage 5.0 (TID 5). 1709 bytes result sent to driver
16/03/01 08:36:18 INFO TaskSetManager: Finished task 0.0 in stage 5.0 (TID 5) in 8 ms on localhost (1/1)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Removed TaskSet 5.0, whose tasks have all completed, from pool
16/03/01 08:36:18 INFO DAGScheduler: ResultStage 5 (take at PreprocessorSpec.scala:50) finished in 0,009 s
16/03/01 08:36:18 INFO DAGScheduler: Job 5 finished: take at PreprocessorSpec.scala:50, took 0,017768 s
16/03/01 08:36:18 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:68
16/03/01 08:36:18 INFO DAGScheduler: Got job 6 (take at PreprocessorSpec.scala:68) with 1 output partitions (allowLocal=false)
16/03/01 08:36:18 INFO DAGScheduler: Final stage: ResultStage 6(take at PreprocessorSpec.scala:68)
16/03/01 08:36:18 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:18 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:18 INFO DAGScheduler: Submitting ResultStage 6 (MapPartitionsRDD[20] at take at PreprocessorSpec.scala:68), which has no missing parents
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(3208) called with curMem=28354, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 3.1 KB, free 530.2 MB)
16/03/01 08:36:18 INFO MemoryStore: ensureFreeSpace(1770) called with curMem=31562, maxMem=556038881
16/03/01 08:36:18 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 1770.0 B, free 530.2 MB)
16/03/01 08:36:18 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on localhost:41014 (size: 1770.0 B, free: 530.3 MB)
16/03/01 08:36:18 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 6 (MapPartitionsRDD[20] at take at PreprocessorSpec.scala:68)
16/03/01 08:36:18 INFO TaskSchedulerImpl: Adding task set 6.0 with 1 tasks
16/03/01 08:36:18 WARN TaskSetManager: Stage 6 contains a task of very large size (257 KB). The maximum recommended task size is 100 KB.
16/03/01 08:36:18 INFO TaskSetManager: Starting task 0.0 in stage 6.0 (TID 6, localhost, PROCESS_LOCAL, 263669 bytes)
16/03/01 08:36:18 INFO Executor: Running task 0.0 in stage 6.0 (TID 6)
16/03/01 08:36:19 INFO Executor: Finished task 0.0 in stage 6.0 (TID 6). 660504 bytes result sent to driver
16/03/01 08:36:19 INFO TaskSetManager: Finished task 0.0 in stage 6.0 (TID 6) in 365 ms on localhost (1/1)
16/03/01 08:36:19 INFO TaskSchedulerImpl: Removed TaskSet 6.0, whose tasks have all completed, from pool
16/03/01 08:36:19 INFO DAGScheduler: ResultStage 6 (take at PreprocessorSpec.scala:68) finished in 0,363 s
16/03/01 08:36:19 INFO DAGScheduler: Job 6 finished: take at PreprocessorSpec.scala:68, took 0,383692 s
DefaultPreprocessor converted 256 images in 0.497s
16/03/01 08:36:19 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:68
16/03/01 08:36:19 INFO DAGScheduler: Got job 7 (take at PreprocessorSpec.scala:68) with 1 output partitions (allowLocal=false)
16/03/01 08:36:19 INFO DAGScheduler: Final stage: ResultStage 7(take at PreprocessorSpec.scala:68)
16/03/01 08:36:19 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:19 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:19 INFO DAGScheduler: Submitting ResultStage 7 (MapPartitionsRDD[23] at take at PreprocessorSpec.scala:68), which has no missing parents
16/03/01 08:36:19 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=33332, maxMem=556038881
16/03/01 08:36:19 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 2.9 KB, free 530.2 MB)
16/03/01 08:36:19 INFO MemoryStore: ensureFreeSpace(1689) called with curMem=36316, maxMem=556038881
16/03/01 08:36:19 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 1689.0 B, free 530.2 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on localhost:41014 (size: 1689.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:19 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 7 (MapPartitionsRDD[23] at take at PreprocessorSpec.scala:68)
16/03/01 08:36:19 INFO TaskSchedulerImpl: Adding task set 7.0 with 1 tasks
16/03/01 08:36:19 INFO TaskSetManager: Starting task 0.0 in stage 7.0 (TID 7, localhost, PROCESS_LOCAL, 67048 bytes)
16/03/01 08:36:19 INFO Executor: Running task 0.0 in stage 7.0 (TID 7)
16/03/01 08:36:19 INFO Executor: Finished task 0.0 in stage 7.0 (TID 7). 67562 bytes result sent to driver
16/03/01 08:36:19 INFO TaskSetManager: Finished task 0.0 in stage 7.0 (TID 7) in 8 ms on localhost (1/1)
16/03/01 08:36:19 INFO TaskSchedulerImpl: Removed TaskSet 7.0, whose tasks have all completed, from pool
16/03/01 08:36:19 INFO DAGScheduler: ResultStage 7 (take at PreprocessorSpec.scala:68) finished in 0,001 s
16/03/01 08:36:19 INFO DAGScheduler: Job 7 finished: take at PreprocessorSpec.scala:68, took 0,019660 s
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_7_piece0 on localhost:41014 in memory (size: 1689.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_6_piece0 on localhost:41014 in memory (size: 1770.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_5_piece0 on localhost:41014 in memory (size: 1687.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_4_piece0 on localhost:41014 in memory (size: 1770.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_3_piece0 on localhost:41014 in memory (size: 1690.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_2_piece0 on localhost:41014 in memory (size: 1691.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_1_piece0 on localhost:41014 in memory (size: 1685.0 B, free: 530.3 MB)
16/03/01 08:36:19 INFO BlockManagerInfo: Removed broadcast_0_piece0 on localhost:41014 in memory (size: 1703.0 B, free: 530.3 MB)
DefaultPreprocessor converted 256 images in 0.577s
16/03/01 08:36:20 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:92
16/03/01 08:36:20 INFO DAGScheduler: Got job 8 (take at PreprocessorSpec.scala:92) with 1 output partitions (allowLocal=false)
16/03/01 08:36:20 INFO DAGScheduler: Final stage: ResultStage 8(take at PreprocessorSpec.scala:92)
16/03/01 08:36:20 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:20 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:20 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[26] at take at PreprocessorSpec.scala:92), which has no missing parents
16/03/01 08:36:20 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=0, maxMem=556038881
16/03/01 08:36:20 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:20 INFO MemoryStore: ensureFreeSpace(1689) called with curMem=2984, maxMem=556038881
16/03/01 08:36:20 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 1689.0 B, free 530.3 MB)
16/03/01 08:36:20 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on localhost:41014 (size: 1689.0 B, free: 530.3 MB)
16/03/01 08:36:20 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 8 (MapPartitionsRDD[26] at take at PreprocessorSpec.scala:92)
16/03/01 08:36:20 INFO TaskSchedulerImpl: Adding task set 8.0 with 1 tasks
16/03/01 08:36:20 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 8, localhost, PROCESS_LOCAL, 1572 bytes)
16/03/01 08:36:20 INFO Executor: Running task 0.0 in stage 8.0 (TID 8)
16/03/01 08:36:20 INFO Executor: Finished task 0.0 in stage 8.0 (TID 8). 1766 bytes result sent to driver
16/03/01 08:36:20 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 8) in 5 ms on localhost (1/1)
16/03/01 08:36:20 INFO TaskSchedulerImpl: Removed TaskSet 8.0, whose tasks have all completed, from pool
16/03/01 08:36:20 INFO DAGScheduler: ResultStage 8 (take at PreprocessorSpec.scala:92) finished in 0,001 s
16/03/01 08:36:20 INFO DAGScheduler: Job 8 finished: take at PreprocessorSpec.scala:92, took 0,016527 s
16/03/01 08:36:20 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:107
16/03/01 08:36:20 INFO DAGScheduler: Got job 9 (take at PreprocessorSpec.scala:107) with 1 output partitions (allowLocal=false)
16/03/01 08:36:20 INFO DAGScheduler: Final stage: ResultStage 9(take at PreprocessorSpec.scala:107)
16/03/01 08:36:20 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:20 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:20 INFO DAGScheduler: Submitting ResultStage 9 (MapPartitionsRDD[29] at take at PreprocessorSpec.scala:107), which has no missing parents
16/03/01 08:36:20 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=4673, maxMem=556038881
16/03/01 08:36:20 INFO MemoryStore: Block broadcast_9 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:20 INFO MemoryStore: ensureFreeSpace(1690) called with curMem=7657, maxMem=556038881
16/03/01 08:36:20 INFO MemoryStore: Block broadcast_9_piece0 stored as bytes in memory (estimated size 1690.0 B, free 530.3 MB)
16/03/01 08:36:20 INFO BlockManagerInfo: Added broadcast_9_piece0 in memory on localhost:41014 (size: 1690.0 B, free: 530.3 MB)
16/03/01 08:36:20 INFO SparkContext: Created broadcast 9 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 9 (MapPartitionsRDD[29] at take at PreprocessorSpec.scala:107)
16/03/01 08:36:20 INFO TaskSchedulerImpl: Adding task set 9.0 with 1 tasks
16/03/01 08:36:20 INFO TaskSetManager: Starting task 0.0 in stage 9.0 (TID 9, localhost, PROCESS_LOCAL, 1572 bytes)
16/03/01 08:36:20 INFO Executor: Running task 0.0 in stage 9.0 (TID 9)
16/03/01 08:36:20 INFO Executor: Finished task 0.0 in stage 9.0 (TID 9). 1766 bytes result sent to driver
16/03/01 08:36:20 INFO TaskSetManager: Finished task 0.0 in stage 9.0 (TID 9) in 7 ms on localhost (1/1)
16/03/01 08:36:20 INFO TaskSchedulerImpl: Removed TaskSet 9.0, whose tasks have all completed, from pool
16/03/01 08:36:20 INFO DAGScheduler: ResultStage 9 (take at PreprocessorSpec.scala:107) finished in 0,006 s
16/03/01 08:36:20 INFO DAGScheduler: Job 9 finished: take at PreprocessorSpec.scala:107, took 0,016329 s
16/03/01 08:36:20 INFO SparkContext: Starting job: take at PreprocessorSpec.scala:128
16/03/01 08:36:20 INFO DAGScheduler: Got job 10 (take at PreprocessorSpec.scala:128) with 1 output partitions (allowLocal=false)
16/03/01 08:36:20 INFO DAGScheduler: Final stage: ResultStage 10(take at PreprocessorSpec.scala:128)
16/03/01 08:36:20 INFO DAGScheduler: Parents of final stage: List()
16/03/01 08:36:20 INFO DAGScheduler: Missing parents: List()
16/03/01 08:36:20 INFO DAGScheduler: Submitting ResultStage 10 (MapPartitionsRDD[32] at take at PreprocessorSpec.scala:128), which has no missing parents
16/03/01 08:36:20 INFO MemoryStore: ensureFreeSpace(2984) called with curMem=9347, maxMem=556038881
16/03/01 08:36:20 INFO MemoryStore: Block broadcast_10 stored as values in memory (estimated size 2.9 KB, free 530.3 MB)
16/03/01 08:36:20 INFO MemoryStore: ensureFreeSpace(1687) called with curMem=12331, maxMem=556038881
16/03/01 08:36:20 INFO MemoryStore: Block broadcast_10_piece0 stored as bytes in memory (estimated size 1687.0 B, free 530.3 MB)
16/03/01 08:36:20 INFO BlockManagerInfo: Added broadcast_10_piece0 in memory on localhost:41014 (size: 1687.0 B, free: 530.3 MB)
16/03/01 08:36:20 INFO SparkContext: Created broadcast 10 from broadcast at DAGScheduler.scala:874
16/03/01 08:36:20 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 10 (MapPartitionsRDD[32] at take at PreprocessorSpec.scala:128)
16/03/01 08:36:20 INFO TaskSchedulerImpl: Adding task set 10.0 with 1 tasks
16/03/01 08:36:20 WARN TaskSetManager: Stage 10 contains a task of very large size (193 KB). The maximum recommended task size is 100 KB.
16/03/01 08:36:20 INFO TaskSetManager: Starting task 0.0 in stage 10.0 (TID 10, localhost, PROCESS_LOCAL, 198120 bytes)
16/03/01 08:36:20 INFO Executor: Running task 0.0 in stage 10.0 (TID 10)
16/03/01 08:36:20 INFO Executor: Finished task 0.0 in stage 10.0 (TID 10). 199274 bytes result sent to driver
16/03/01 08:36:20 INFO TaskSetManager: Finished task 0.0 in stage 10.0 (TID 10) in 7 ms on localhost (1/1)
16/03/01 08:36:20 INFO TaskSchedulerImpl: Removed TaskSet 10.0, whose tasks have all completed, from pool
16/03/01 08:36:20 INFO DAGScheduler: ResultStage 10 (take at PreprocessorSpec.scala:128) finished in 0,007 s
16/03/01 08:36:20 INFO DAGScheduler: Job 10 finished: take at PreprocessorSpec.scala:128, took 0,016381 s
16/03/01 08:36:20 INFO BlockManagerInfo: Removed broadcast_10_piece0 on localhost:41014 in memory (size: 1687.0 B, free: 530.3 MB)
16/03/01 08:36:20 INFO BlockManagerInfo: Removed broadcast_9_piece0 on localhost:41014 in memory (size: 1690.0 B, free: 530.3 MB)
16/03/01 08:36:20 INFO BlockManagerInfo: Removed broadcast_8_piece0 on localhost:41014 in memory (size: 1689.0 B, free: 530.3 MB)
ImageNetPreprocessor converted 256 images in 0.226s
16/03/01 08:36:20 INFO SparkUI: Stopped Spark web UI at
http://192.168.56.4:404016/03/01 08:36:20 INFO DAGScheduler: Stopping DAGScheduler
16/03/01 08:36:20 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/03/01 08:36:20 INFO Utils: path = /tmp/spark-d59ce13b-ce92-4a6b-9f48-309abf18c6a4/blockmgr-448725ff-1b71-4d13-afa2-91df5e250370, already present as root for deletion.
16/03/01 08:36:20 INFO MemoryStore: MemoryStore cleared
16/03/01 08:36:20 INFO BlockManager: BlockManager stopped
16/03/01 08:36:20 INFO BlockManagerMaster: BlockManagerMaster stopped
16/03/01 08:36:20 INFO SparkContext: Successfully stopped SparkContext
[info] PreprocessorSpec:
[info] DefaultPreprocessor
[info] - should preserve scalar values
[info] DefaultPreprocessor
[info] - should preserve array values16/03/01 08:36:20 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!