java.lang.UnsatisfiedLinkError (no disni in java.library.path) when running crail-terasort

569 views

Skip to first unread message

sunja...@gmail.com

unread,

Apr 11, 2017, 5:37:15 AM4/11/17

to zrlio-users

Hi, I encounter a weird problem when running crail-terasort.

When I run the part "sorting data using crail-terasort", I met "java.lang.UnsatisfiedLinkError (no disni in java.library.path)".

Here I use 1 machine(tian70) as master and namenode, 3 machines(tian67,tian68,tian69) as worker and StorageServer.

The server is tian70. And I submit the crail-terasort application :

spark-submit -v --num-executors 3 \
--executor-cores 1 --executor-memory 4G \
--driver-memory 4G --master spark://spark-master:7077 \
--class com.ibm.crail.terasort.TeraSort \
/home/tian/softwares/crail-terasort/target/crail-terasort-2.0-jar-with-dependencies.jar \
-i /home/tian/sunjane/terasort_in-1g -o /home/tian/sunjane/terasort_out-1g

Then error happened like below

17/04/11 17:08:55 WARN TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1, 192.168.30.69): java.lang.UnsatisfiedLinkError: no disni in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.ibm.disni.rdma.verbs.impl.NativeDispatcher.<clinit>(NativeDispatcher.java:36)
	at com.ibm.disni.rdma.verbs.impl.RdmaProviderNat.<init>(RdmaProviderNat.java:43)
	at com.ibm.disni.rdma.verbs.RdmaProvider.provider(RdmaProvider.java:58)
	at com.ibm.disni.rdma.verbs.RdmaCm.open(RdmaCm.java:49)
	at com.ibm.disni.rdma.verbs.RdmaEventChannel.createEventChannel(RdmaEventChannel.java:56)
	at com.ibm.disni.rdma.RdmaCmProcessor.<init>(RdmaCmProcessor.java:48)
	at com.ibm.disni.rdma.RdmaEndpointGroup.<init>(RdmaEndpointGroup.java:61)
	at com.ibm.darpc.RpcEndpointGroup.<init>(RpcEndpointGroup.java:47)
	at com.ibm.darpc.RpcClientGroup.<init>(RpcClientGroup.java:20)
	at com.ibm.darpc.RpcClientGroup.createClientGroup(RpcClientGroup.java:13)
	at com.ibm.crail.namenode.rpc.darpc.DaRPCNameNode.getRpcClient(DaRPCNameNode.java:57)
	at com.ibm.crail.core.CoreFileSystem.<init>(CoreFileSystem.java:119)
	at com.ibm.crail.CrailFS.newInstance(CrailFS.java:82)
	at org.apache.spark.storage.CrailStore.org$apache$spark$storage$CrailStore$$init(CrailStore.scala:110)
	at org.apache.spark.storage.CrailStore$.get(CrailStore.scala:589)
	at org.apache.spark.shuffle.crail.CrailShuffleWriter.<init>(CrailShuffleWriter.scala:45)
	at org.apache.spark.shuffle.crail.CrailShuffleManager.getWriter(CrailShuffleManager.scala:79)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:78)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
	at org.apache.spark.scheduler.Task.run(Task.scala:85)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I find that 3 worker machine all met this problem. It seems that I miss the libdisni.so. In fact I have add the libdisni.so into crail-1.0-bin/lib in all the four machine. But

it seems not to work. So I guess whether I configure something wrong? By the way, the example of disni module and crail can work well.

tian@tian70:~/softwares/crail-1.0-bin$ ls -lt lib
total 2792
-rw-r--r-- 1 tian tian 1146500 Apr 11 15:14 libdisni.a
-rwxr-xr-x 1 tian tian     967 Apr 11 15:14 libdisni.la
-rwxr-xr-x 1 tian tian  565603 Apr 11 15:14 libdisni.so
-rwxr-xr-x 1 tian tian  565603 Apr 11 15:14 libdisni.so.0
-rwxr-xr-x 1 tian tian  565603 Apr 11 15:14 libdisni.so.0.0.0

And I add the LD_LIBRARY_PATH in ~/.bashrc in all the four machine. Should the environment variables set here?

export LD_LIBRARY_PATH=$CRAIL_HOME/lib:$LD_LIBRARY_PATH

The OS I use is :

tian@tian70:~/softwares/crail-1.0-bin$ uname -a
Linux tian70 3.19.0-25-generic #26~14.04.1-Ubuntu SMP Fri Jul 24 21:16:20 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

The printlog I can catch is like below:

with 1 cores
17/04/11 17:23:40 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/88 on hostPort 192.168.30.69:33975 with 1 cores, 4.0 GB RAM
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/0 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/1 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/2 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/3 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/4 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/5 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/6 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/7 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/31 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/32 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/33 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/34 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/35 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/36 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/37 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/38 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/39 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/40 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/41 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/42 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/43 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/44 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/45 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/46 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/47 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/48 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/49 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/50 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/58 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/59 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/60 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/61 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/62 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/63 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/64 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/8 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/65 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/9 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/66 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/51 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/67 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/68 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/52 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/53 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/69 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/70 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/71 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/72 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/73 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/74 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/75 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/54 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/76 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/77 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/55 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/56 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/78 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/79 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/10 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/11 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/12 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/13 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/14 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/15 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/16 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/17 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/18 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/19 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/20 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/21 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/22 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/23 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/80 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/24 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/81 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/57 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/82 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/25 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/26 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/83 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/84 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/27 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/28 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/29 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/30 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/85 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/86 is now RUNNING
17/04/11 17:23:40 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/87 is now RUNNING
17/04/11 17:23:41 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/88 is now RUNNING
17/04/11 17:23:41 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/04/11 17:23:41 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 159.3 KB, free 2004.4 MB)
17/04/11 17:23:41 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 17.2 KB, free 2004.4 MB)
17/04/11 17:23:41 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.30.70:43214 (size: 17.2 KB, free: 2004.6 MB)
17/04/11 17:23:41 INFO SparkContext: Created broadcast 0 from newAPIHadoopFile at TeraSort.scala:104

TestName          : loadsortstore 
InputDir          : /home/tian/sunjane/terasort_in-1g
OutputDir         : /home/tian/sunjane/terasort_out-1g
BufferSize        : kryo: 1048576 f22: 104857600
Serializer        : none
PartitionSize     : sizeNotSet, using the default from HDFS
Sync output       : false
No of warmup keys : 0
Verbose           : false
spark options :  none  


17/04/11 17:23:41 INFO FileInputFormat: Total input paths to process : 4
17/04/11 17:23:41 INFO deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
17/04/11 17:23:41 INFO SparkContext: Starting job: saveAsNewAPIHadoopFile at TeraSort.scala:146
17/04/11 17:23:41 INFO CrailStore: CrailStore starting version 282
17/04/11 17:23:41 INFO CrailStore: spark.crail.shuffle.affinity true
17/04/11 17:23:41 INFO CrailStore: spark.crail.deleteonclose false
17/04/11 17:23:41 INFO CrailStore: spark.crail.deleteOnStart true
17/04/11 17:23:41 INFO CrailStore: spark.crail.shuffle.outstanding 1
17/04/11 17:23:41 INFO CrailStore: spark.crail.preallocate 0
17/04/11 17:23:41 INFO CrailStore: spark.crail.shuffleCycle 2147483647
17/04/11 17:23:41 INFO CrailStore: spark.crail.writeAhead 0
17/04/11 17:23:41 INFO CrailStore: spark.crail.debug false
17/04/11 17:23:41 INFO crail: sunjane change crail_home: /home/tian/softwares/crail-1.0-bin
17/04/11 17:23:41 INFO crail: creating singleton crail file system
17/04/11 17:23:41 INFO crail: crail.version 2804
17/04/11 17:23:41 INFO crail: crail.namenode.address crail://192.168.30.70:9060
17/04/11 17:23:41 INFO crail: crail.namenode.blockselection roundrobin
17/04/11 17:23:41 INFO crail: crail.namenode.fileblocks 16
17/04/11 17:23:41 INFO crail: crail.namenode.rpc.type com.ibm.crail.namenode.rpc.darpc.DaRPCNameNode
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.polling false
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.type passive
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.affinity 1
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.maxinline 0
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.recvQueue 32
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.sendQueue 32
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.pollsize 32
17/04/11 17:23:41 INFO crail: crail.namenode.darpc.clustersize 128
17/04/11 17:23:41 INFO crail: crail.storage.types com.ibm.crail.storage.rdma.RdmaStorageTier
17/04/11 17:23:41 INFO crail: crail.directory.depth 16
17/04/11 17:23:41 INFO crail: crail.token.expiration 10
17/04/11 17:23:41 INFO crail: crail.blocksize 1048576
17/04/11 17:23:41 INFO crail: crail.cachelimit 67108864
17/04/11 17:23:41 INFO crail: crail.cachepath /mnt/hugeCache
17/04/11 17:23:41 INFO crail: crail.user stu
17/04/11 17:23:41 INFO crail: crail.shadow.replication 1
17/04/11 17:23:41 INFO crail: crail.debug false
17/04/11 17:23:41 INFO crail: crail.statistics true
17/04/11 17:23:41 INFO crail: crail.rpc.timeout 1000
17/04/11 17:23:41 INFO crail: crail.data.timeout 1000
17/04/11 17:23:41 INFO crail: crail.buffersize 1048576
17/04/11 17:23:41 INFO crail: crail.default.writeahead 2147483647
17/04/11 17:23:41 INFO crail: crail.singleton true
17/04/11 17:23:41 INFO crail: crail.regionsize 67108864
17/04/11 17:23:41 INFO crail: crail.directoryrecord 512
17/04/11 17:23:41 INFO crail: crail.directoryrandomize true
17/04/11 17:23:41 INFO crail: crail.storage.rdma.interface p4p1
17/04/11 17:23:41 INFO crail: crail.storage.rdma.port 50040
17/04/11 17:23:41 INFO crail: crail.storage.rdma.storagelimit 536870912
17/04/11 17:23:41 INFO crail: crail.storage.rdma.allocationsize 67108864
17/04/11 17:23:41 INFO crail: crail.storage.rdma.datapath /mnt/hugeData
17/04/11 17:23:41 INFO crail: crail.storage.rdma.indexpath /home/tian/sunjane/crail-index
17/04/11 17:23:41 INFO crail: crail.storage.rdma.localmap true
17/04/11 17:23:41 INFO crail: crail.storage.rdma.queuesize 32
17/04/11 17:23:41 INFO crail: crail.storage.rdma.type passive
17/04/11 17:23:41 INFO crail: adding tier to cache 0
17/04/11 17:23:41 INFO disni: creating  RdmaProvider of type 'nat'
17/04/11 17:23:41 INFO disni: jverbs jni version 30
17/04/11 17:23:41 INFO disni: sock_addr_in size mismatch, jverbs size 28, native size 16
17/04/11 17:23:41 INFO disni: IbvRecvWR size match, jverbs size 32, native size 32
17/04/11 17:23:41 INFO disni: IbvSendWR size mismatch, jverbs size 72, native size 80
17/04/11 17:23:41 INFO disni: IbvWC size match, jverbs size 48, native size 48
17/04/11 17:23:41 INFO disni: IbvSge size match, jverbs size 16, native size 16
17/04/11 17:23:41 INFO disni: Remote addr offset match, jverbs size 40, native size 40
17/04/11 17:23:41 INFO disni: Rkey offset match, jverbs size 48, native size 48
17/04/11 17:23:41 INFO disni: createEventChannel, objId 140316514673712
17/04/11 17:23:41 INFO crail: rpc group started, recvQueue 32
17/04/11 17:23:41 INFO disni: launching cm processor, cmChannel 0
17/04/11 17:23:41 INFO disni: createId, id 140316514729088
17/04/11 17:23:42 INFO disni: new client endpoint, id 0, idPriv 0
17/04/11 17:23:42 INFO darpc: RPC client endpoint, with buffer size = 244, pipeline 32
17/04/11 17:23:42 INFO crail: connecting to namenode at /192.168.30.70:9060
17/04/11 17:23:42 INFO disni: resolveAddr, addres /192.168.30.70:9060
17/04/11 17:23:42 INFO disni: resolveRoute, id 0
17/04/11 17:23:42 INFO disni: allocPd, objId 140316514726224
17/04/11 17:23:42 INFO disni: setting up protection domain, context 423, pd 3
17/04/11 17:23:42 INFO disni: new endpoint CQ processor
17/04/11 17:23:42 INFO disni: createCompChannel, context 140315709165360
17/04/11 17:23:42 INFO disni: createCQ, objId 140316514722144, ncqe 64
17/04/11 17:23:42 INFO disni: createQP, objId 140316514724424, send_wr size 32, recv_wr_size 32
17/04/11 17:23:42 INFO disni: connect, id 0
17/04/11 17:23:42 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.30.70:41509, dstAddress /192.168.30.70:9060
17/04/11 17:23:42 INFO darpc: new direct rpc stream
17/04/11 17:23:42 INFO crail: connected to namenode at /192.168.30.70:9060
17/04/11 17:23:42 INFO crail: buffer cache, allocationCount 1, bufferCount 64
17/04/11 17:23:42 INFO CrailStore: creating main dir /spark
17/04/11 17:23:42 INFO CrailStore: creating main dir /spark
17/04/11 17:23:42 INFO crail: passive data client 
17/04/11 17:23:42 INFO disni: createEventChannel, objId 140316514908480
17/04/11 17:23:42 INFO disni: passive endpoint group, maxWR 32, maxSge 4, cqSize 64
17/04/11 17:23:42 INFO disni: launching cm processor, cmChannel 0
17/04/11 17:23:42 INFO disni: createId, id 140316514921840
17/04/11 17:23:42 INFO disni: new client endpoint, id 1, idPriv 0
17/04/11 17:23:42 INFO disni: resolveAddr, addres /192.168.30.68:50040
17/04/11 17:23:42 INFO disni: resolveRoute, id 0
17/04/11 17:23:42 INFO disni: setting up protection domain, context 423, pd 3
17/04/11 17:23:42 INFO disni: setting up cq processor
17/04/11 17:23:42 INFO disni: new endpoint CQ processor
17/04/11 17:23:42 INFO disni: createCompChannel, context 140315709165360
17/04/11 17:23:42 INFO disni: createCQ, objId 140316514923472, ncqe 64
17/04/11 17:23:42 INFO disni: createQP, objId 140316514920648, send_wr size 32, recv_wr_size 32
17/04/11 17:23:42 INFO disni: connect, id 0
17/04/11 17:23:42 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.30.70:42581, dstAddress /192.168.30.68:50040
17/04/11 17:23:42 INFO disni: createId, id 140316514941760
17/04/11 17:23:42 INFO disni: new client endpoint, id 2, idPriv 0
17/04/11 17:23:42 INFO disni: resolveAddr, addres /192.168.30.69:50040
17/04/11 17:23:42 INFO disni: resolveRoute, id 0
17/04/11 17:23:42 INFO disni: setting up protection domain, context 423, pd 3
17/04/11 17:23:42 INFO disni: setting up cq processor
17/04/11 17:23:42 INFO disni: new endpoint CQ processor
17/04/11 17:23:42 INFO disni: createCompChannel, context 140315709165360
17/04/11 17:23:42 INFO disni: createCQ, objId 140316514942352, ncqe 64
17/04/11 17:23:42 INFO disni: createQP, objId 140316514940744, send_wr size 32, recv_wr_size 32
17/04/11 17:23:42 INFO disni: connect, id 0
17/04/11 17:23:42 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.30.70:34450, dstAddress /192.168.30.69:50040
17/04/11 17:23:42 INFO CrailStore: creating main dir done /spark
17/04/11 17:23:42 INFO CrailStore: creating hostFile /spark/meta/hosts/-874018885
17/04/11 17:23:42 INFO CrailStore: creating hostFile done /spark/meta/hosts/-874018885
17/04/11 17:23:42 INFO CrailStore: buffer cache warmup 
17/04/11 17:23:42 INFO disni: createId, id 140316514960336
17/04/11 17:23:42 INFO disni: new client endpoint, id 3, idPriv 0
17/04/11 17:23:42 INFO disni: resolveAddr, addres /192.168.30.67:50040
17/04/11 17:23:42 INFO disni: resolveRoute, id 0
17/04/11 17:23:42 INFO disni: setting up protection domain, context 423, pd 3
17/04/11 17:23:42 INFO disni: setting up cq processor
17/04/11 17:23:42 INFO disni: new endpoint CQ processor
17/04/11 17:23:42 INFO disni: createCompChannel, context 140315709165360
17/04/11 17:23:42 INFO disni: createCQ, objId 140316514960928, ncqe 64
17/04/11 17:23:42 INFO disni: createQP, objId 140316514959368, send_wr size 32, recv_wr_size 32
17/04/11 17:23:42 INFO disni: connect, id 0
17/04/11 17:23:42 INFO disni: got event type + RDMA_CM_EVENT_ESTABLISHED, srcAddress /192.168.30.70:36752, dstAddress /192.168.30.67:50040
17/04/11 17:23:42 INFO CrailStore: buffer cache warmup done 
17/04/11 17:23:42 INFO crail: CrailStatistics, tag=init
17/04/11 17:23:42 INFO crail: provider=IO/input [total 0, localOps 0, remoteOps 0, localDirOps 0, remoteDirOps 0, cached 0, nonBlocking 0, blocking 0, prefetched 0, prefetchedNonBlocking 0, prefetchedBlocking 0, capacity 0, totalStreams 0, avgCapacity 0, avgOpLen 0]
17/04/11 17:23:42 INFO crail: provider=MappedBufferCache [cacheGet 10, cachePut 10, cacheMiss 1, cacheSize 64, cacheMax 1, cacheMissMap 1, cacheMissHeap 0]
17/04/11 17:23:42 INFO crail: provider=IO/output [total 10, localOps 0, remoteOps 10, localDirOps 0, remoteDirOps 10, cached 10, nonBlocking 0, blocking 0, prefetched 0, prefetchedNonBlocking 0, prefetchedBlocking 0, capacity 3150848, totalStreams 11, avgCapacity 286440, avgOpLen 512]
17/04/11 17:23:42 INFO crail: provider=Streams [open 11, openInput 0, openOutput 11, openInputDir 0, openOutputDir 10, close 11, closeInput 0, closeOutput 11, closeInputDir 0, closeOutputDir 10, maxInput 0, maxOutput 1]
17/04/11 17:23:42 INFO crail: provider=EndpointCache [size 3]
17/04/11 17:23:42 INFO DAGScheduler: Registering RDD 0 (newAPIHadoopFile at TeraSort.scala:104)
17/04/11 17:23:42 INFO DAGScheduler: Got job 0 (saveAsNewAPIHadoopFile at TeraSort.scala:146) with 32 output partitions
17/04/11 17:23:42 INFO DAGScheduler: Final stage: ResultStage 1 (saveAsNewAPIHadoopFile at TeraSort.scala:146)
17/04/11 17:23:42 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 0)
17/04/11 17:23:42 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 0)
17/04/11 17:23:42 INFO DAGScheduler: Submitting ShuffleMapStage 0 (/home/tian/sunjane/terasort_in-1g NewHadoopRDD[0] at newAPIHadoopFile at TeraSort.scala:104), which has no missing parents
17/04/11 17:23:42 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 2.9 KB, free 2004.4 MB)
17/04/11 17:23:42 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 1891.0 B, free 2004.4 MB)
17/04/11 17:23:42 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.70:43214 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:42 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1012
17/04/11 17:23:42 INFO DAGScheduler: Submitting 32 missing tasks from ShuffleMapStage 0 (/home/tian/sunjane/terasort_in-1g NewHadoopRDD[0] at newAPIHadoopFile at TeraSort.scala:104)
17/04/11 17:23:42 INFO TaskSchedulerImpl: Adding task set 0.0 with 32 tasks
17/04/11 17:23:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49067) with ID 50
17/04/11 17:23:44 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 192.168.30.68, partition 0, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 0 on executor id: 50 hostname: 192.168.30.68.
17/04/11 17:23:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49068) with ID 39
17/04/11 17:23:44 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, 192.168.30.68, partition 1, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 1 on executor id: 39 hostname: 192.168.30.68.
17/04/11 17:23:44 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:40324 with 2004.6 MB RAM, BlockManagerId(50, 192.168.30.68, 40324)
17/04/11 17:23:44 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:36507 with 2004.6 MB RAM, BlockManagerId(39, 192.168.30.68, 36507)
17/04/11 17:23:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39909) with ID 65
17/04/11 17:23:44 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, 192.168.30.69, partition 2, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:44 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 2 on executor id: 65 hostname: 192.168.30.69.
17/04/11 17:23:44 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:48265 with 2004.6 MB RAM, BlockManagerId(65, 192.168.30.69, 48265)
17/04/11 17:23:44 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:40324 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:45 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.69:48265 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:45 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:36507 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:45 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, 192.168.30.68, partition 3, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 3 on executor id: 50 hostname: 192.168.30.68.
17/04/11 17:23:45 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.30.68): java.lang.UnsatisfiedLinkError: no disni in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.ibm.disni.rdma.verbs.impl.NativeDispatcher.<clinit>(NativeDispatcher.java:36)
	at com.ibm.disni.rdma.verbs.impl.RdmaProviderNat.<init>(RdmaProviderNat.java:43)
	at com.ibm.disni.rdma.verbs.RdmaProvider.provider(RdmaProvider.java:58)
	at com.ibm.disni.rdma.verbs.RdmaCm.open(RdmaCm.java:49)
	at com.ibm.disni.rdma.verbs.RdmaEventChannel.createEventChannel(RdmaEventChannel.java:56)
	at com.ibm.disni.rdma.RdmaCmProcessor.<init>(RdmaCmProcessor.java:48)
	at com.ibm.disni.rdma.RdmaEndpointGroup.<init>(RdmaEndpointGroup.java:61)
	at com.ibm.darpc.RpcEndpointGroup.<init>(RpcEndpointGroup.java:47)
	at com.ibm.darpc.RpcClientGroup.<init>(RpcClientGroup.java:20)
	at com.ibm.darpc.RpcClientGroup.createClientGroup(RpcClientGroup.java:13)
	at com.ibm.crail.namenode.rpc.darpc.DaRPCNameNode.getRpcClient(DaRPCNameNode.java:57)
	at com.ibm.crail.core.CoreFileSystem.<init>(CoreFileSystem.java:119)
	at com.ibm.crail.CrailFS.newInstance(CrailFS.java:82)
	at org.apache.spark.storage.CrailStore.org$apache$spark$storage$CrailStore$$init(CrailStore.scala:110)
	at org.apache.spark.storage.CrailStore$.get(CrailStore.scala:589)
	at org.apache.spark.shuffle.crail.CrailShuffleWriter.<init>(CrailShuffleWriter.scala:45)
	at org.apache.spark.shuffle.crail.CrailShuffleManager.getWriter(CrailShuffleManager.scala:79)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:78)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
	at org.apache.spark.scheduler.Task.run(Task.scala:85)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

17/04/11 17:23:45 INFO TaskSetManager: Starting task 0.1 in stage 0.0 (TID 4, 192.168.30.69, partition 0, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 4 on executor id: 65 hostname: 192.168.30.69.
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56518) with ID 16
17/04/11 17:23:45 INFO TaskSetManager: Lost task 2.0 in stage 0.0 (TID 2) on executor 192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 1]
17/04/11 17:23:45 INFO TaskSetManager: Starting task 2.1 in stage 0.0 (TID 5, 192.168.30.67, partition 2, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 5 on executor id: 16 hostname: 192.168.30.67.
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56517) with ID 1
17/04/11 17:23:45 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 6, 192.168.30.67, partition 4, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 6 on executor id: 1 hostname: 192.168.30.67.
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:56395 with 2004.6 MB RAM, BlockManagerId(16, 192.168.30.67, 56395)
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:42897 with 2004.6 MB RAM, BlockManagerId(1, 192.168.30.67, 42897)
17/04/11 17:23:45 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 7, 192.168.30.68, partition 5, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:45 INFO TaskSetManager: Lost task 1.0 in stage 0.0 (TID 1) on executor 192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 2]
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 7 on executor id: 39 hostname: 192.168.30.68.
17/04/11 17:23:45 ERROR TaskSchedulerImpl: Lost executor 50 on 192.168.30.68: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/50 is now EXITED (Command exited with code 50)
17/04/11 17:23:45 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/50 removed: Command exited with code 50
17/04/11 17:23:45 WARN TaskSetManager: Lost task 3.0 in stage 0.0 (TID 3, 192.168.30.68): ExecutorLostFailure (executor 50 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:45 INFO DAGScheduler: Executor lost: 50 (epoch 0)
17/04/11 17:23:45 INFO BlockManagerMaster: Removal of executor 50 requested
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 50
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/89 on worker-20170411170842-192.168.30.68-41873 (192.168.30.68:41873) with 1 cores
17/04/11 17:23:45 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/89 on hostPort 192.168.30.68:41873 with 1 cores, 4.0 GB RAM
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/89 is now RUNNING
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 50 from BlockManagerMaster.
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(50, 192.168.30.68, 40324)
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 50 from BlockManagerMaster.
17/04/11 17:23:45 INFO BlockManagerMaster: Removed 50 successfully in removeExecutor
17/04/11 17:23:45 ERROR TaskSchedulerImpl: Lost executor 65 on 192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:45 WARN TaskSetManager: Lost task 0.1 in stage 0.0 (TID 4, 192.168.30.69): ExecutorLostFailure (executor 65 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:45 INFO DAGScheduler: Executor lost: 65 (epoch 1)
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 65 from BlockManagerMaster.
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(65, 192.168.30.69, 48265)
17/04/11 17:23:45 INFO BlockManagerMaster: Removed 65 successfully in removeExecutor
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/65 is now EXITED (Command exited with code 50)
17/04/11 17:23:45 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/65 removed: Command exited with code 50
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 65 from BlockManagerMaster.
17/04/11 17:23:45 INFO BlockManagerMaster: Removal of executor 65 requested
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 65
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/90 on worker-20170411170845-192.168.30.69-33975 (192.168.30.69:33975) with 1 cores
17/04/11 17:23:45 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/90 on hostPort 192.168.30.69:33975 with 1 cores, 4.0 GB RAM
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/90 is now RUNNING
17/04/11 17:23:45 ERROR TaskSchedulerImpl: Lost executor 39 on 192.168.30.68: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:45 WARN TaskSetManager: Lost task 5.0 in stage 0.0 (TID 7, 192.168.30.68): ExecutorLostFailure (executor 39 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:45 INFO DAGScheduler: Executor lost: 39 (epoch 2)
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 39 from BlockManagerMaster.
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(39, 192.168.30.68, 36507)
17/04/11 17:23:45 INFO BlockManagerMaster: Removed 39 successfully in removeExecutor
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/39 is now EXITED (Command exited with code 50)
17/04/11 17:23:45 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/39 removed: Command exited with code 50
17/04/11 17:23:45 INFO BlockManagerMasterEndpoint: Trying to remove executor 39 from BlockManagerMaster.
17/04/11 17:23:45 INFO BlockManagerMaster: Removal of executor 39 requested
17/04/11 17:23:45 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 39
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/91 on worker-20170411170842-192.168.30.68-41873 (192.168.30.68:41873) with 1 cores
17/04/11 17:23:45 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/91 on hostPort 192.168.30.68:41873 with 1 cores, 4.0 GB RAM
17/04/11 17:23:45 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/91 is now RUNNING
17/04/11 17:23:46 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.67:56395 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:46 INFO TaskSetManager: Starting task 5.1 in stage 0.0 (TID 8, 192.168.30.67, partition 5, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:46 INFO TaskSetManager: Lost task 2.1 in stage 0.0 (TID 5) on executor 192.168.30.67: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 3]
17/04/11 17:23:46 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 8 on executor id: 16 hostname: 192.168.30.67.
17/04/11 17:23:47 ERROR TaskSchedulerImpl: Lost executor 16 on 192.168.30.67: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:47 WARN TaskSetManager: Lost task 5.1 in stage 0.0 (TID 8, 192.168.30.67): ExecutorLostFailure (executor 16 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:47 INFO DAGScheduler: Executor lost: 16 (epoch 3)
17/04/11 17:23:47 INFO BlockManagerMasterEndpoint: Trying to remove executor 16 from BlockManagerMaster.
17/04/11 17:23:47 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(16, 192.168.30.67, 56395)
17/04/11 17:23:47 INFO BlockManagerMaster: Removed 16 successfully in removeExecutor
17/04/11 17:23:47 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/16 is now EXITED (Command exited with code 50)
17/04/11 17:23:47 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/16 removed: Command exited with code 50
17/04/11 17:23:47 INFO BlockManagerMaster: Removal of executor 16 requested
17/04/11 17:23:47 INFO BlockManagerMasterEndpoint: Trying to remove executor 16 from BlockManagerMaster.
17/04/11 17:23:47 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 16
17/04/11 17:23:47 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/92 on worker-20170411170843-192.168.30.67-35315 (192.168.30.67:35315) with 1 cores
17/04/11 17:23:47 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/92 on hostPort 192.168.30.67:35315 with 1 cores, 4.0 GB RAM
17/04/11 17:23:47 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/92 is now RUNNING
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49092) with ID 48
17/04/11 17:23:48 INFO TaskSetManager: Starting task 5.2 in stage 0.0 (TID 9, 192.168.30.68, partition 5, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 9 on executor id: 48 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49098) with ID 33
17/04/11 17:23:48 INFO TaskSetManager: Starting task 2.2 in stage 0.0 (TID 10, 192.168.30.68, partition 2, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 10 on executor id: 33 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49103) with ID 34
17/04/11 17:23:48 INFO TaskSetManager: Starting task 0.2 in stage 0.0 (TID 11, 192.168.30.68, partition 0, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 11 on executor id: 34 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49108) with ID 49
17/04/11 17:23:48 INFO TaskSetManager: Starting task 3.1 in stage 0.0 (TID 12, 192.168.30.68, partition 3, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 12 on executor id: 49 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49094) with ID 55
17/04/11 17:23:48 INFO TaskSetManager: Starting task 1.1 in stage 0.0 (TID 13, 192.168.30.68, partition 1, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 13 on executor id: 55 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:33256 with 2004.6 MB RAM, BlockManagerId(48, 192.168.30.68, 33256)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39922) with ID 63
17/04/11 17:23:48 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 14, 192.168.30.69, partition 6, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 14 on executor id: 63 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49110) with ID 37
17/04/11 17:23:48 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 15, 192.168.30.68, partition 7, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 15 on executor id: 37 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49107) with ID 35
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:58981 with 2004.6 MB RAM, BlockManagerId(33, 192.168.30.68, 58981)
17/04/11 17:23:48 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 16, 192.168.30.68, partition 8, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 16 on executor id: 35 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39921) with ID 64
17/04/11 17:23:48 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 17, 192.168.30.69, partition 9, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 17 on executor id: 64 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49111) with ID 54
17/04/11 17:23:48 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 18, 192.168.30.68, partition 10, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:36631 with 2004.6 MB RAM, BlockManagerId(34, 192.168.30.68, 36631)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 18 on executor id: 54 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49105) with ID 53
17/04/11 17:23:48 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 19, 192.168.30.68, partition 11, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 19 on executor id: 53 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49109) with ID 42
17/04/11 17:23:48 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 20, 192.168.30.68, partition 12, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 20 on executor id: 42 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:49717 with 2004.6 MB RAM, BlockManagerId(49, 192.168.30.68, 49717)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49097) with ID 38
17/04/11 17:23:48 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 21, 192.168.30.68, partition 13, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 21 on executor id: 38 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49114) with ID 45
17/04/11 17:23:48 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 22, 192.168.30.68, partition 14, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 22 on executor id: 45 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49100) with ID 57
17/04/11 17:23:48 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 23, 192.168.30.68, partition 15, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 23 on executor id: 57 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:45377 with 2004.6 MB RAM, BlockManagerId(55, 192.168.30.68, 45377)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:47494 with 2004.6 MB RAM, BlockManagerId(63, 192.168.30.69, 47494)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:42706 with 2004.6 MB RAM, BlockManagerId(35, 192.168.30.68, 42706)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49113) with ID 56
17/04/11 17:23:48 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 24, 192.168.30.68, partition 16, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 24 on executor id: 56 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:54510 with 2004.6 MB RAM, BlockManagerId(54, 192.168.30.68, 54510)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:49646 with 2004.6 MB RAM, BlockManagerId(37, 192.168.30.68, 49646)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:60526 with 2004.6 MB RAM, BlockManagerId(38, 192.168.30.68, 60526)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:45033 with 2004.6 MB RAM, BlockManagerId(53, 192.168.30.68, 45033)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49104) with ID 43
17/04/11 17:23:48 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 25, 192.168.30.68, partition 17, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 25 on executor id: 43 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:46145 with 2004.6 MB RAM, BlockManagerId(42, 192.168.30.68, 46145)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39935) with ID 69
17/04/11 17:23:48 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 26, 192.168.30.69, partition 18, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 26 on executor id: 69 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:55942 with 2004.6 MB RAM, BlockManagerId(57, 192.168.30.68, 55942)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:58690 with 2004.6 MB RAM, BlockManagerId(45, 192.168.30.68, 58690)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49116) with ID 46
17/04/11 17:23:48 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 27, 192.168.30.68, partition 19, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 27 on executor id: 46 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49117) with ID 44
17/04/11 17:23:48 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 28, 192.168.30.68, partition 20, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 28 on executor id: 44 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:44989 with 2004.6 MB RAM, BlockManagerId(56, 192.168.30.68, 44989)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39939) with ID 61
17/04/11 17:23:48 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 29, 192.168.30.69, partition 21, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 29 on executor id: 61 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:34905 with 2004.6 MB RAM, BlockManagerId(64, 192.168.30.69, 34905)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:51970 with 2004.6 MB RAM, BlockManagerId(43, 192.168.30.68, 51970)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39929) with ID 75
17/04/11 17:23:48 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 30, 192.168.30.69, partition 22, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 30 on executor id: 75 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49120) with ID 40
17/04/11 17:23:48 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 31, 192.168.30.68, partition 23, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 31 on executor id: 40 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39930) with ID 67
17/04/11 17:23:48 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 32, 192.168.30.69, partition 24, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 32 on executor id: 67 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49121) with ID 47
17/04/11 17:23:48 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 33, 192.168.30.68, partition 25, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 33 on executor id: 47 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49122) with ID 52
17/04/11 17:23:48 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 34, 192.168.30.68, partition 26, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 34 on executor id: 52 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49118) with ID 31
17/04/11 17:23:48 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 35, 192.168.30.68, partition 27, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 35 on executor id: 31 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:38887 with 2004.6 MB RAM, BlockManagerId(46, 192.168.30.68, 38887)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49123) with ID 51
17/04/11 17:23:48 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 36, 192.168.30.68, partition 28, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 36 on executor id: 51 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:54624 with 2004.6 MB RAM, BlockManagerId(44, 192.168.30.68, 54624)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49119) with ID 32
17/04/11 17:23:48 INFO TaskSetManager: Starting task 29.0 in stage 0.0 (TID 37, 192.168.30.68, partition 29, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 37 on executor id: 32 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49115) with ID 41
17/04/11 17:23:48 INFO TaskSetManager: Starting task 30.0 in stage 0.0 (TID 38, 192.168.30.68, partition 30, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 38 on executor id: 41 hostname: 192.168.30.68.
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:49003 with 2004.6 MB RAM, BlockManagerId(69, 192.168.30.69, 49003)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39948) with ID 62
17/04/11 17:23:48 INFO TaskSetManager: Starting task 31.0 in stage 0.0 (TID 39, 192.168.30.69, partition 31, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 39 on executor id: 62 hostname: 192.168.30.69.
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39953) with ID 70
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:58708 with 2004.6 MB RAM, BlockManagerId(40, 192.168.30.68, 58708)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:39928 with 2004.6 MB RAM, BlockManagerId(47, 192.168.30.68, 39928)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:56256 with 2004.6 MB RAM, BlockManagerId(67, 192.168.30.69, 56256)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:34588 with 2004.6 MB RAM, BlockManagerId(52, 192.168.30.68, 34588)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:59045 with 2004.6 MB RAM, BlockManagerId(61, 192.168.30.69, 59045)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:34362 with 2004.6 MB RAM, BlockManagerId(51, 192.168.30.68, 34362)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39951) with ID 66
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:59480 with 2004.6 MB RAM, BlockManagerId(32, 192.168.30.68, 59480)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39946) with ID 76
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:37423 with 2004.6 MB RAM, BlockManagerId(31, 192.168.30.68, 37423)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39955) with ID 85
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:52995 with 2004.6 MB RAM, BlockManagerId(41, 192.168.30.68, 52995)
17/04/11 17:23:48 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:55214 with 2004.6 MB RAM, BlockManagerId(75, 192.168.30.69, 55214)
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39962) with ID 81
17/04/11 17:23:48 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39954) with ID 80
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:48113 with 2004.6 MB RAM, BlockManagerId(62, 192.168.30.69, 48113)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:35131 with 2004.6 MB RAM, BlockManagerId(76, 192.168.30.69, 35131)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39952) with ID 77
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:52643 with 2004.6 MB RAM, BlockManagerId(66, 192.168.30.69, 52643)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:42134 with 2004.6 MB RAM, BlockManagerId(70, 192.168.30.69, 42134)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:35107 with 2004.6 MB RAM, BlockManagerId(85, 192.168.30.69, 35107)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49129) with ID 36
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39957) with ID 73
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39963) with ID 71
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:46678 with 2004.6 MB RAM, BlockManagerId(81, 192.168.30.69, 46678)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39956) with ID 86
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:57809 with 2004.6 MB RAM, BlockManagerId(80, 192.168.30.69, 57809)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39961) with ID 79
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:54223 with 2004.6 MB RAM, BlockManagerId(36, 192.168.30.68, 54223)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:41981 with 2004.6 MB RAM, BlockManagerId(77, 192.168.30.69, 41981)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39965) with ID 88
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39958) with ID 74
17/04/11 17:23:49 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:58981 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39970) with ID 78
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56548) with ID 24
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39966) with ID 82
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56553) with ID 10
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39960) with ID 84
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:57828 with 2004.6 MB RAM, BlockManagerId(73, 192.168.30.69, 57828)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:46638 with 2004.6 MB RAM, BlockManagerId(86, 192.168.30.69, 46638)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39959) with ID 58
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:41299 with 2004.6 MB RAM, BlockManagerId(71, 192.168.30.69, 41299)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39964) with ID 72
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39972) with ID 87
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56556) with ID 28
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:57691 with 2004.6 MB RAM, BlockManagerId(74, 192.168.30.69, 57691)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:48721 with 2004.6 MB RAM, BlockManagerId(88, 192.168.30.69, 48721)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39968) with ID 68
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:43580 with 2004.6 MB RAM, BlockManagerId(79, 192.168.30.69, 43580)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:56655 with 2004.6 MB RAM, BlockManagerId(84, 192.168.30.69, 56655)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:38635 with 2004.6 MB RAM, BlockManagerId(78, 192.168.30.69, 38635)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39967) with ID 60
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:58188 with 2004.6 MB RAM, BlockManagerId(58, 192.168.30.69, 58188)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:37275 with 2004.6 MB RAM, BlockManagerId(10, 192.168.30.67, 37275)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:38654 with 2004.6 MB RAM, BlockManagerId(82, 192.168.30.69, 38654)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:57057 with 2004.6 MB RAM, BlockManagerId(87, 192.168.30.69, 57057)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:42944 with 2004.6 MB RAM, BlockManagerId(72, 192.168.30.69, 42944)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39973) with ID 59
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:46147 with 2004.6 MB RAM, BlockManagerId(68, 192.168.30.69, 46147)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:33370 with 2004.6 MB RAM, BlockManagerId(28, 192.168.30.67, 33370)
17/04/11 17:23:49 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:36631 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:59261 with 2004.6 MB RAM, BlockManagerId(24, 192.168.30.67, 59261)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:54347 with 2004.6 MB RAM, BlockManagerId(60, 192.168.30.69, 54347)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:45307 with 2004.6 MB RAM, BlockManagerId(59, 192.168.30.69, 45307)
17/04/11 17:23:49 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.69:47494 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:49 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.69:49003 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56551) with ID 13
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56555) with ID 12
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56552) with ID 11
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56562) with ID 23
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56560) with ID 9
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56563) with ID 5
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39981) with ID 83
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56567) with ID 0
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56568) with ID 3
17/04/11 17:23:49 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.69:34905 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:57817 with 2004.6 MB RAM, BlockManagerId(83, 192.168.30.69, 57817)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:54843 with 2004.6 MB RAM, BlockManagerId(23, 192.168.30.67, 54843)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:55244 with 2004.6 MB RAM, BlockManagerId(13, 192.168.30.67, 55244)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56565) with ID 4
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56559) with ID 6
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:58766 with 2004.6 MB RAM, BlockManagerId(11, 192.168.30.67, 58766)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:47341 with 2004.6 MB RAM, BlockManagerId(5, 192.168.30.67, 47341)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:41938 with 2004.6 MB RAM, BlockManagerId(12, 192.168.30.67, 41938)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:48096 with 2004.6 MB RAM, BlockManagerId(9, 192.168.30.67, 48096)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:50704 with 2004.6 MB RAM, BlockManagerId(3, 192.168.30.67, 50704)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56571) with ID 20
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:57484 with 2004.6 MB RAM, BlockManagerId(0, 192.168.30.67, 57484)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56566) with ID 14
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56561) with ID 29
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56572) with ID 7
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56573) with ID 30
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:41106 with 2004.6 MB RAM, BlockManagerId(4, 192.168.30.67, 41106)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56569) with ID 19
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:50254 with 2004.6 MB RAM, BlockManagerId(20, 192.168.30.67, 50254)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:35493 with 2004.6 MB RAM, BlockManagerId(29, 192.168.30.67, 35493)
17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56574) with ID 15
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:36883 with 2004.6 MB RAM, BlockManagerId(14, 192.168.30.67, 36883)
17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:41664 with 2004.6 MB RAM, BlockManagerId(6, 192.168.30.67, 41664)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56570) with ID 2
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:60281 with 2004.6 MB RAM, BlockManagerId(7, 192.168.30.67, 60281)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:35042 with 2004.6 MB RAM, BlockManagerId(19, 192.168.30.67, 35042)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56576) with ID 21
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56578) with ID 8
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:40105 with 2004.6 MB RAM, BlockManagerId(30, 192.168.30.67, 40105)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:52143 with 2004.6 MB RAM, BlockManagerId(2, 192.168.30.67, 52143)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:41961 with 2004.6 MB RAM, BlockManagerId(15, 192.168.30.67, 41961)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56575) with ID 17
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:58767 with 2004.6 MB RAM, BlockManagerId(21, 192.168.30.67, 58767)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56577) with ID 27
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:59823 with 2004.6 MB RAM, BlockManagerId(8, 192.168.30.67, 59823)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56580) with ID 25
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:47906 with 2004.6 MB RAM, BlockManagerId(17, 192.168.30.67, 47906)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:36434 with 2004.6 MB RAM, BlockManagerId(27, 192.168.30.67, 36434)
17/04/11 17:23:50 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:33256 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56582) with ID 26
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:53112 with 2004.6 MB RAM, BlockManagerId(25, 192.168.30.67, 53112)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56585) with ID 18
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:52363 with 2004.6 MB RAM, BlockManagerId(26, 192.168.30.67, 52363)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56584) with ID 22
17/04/11 17:23:50 INFO TaskSetManager: Lost task 6.0 in stage 0.0 (TID 14) on executor 192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 4]
17/04/11 17:23:50 INFO TaskSetManager: Starting task 6.1 in stage 0.0 (TID 40, 192.168.30.69, partition 6, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 40 on executor id: 72 hostname: 192.168.30.69.
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:50961 with 2004.6 MB RAM, BlockManagerId(18, 192.168.30.67, 50961)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:55689 with 2004.6 MB RAM, BlockManagerId(22, 192.168.30.67, 55689)
17/04/11 17:23:50 INFO TaskSetManager: Lost task 18.0 in stage 0.0 (TID 26) on executor 192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 5]
17/04/11 17:23:50 INFO TaskSetManager: Starting task 18.1 in stage 0.0 (TID 41, 192.168.30.67, partition 18, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 41 on executor id: 4 hostname: 192.168.30.67.
17/04/11 17:23:50 INFO TaskSetManager: Lost task 9.0 in stage 0.0 (TID 17) on executor 192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 6]
17/04/11 17:23:50 INFO TaskSetManager: Starting task 9.1 in stage 0.0 (TID 42, 192.168.30.69, partition 9, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 42 on executor id: 59 hostname: 192.168.30.69.
17/04/11 17:23:50 INFO TaskSetManager: Lost task 2.2 in stage 0.0 (TID 10) on executor 192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 7]
17/04/11 17:23:50 INFO TaskSetManager: Starting task 2.3 in stage 0.0 (TID 43, 192.168.30.67, partition 2, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 43 on executor id: 10 hostname: 192.168.30.67.
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:40014) with ID 90
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.69:58570 with 2004.6 MB RAM, BlockManagerId(90, 192.168.30.69, 58570)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49178) with ID 89
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:60710 with 2004.6 MB RAM, BlockManagerId(89, 192.168.30.68, 60710)
17/04/11 17:23:50 ERROR TaskSchedulerImpl: Lost executor 33 on 192.168.30.68: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:50 INFO DAGScheduler: Executor lost: 33 (epoch 4)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Trying to remove executor 33 from BlockManagerMaster.
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(33, 192.168.30.68, 58981)
17/04/11 17:23:50 INFO BlockManagerMaster: Removed 33 successfully in removeExecutor
17/04/11 17:23:50 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/33 is now EXITED (Command exited with code 50)
17/04/11 17:23:50 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/33 removed: Command exited with code 50
17/04/11 17:23:50 INFO BlockManagerMaster: Removal of executor 33 requested
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Trying to remove executor 33 from BlockManagerMaster.
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 33
17/04/11 17:23:50 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/93 on worker-20170411170842-192.168.30.68-41873 (192.168.30.68:41873) with 1 cores
17/04/11 17:23:50 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/93 on hostPort 192.168.30.68:41873 with 1 cores, 4.0 GB RAM
17/04/11 17:23:50 ERROR TaskSchedulerImpl: Lost executor 63 on 192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:50 INFO DAGScheduler: Executor lost: 63 (epoch 5)
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Trying to remove executor 63 from BlockManagerMaster.
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(63, 192.168.30.69, 47494)
17/04/11 17:23:50 INFO BlockManagerMaster: Removed 63 successfully in removeExecutor
17/04/11 17:23:50 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/63 is now EXITED (Command exited with code 50)
17/04/11 17:23:50 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/63 removed: Command exited with code 50
17/04/11 17:23:50 INFO BlockManagerMasterEndpoint: Trying to remove executor 63 from BlockManagerMaster.
17/04/11 17:23:50 INFO BlockManagerMaster: Removal of executor 63 requested
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 63
17/04/11 17:23:50 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/94 on worker-20170411170845-192.168.30.69-33975 (192.168.30.69:33975) with 1 cores
17/04/11 17:23:50 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/94 on hostPort 192.168.30.69:33975 with 1 cores, 4.0 GB RAM
17/04/11 17:23:50 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/94 is now RUNNING
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.68:49180) with ID 91
17/04/11 17:23:50 INFO TaskSetManager: Lost task 0.2 in stage 0.0 (TID 11) on executor 192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 8]
17/04/11 17:23:50 INFO TaskSetManager: Starting task 0.3 in stage 0.0 (TID 44, 192.168.30.67, partition 0, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:50 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 44 on executor id: 29 hostname: 192.168.30.67.
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.68:49722 with 2004.6 MB RAM, BlockManagerId(91, 192.168.30.68, 49722)
17/04/11 17:23:51 ERROR TaskSchedulerImpl: Lost executor 64 on 192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:51 INFO DAGScheduler: Executor lost: 64 (epoch 6)
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 64 from BlockManagerMaster.
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(64, 192.168.30.69, 34905)
17/04/11 17:23:51 INFO BlockManagerMaster: Removed 64 successfully in removeExecutor
17/04/11 17:23:51 ERROR TaskSchedulerImpl: Lost executor 69 on 192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:51 INFO DAGScheduler: Executor lost: 69 (epoch 7)
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 69 from BlockManagerMaster.
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(69, 192.168.30.69, 49003)
17/04/11 17:23:51 INFO BlockManagerMaster: Removed 69 successfully in removeExecutor
17/04/11 17:23:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/64 is now EXITED (Command exited with code 50)
17/04/11 17:23:51 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/64 removed: Command exited with code 50
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 64 from BlockManagerMaster.
17/04/11 17:23:51 INFO BlockManagerMaster: Removal of executor 64 requested
17/04/11 17:23:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 64
17/04/11 17:23:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/95 on worker-20170411170845-192.168.30.69-33975 (192.168.30.69:33975) with 1 cores
17/04/11 17:23:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/95 on hostPort 192.168.30.69:33975 with 1 cores, 4.0 GB RAM
17/04/11 17:23:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/69 is now EXITED (Command exited with code 50)
17/04/11 17:23:51 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/69 removed: Command exited with code 50
17/04/11 17:23:51 INFO BlockManagerMaster: Removal of executor 69 requested
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 69 from BlockManagerMaster.
17/04/11 17:23:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 69
17/04/11 17:23:51 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/96 on worker-20170411170845-192.168.30.69-33975 (192.168.30.69:33975) with 1 cores
17/04/11 17:23:51 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/96 on hostPort 192.168.30.69:33975 with 1 cores, 4.0 GB RAM
17/04/11 17:23:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/95 is now RUNNING
17/04/11 17:23:51 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/96 is now RUNNING
17/04/11 17:23:51 ERROR TaskSchedulerImpl: Lost executor 34 on 192.168.30.68: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:51 INFO DAGScheduler: Executor lost: 34 (epoch 8)
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Trying to remove executor 34 from BlockManagerMaster.
17/04/11 17:23:51 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(34, 192.168.30.68, 36631)
17/04/11 17:23:51 INFO BlockManagerMaster: Removed 34 successfully in removeExecutor
17/04/11 17:23:51 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.69:45307 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:51 INFO TaskSetManager: Lost task 9.1 in stage 0.0 (TID 42) on executor 192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 9]
17/04/11 17:23:51 INFO TaskSetManager: Starting task 9.2 in stage 0.0 (TID 45, 192.168.30.67, partition 9, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 45 on executor id: 28 hostname: 192.168.30.67.
17/04/11 17:23:52 ERROR TaskSchedulerImpl: Lost executor 59 on 192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:52 INFO DAGScheduler: Executor lost: 59 (epoch 9)
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 59 from BlockManagerMaster.
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(59, 192.168.30.69, 45307)
17/04/11 17:23:52 INFO BlockManagerMaster: Removed 59 successfully in removeExecutor
17/04/11 17:23:52 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/59 is now EXITED (Command exited with code 50)
17/04/11 17:23:52 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/59 removed: Command exited with code 50
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 59 from BlockManagerMaster.
17/04/11 17:23:52 INFO BlockManagerMaster: Removal of executor 59 requested
17/04/11 17:23:52 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 59
17/04/11 17:23:52 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/97 on worker-20170411170845-192.168.30.69-33975 (192.168.30.69:33975) with 1 cores
17/04/11 17:23:52 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/97 on hostPort 192.168.30.69:33975 with 1 cores, 4.0 GB RAM
17/04/11 17:23:52 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/97 is now RUNNING
17/04/11 17:23:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.67:33370 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:49646 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:45033 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:49717 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:52 INFO TaskSetManager: Lost task 5.2 in stage 0.0 (TID 9) on executor 192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 10]
17/04/11 17:23:52 INFO TaskSetManager: Starting task 5.3 in stage 0.0 (TID 46, 192.168.30.67, partition 5, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:52 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 46 on executor id: 9 hostname: 192.168.30.67.
17/04/11 17:23:52 INFO TaskSetManager: Lost task 9.2 in stage 0.0 (TID 45) on executor 192.168.30.67: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 11]
17/04/11 17:23:52 INFO TaskSetManager: Starting task 9.3 in stage 0.0 (TID 47, 192.168.30.69, partition 9, PROCESS_LOCAL, 5640 bytes)
17/04/11 17:23:52 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Launching task 47 on executor id: 82 hostname: 192.168.30.69.
17/04/11 17:23:52 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.67:56622) with ID 92
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.30.67:50012 with 2004.6 MB RAM, BlockManagerId(92, 192.168.30.67, 50012)
17/04/11 17:23:52 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:45377 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:52 ERROR TaskSchedulerImpl: Lost executor 28 on 192.168.30.67: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.
17/04/11 17:23:52 INFO DAGScheduler: Executor lost: 28 (epoch 10)
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 28 from BlockManagerMaster.
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(28, 192.168.30.67, 33370)
17/04/11 17:23:52 INFO BlockManagerMaster: Removed 28 successfully in removeExecutor
17/04/11 17:23:52 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/28 is now EXITED (Command exited with code 50)
17/04/11 17:23:52 INFO StandaloneSchedulerBackend: Executor app-20170411172340-0002/28 removed: Command exited with code 50
17/04/11 17:23:52 INFO BlockManagerMaster: Removal of executor 28 requested
17/04/11 17:23:52 INFO BlockManagerMasterEndpoint: Trying to remove executor 28 from BlockManagerMaster.
17/04/11 17:23:52 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asked to remove non-existent executor 28
17/04/11 17:23:52 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20170411172340-0002/98 on worker-20170411170843-192.168.30.67-35315 (192.168.30.67:35315) with 1 cores
17/04/11 17:23:52 INFO StandaloneSchedulerBackend: Granted executor ID app-20170411172340-0002/98 on hostPort 192.168.30.67:35315 with 1 cores, 4.0 GB RAM
17/04/11 17:23:52 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20170411172340-0002/98 is now RUNNING
17/04/11 17:23:53 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:42706 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:53 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.69:38654 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:53 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.68:55942 (size: 1891.0 B, free: 2004.6 MB)
17/04/11 17:23:53 INFO TaskSetManager: Lost task 9.3 in stage 0.0 (TID 47) on executor 192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 12]
17/04/11 17:23:53 ERROR TaskSetManager: Task 9 in stage 0.0 failed 4 times; aborting job
17/04/11 17:23:53 INFO TaskSchedulerImpl: Cancelling stage 0
17/04/11 17:23:53 INFO TaskSchedulerImpl: Stage 0 was cancelled
17/04/11 17:23:53 INFO DAGScheduler: ShuffleMapStage 0 (newAPIHadoopFile at TeraSort.scala:104) failed in 11.152 s
17/04/11 17:23:53 INFO DAGScheduler: Job 0 failed: saveAsNewAPIHadoopFile at TeraSort.scala:146, took 11.558018 s
17/04/11 17:23:53 INFO SparkUI: Stopped Spark web UI at http://192.168.30.70:4040
17/04/11 17:23:53 INFO StandaloneSchedulerBackend: Shutting down all executors
17/04/11 17:23:53 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
17/04/11 17:23:53 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/04/11 17:23:53 INFO CrailShuffleManager: shutting down crail shuffle manager
17/04/11 17:23:53 INFO CrailStore: stopping CrailStore
17/04/11 17:23:53 INFO crail: Closing CrailFS singleton
17/04/11 17:23:53 INFO crail: mapped client cache closed
17/04/11 17:23:53 INFO disni: shutting down group
17/04/11 17:23:53 INFO disni: closing client endpoint
17/04/11 17:23:53 INFO disni: disconnect, id 0
17/04/11 17:23:53 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:152)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:132)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:571)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:179)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:108)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
17/04/11 17:23:53 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:152)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:132)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:571)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:179)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:108)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
17/04/11 17:23:53 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:152)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:132)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:571)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:179)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:108)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
17/04/11 17:23:53 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message.
org.apache.spark.SparkException: Could not find CoarseGrainedScheduler.
	at org.apache.spark.rpc.netty.Dispatcher.postMessage(Dispatcher.scala:152)
	at org.apache.spark.rpc.netty.Dispatcher.postOneWayMessage(Dispatcher.scala:132)
	at org.apache.spark.rpc.netty.NettyRpcHandler.receive(NettyRpcEnv.scala:571)
	at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequestHandler.java:179)
	at org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:108)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:119)
	at org.apache.spark.network.server.TransportChannelHandler.channelRead0(TransportChannelHandler.java:51)
	at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:266)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:308)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:294)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:846)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
17/04/11 17:23:53 WARN TransportChannelHandler: Exception in connection from /192.168.30.68:49092
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
17/04/11 17:23:53 WARN TransportChannelHandler: Exception in connection from /192.168.30.69:39966
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
	at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:313)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:881)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:242)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:119)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:511)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
	at java.lang.Thread.run(Thread.java:745)
17/04/11 17:23:54 INFO disni: destroyQP, id 0
17/04/11 17:23:54 INFO disni: destroyCmId, id 0
17/04/11 17:23:54 INFO disni: closing client done
17/04/11 17:23:54 INFO disni: closing client endpoint
17/04/11 17:23:54 INFO disni: disconnect, id 0
17/04/11 17:23:55 INFO disni: destroyQP, id 0
17/04/11 17:23:55 INFO disni: destroyCmId, id 0
17/04/11 17:23:55 INFO disni: closing client done
17/04/11 17:23:55 INFO disni: closing client endpoint
17/04/11 17:23:55 INFO disni: disconnect, id 0
17/04/11 17:23:56 INFO disni: destroyQP, id 0
17/04/11 17:23:56 INFO disni: destroyCmId, id 0
17/04/11 17:23:56 INFO disni: closing client done
17/04/11 17:23:56 INFO disni: shutting down cm processor
17/04/11 17:23:56 INFO disni: terminating cm polling, closed true
17/04/11 17:23:56 INFO disni: cm processor down
17/04/11 17:23:56 INFO disni: destroyEventChannel, channel 0
17/04/11 17:23:56 INFO disni: cm channel down
17/04/11 17:23:56 INFO disni: shutting down group done
17/04/11 17:23:56 INFO disni: closing client endpoint
17/04/11 17:23:56 INFO disni: disconnect, id 0
17/04/11 17:23:57 INFO disni: destroyQP, id 0
17/04/11 17:23:57 INFO disni: destroyCmId, id 0
17/04/11 17:23:57 INFO disni: closing client done
17/04/11 17:23:57 INFO disni: shutting down group
17/04/11 17:23:57 INFO disni: shutting down cm processor
17/04/11 17:23:57 INFO disni: terminating cm polling, closed true
17/04/11 17:23:57 INFO disni: cm processor down
17/04/11 17:23:57 INFO disni: destroyEventChannel, channel 0
17/04/11 17:23:57 INFO disni: cm channel down
17/04/11 17:23:57 INFO disni: shutting down group done
17/04/11 17:23:57 INFO darpc: rpc group down
17/04/11 17:23:57 INFO crail: CrailStatistics, tag=close
17/04/11 17:23:57 INFO crail: provider=IO/input [total 0, localOps 0, remoteOps 0, localDirOps 0, remoteDirOps 0, cached 0, nonBlocking 0, blocking 0, prefetched 0, prefetchedNonBlocking 0, prefetchedBlocking 0, capacity 0, totalStreams 0, avgCapacity 0, avgOpLen 0]
17/04/11 17:23:57 INFO crail: provider=MappedBufferCache [cacheGet 33, cachePut 33, cacheMiss 0, cacheSize 0, cacheMax 33, cacheMissMap 0, cacheMissHeap 0]
17/04/11 17:23:57 INFO crail: provider=IO/output [total 33, localOps 0, remoteOps 33, localDirOps 0, remoteDirOps 33, cached 33, nonBlocking 0, blocking 0, prefetched 0, prefetchedNonBlocking 0, prefetchedBlocking 0, capacity 130040320, totalStreams 33, avgCapacity 3940615, avgOpLen 512]
17/04/11 17:23:57 INFO crail: provider=Streams [open 33, openInput 0, openOutput 33, openInputDir 0, openOutputDir 33, close 33, closeInput 0, closeOutput 33, closeInputDir 0, closeOutputDir 33, maxInput 0, maxOutput 33]
17/04/11 17:23:57 INFO crail: provider=EndpointCache [size 3]
17/04/11 17:23:57 INFO MemoryStore: MemoryStore cleared
17/04/11 17:23:57 INFO BlockManager: BlockManager stopped
17/04/11 17:23:57 INFO BlockManagerMaster: BlockManagerMaster stopped
17/04/11 17:23:57 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/04/11 17:23:57 INFO SparkContext: Successfully stopped SparkContext
Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 0.0 failed 4 times, most recent failure: Lost task 9.3 in stage 0.0 (TID 47, 192.168.30.69): java.lang.UnsatisfiedLinkError: no disni in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.ibm.disni.rdma.verbs.impl.NativeDispatcher.<clinit>(NativeDispatcher.java:36)
	at com.ibm.disni.rdma.verbs.impl.RdmaProviderNat.<init>(RdmaProviderNat.java:43)
	at com.ibm.disni.rdma.verbs.RdmaProvider.provider(RdmaProvider.java:58)
	at com.ibm.disni.rdma.verbs.RdmaCm.open(RdmaCm.java:49)
	at com.ibm.disni.rdma.verbs.RdmaEventChannel.createEventChannel(RdmaEventChannel.java:56)
	at com.ibm.disni.rdma.RdmaCmProcessor.<init>(RdmaCmProcessor.java:48)
	at com.ibm.disni.rdma.RdmaEndpointGroup.<init>(RdmaEndpointGroup.java:61)
	at com.ibm.darpc.RpcEndpointGroup.<init>(RpcEndpointGroup.java:47)
	at com.ibm.darpc.RpcClientGroup.<init>(RpcClientGroup.java:20)
	at com.ibm.darpc.RpcClientGroup.createClientGroup(RpcClientGroup.java:13)
	at com.ibm.crail.namenode.rpc.darpc.DaRPCNameNode.getRpcClient(DaRPCNameNode.java:57)
	at com.ibm.crail.core.CoreFileSystem.<init>(CoreFileSystem.java:119)
	at com.ibm.crail.CrailFS.newInstance(CrailFS.java:82)
	at org.apache.spark.storage.CrailStore.org$apache$spark$storage$CrailStore$$init(CrailStore.scala:110)
	at org.apache.spark.storage.CrailStore$.get(CrailStore.scala:589)
	at org.apache.spark.shuffle.crail.CrailShuffleWriter.<init>(CrailShuffleWriter.scala:45)
	at org.apache.spark.shuffle.crail.CrailShuffleManager.getWriter(CrailShuffleManager.scala:79)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:78)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
	at org.apache.spark.scheduler.Task.run(Task.scala:85)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1450)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1438)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1437)
	at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1437)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
	at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:811)
	at scala.Option.foreach(Option.scala:257)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:811)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1659)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1618)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1607)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1871)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1884)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:1904)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply$mcV$sp(PairRDDFunctions.scala:1151)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1.apply(PairRDDFunctions.scala:1078)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopDataset(PairRDDFunctions.scala:1078)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply$mcV$sp(PairRDDFunctions.scala:998)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:989)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$2.apply(PairRDDFunctions.scala:989)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:989)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:977)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$1.apply(PairRDDFunctions.scala:977)
	at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopFile$1.apply(PairRDDFunctions.scala:977)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
	at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
	at org.apache.spark.rdd.RDD.withScope(RDD.scala:358)
	at org.apache.spark.rdd.PairRDDFunctions.saveAsNewAPIHadoopFile(PairRDDFunctions.scala:976)
	at com.ibm.crail.terasort.TeraSort$.doSortWithOpts(TeraSort.scala:146)
	at com.ibm.crail.terasort.TeraSort$.main(TeraSort.scala:51)
	at com.ibm.crail.terasort.TeraSort.main(TeraSort.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:729)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.UnsatisfiedLinkError: no disni in java.library.path
	at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
	at java.lang.Runtime.loadLibrary0(Runtime.java:870)
	at java.lang.System.loadLibrary(System.java:1122)
	at com.ibm.disni.rdma.verbs.impl.NativeDispatcher.<clinit>(NativeDispatcher.java:36)
	at com.ibm.disni.rdma.verbs.impl.RdmaProviderNat.<init>(RdmaProviderNat.java:43)
	at com.ibm.disni.rdma.verbs.RdmaProvider.provider(RdmaProvider.java:58)
	at com.ibm.disni.rdma.verbs.RdmaCm.open(RdmaCm.java:49)
	at com.ibm.disni.rdma.verbs.RdmaEventChannel.createEventChannel(RdmaEventChannel.java:56)
	at com.ibm.disni.rdma.RdmaCmProcessor.<init>(RdmaCmProcessor.java:48)
	at com.ibm.disni.rdma.RdmaEndpointGroup.<init>(RdmaEndpointGroup.java:61)
	at com.ibm.darpc.RpcEndpointGroup.<init>(RpcEndpointGroup.java:47)
	at com.ibm.darpc.RpcClientGroup.<init>(RpcClientGroup.java:20)
	at com.ibm.darpc.RpcClientGroup.createClientGroup(RpcClientGroup.java:13)
	at com.ibm.crail.namenode.rpc.darpc.DaRPCNameNode.getRpcClient(DaRPCNameNode.java:57)
	at com.ibm.crail.core.CoreFileSystem.<init>(CoreFileSystem.java:119)
	at com.ibm.crail.CrailFS.newInstance(CrailFS.java:82)
	at org.apache.spark.storage.CrailStore.org$apache$spark$storage$CrailStore$$init(CrailStore.scala:110)
	at org.apache.spark.storage.CrailStore$.get(CrailStore.scala:589)
	at org.apache.spark.shuffle.crail.CrailShuffleWriter.<init>(CrailShuffleWriter.scala:45)
	at org.apache.spark.shuffle.crail.CrailShuffleManager.getWriter(CrailShuffleManager.scala:79)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:78)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
	at org.apache.spark.scheduler.Task.run(Task.scala:85)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Jonas Pfefferle1

unread,

Apr 11, 2017, 5:44:48 AM4/11/17

to sunja...@gmail.com, zrlio-users

Make sure you put the LD_LIBRARY_PATH in the very beginning of your bashrc file otherwise it will not be picked up in non-interactive environments. Another alternative is to use "spark.executor.extraJavaOptions" with "-Djava.library.path= <your path to disni>"

Regards,
Jonas

sunjane007---11/04/2017 11:37:33---Hi, I encounter a weird problem when running crail-terasort. When I run the part "sorting data usin

17/04/11 17:23:49 INFO CoarseGrainedSchedulerBackend$Drive rEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.30.69:39957) with ID 73

at org.apache.spark.network.server.TransportRequestHandler.processOneWayMessage(TransportRequest Handler.java:179)

--
You received this message because you are subscribed to the Google Groups "zrlio-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zrlio-users...@googlegroups.com.
To post to this group, send email to zrlio...@googlegroups.com.
Visit this group at https://groups.google.com/group/zrlio-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/zrlio-users/81edb5b8-aa9b-416e-bcd3-4d3a8969b165%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

sunja...@gmail.com

unread,

Apr 11, 2017, 8:51:40 AM4/11/17

to zrlio-users, sunja...@gmail.com, J...@zurich.ibm.com

在 2017年4月11日星期二 UTC+8下午5:44:48，Jonas Pfefferle1写道：

To unsubscribe from this group and stop receiving emails from it, send an email to zrlio-users+unsubscribe@googlegroups.com.

Jonas Pfefferle1

unread,

Apr 11, 2017, 8:57:34 AM4/11/17

to sunja...@gmail.com, zrlio-users

Indeed. This works if you are using spark standalone, however I believe it does not work if you are using yarn. The alternative there is to put it in yarn-site.xml:

<property>
<name>yarn.nodemanager.admin-env</name>
<value>LD_LIBRARY_PATH=path_to_crail_lib</value>
</property>

sunjane007---11/04/2017 14:52:07---Thank you very much. I didn't try the first one. But It works with the alternative solution. By

From: sunja...@gmail.com
To: zrlio-users <zrlio...@googlegroups.com>
Cc: sunja...@gmail.com, J...@zurich.ibm.com
Date: 11/04/2017 14:52
Subject: Re: [zrlio-users] java.lang.UnsatisfiedLinkError (no disni in java.library.path) when running crail-terasort

Thank you very much.
I didn't try the first one. But It works with the alternative solution. By the way, I find that adding "export LD_LIBRARY_PATH=$CRAIL_HOME/lib:$LD_LIBRARY_PATH " in spark-env.sh also works.

在 2017年4月11日星期二 UTC+8下午5:44:48，Jonas Pfefferle1写道：
Make sure you put the LD_LIBRARY_PATH in the very beginning of your bashrc file otherwise it will not be picked up in non-interactive environments. Another alternative is to use "spark.executor.extraJavaOptions" with "-Djava.library.path= <your path to disni>"

Regards,
Jonas

with 2004.6 MB RAM, BlockManagerId(84, 192.168.30.69, 56655) 17/04/11 17:23:49 INFO BlockManagerMasterEndpoint: Registering block manager192.168.30.69:38635 with 2004. MB RAM, BlockManagerId(78, 192.168.30.69, 38635)

192.168.30.69:39967) with ID 60

192.168.30.69:58188 with 2004.6 MB RAM, BlockManagerId(58, 192.168.30.69, 58188)

192.168.30.67:37275 with 2004.6 MB RAM, BlockManagerId(10, 192.168.30.67, 37275)

192.168.30.69:38654 with 2004.6 MB RAM, BlockManagerId(82, 192.168.30.69, 38654)

192.168.30.69:57057 with 2004.6 MB RAM, BlockManagerId(87, 192.168.30.69, 57057)

192.168.30.69:42944 with 2004.6 MB RAM, BlockManagerId(72, 192.168.30.69, 42944)

192.168.30.69:39973) with ID 59

192.168.30.69:46147 with 2004.6 MB RAM, BlockManagerId(68, 192.168.30.69, 46147)

192.168.30.67:33370 with 2004.6 MB RAM, BlockManagerId(28, 192.168.30.67, 33370)

192.168.30.68:36631 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.67:59261 with 2004.6 MB RAM, BlockManagerId(24, 192.168.30.67, 59261)

192.168.30.69:54347 with 2004.6 MB RAM, BlockManagerId(60, 192.168.30.69, 54347)

192.168.30.69:45307 with 2004.6 MB RAM, BlockManagerId(59, 192.168.30.69, 45307)

192.168.30.69:47494 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.69:49003 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.67:56551) with ID 13

192.168.30.67:56555) with ID 12

192.168.30.67:56552) with ID 11

192.168.30.67:56562) with ID 23

192.168.30.67:56560) with ID 9

192.168.30.67:56563) with ID 5

192.168.30.69:39981) with ID 83

192.168.30.67:56567) with ID 0

192.168.30.67:56568) with ID 3

192.168.30.69:34905 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.69:57817 with 2004.6 MB RAM, BlockManagerId(83, 192.168.30.69, 57817)

192.168.30.67:54843 with 2004.6 MB RAM, BlockManagerId(23, 192.168.30.67, 54843)

192.168.30.67:55244 with 2004.6 MB RAM, BlockManagerId(13, 192.168.30.67, 55244)

192.168.30.67:56565) with ID 4

192.168.30.67:56559) with ID 6

192.168.30.67:58766 with 2004.6 MB RAM, BlockManagerId(11, 192.168.30.67, 58766)

192.168.30.67:47341 with 2004.6 MB RAM, BlockManagerId(5, 192.168.30.67, 47341)

192.168.30.67:41938 with 2004.6 MB RAM, BlockManagerId(12, 192.168.30.67, 41938)

192.168.30.67:48096 with 2004.6 MB RAM, BlockManagerId(9, 192.168.30.67, 48096)

192.168.30.67:50704 with 2004.6 MB RAM, BlockManagerId(3, 192.168.30.67, 50704)

192.168.30.67:56571) with ID 20

192.168.30.67:57484 with 2004.6 MB RAM, BlockManagerId(0, 192.168.30.67, 57484)

192.168.30.67:56566) with ID 14

192.168.30.67:56561) with ID 29

192.168.30.67:56572) with ID 7

192.168.30.67:56573) with ID 30

192.168.30.67:41106 with 2004.6 MB RAM, BlockManagerId(4, 192.168.30.67, 41106)

192.168.30.67:56569) with ID 19

192.168.30.67:50254 with 2004.6 MB RAM, BlockManagerId(20, 192.168.30.67, 50254)

192.168.30.67:35493 with 2004.6 MB RAM, BlockManagerId(29, 192.168.30.67, 35493)

192.168.30.67:56574) with ID 15

192.168.30.67:36883 with 2004.6 MB RAM, BlockManagerId(14, 192.168.30.67, 36883)

192.168.30.67:41664 with 2004.6 MB RAM, BlockManagerId(6, 192.168.30.67, 41664)

192.168.30.67:56570) with ID 2

192.168.30.67:60281 with 2004.6 MB RAM, BlockManagerId(7, 192.168.30.67, 60281)

192.168.30.67:35042 with 2004.6 MB RAM, BlockManagerId(19, 192.168.30.67, 35042)

192.168.30.67:56576) with ID 21

192.168.30.67:56578) with ID 8

192.168.30.67:40105 with 2004.6 MB RAM, BlockManagerId(30, 192.168.30.67, 40105)

192.168.30.67:52143 with 2004.6 MB RAM, BlockManagerId(2, 192.168.30.67, 52143)

192.168.30.67:41961 with 2004.6 MB RAM, BlockManagerId(15, 192.168.30.67, 41961)

192.168.30.67:56575) with ID 17

192.168.30.67:58767 with 2004.6 MB RAM, BlockManagerId(21, 192.168.30.67, 58767)

192.168.30.67:56577) with ID 27

192.168.30.67:59823 with 2004.6 MB RAM, BlockManagerId(8, 192.168.30.67, 59823)

192.168.30.67:56580) with ID 25

192.168.30.67:47906 with 2004.6 MB RAM, BlockManagerId(17, 192.168.30.67, 47906)

192.168.30.67:36434 with 2004.6 MB RAM, BlockManagerId(27, 192.168.30.67, 36434)

192.168.30.68:33256 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.67:56582) with ID 26

192.168.30.67:53112 with 2004.6 MB RAM, BlockManagerId(25, 192.168.30.67, 53112)

192.168.30.67:56585) with ID 18

192.168.30.67:52363 with 2004.6 MB RAM, BlockManagerId(26, 192.168.30.67, 52363)

192.168.30.67:56584) with ID 22

192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 4]

192.168.30.67:50961 with 2004.6 MB RAM, BlockManagerId(18, 192.168.30.67, 50961)

192.168.30.67:55689 with 2004.6 MB RAM, BlockManagerId(22, 192.168.30.67, 55689)

192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 5]

192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 6]

192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 7]

192.168.30.69:40014) with ID 90

192.168.30.69:58570 with 2004.6 MB RAM, BlockManagerId(90, 192.168.30.69, 58570)

192.168.30.68:49178) with ID 89

192.168.30.68:60710 with 2004.6 MB RAM, BlockManagerId(89, 192.168.30.68, 60710)

192.168.30.68: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

192.168.30.68:41873) with 1 cores

192.168.30.68:41873 with 1 cores, 4.0 GB RAM

192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

192.168.30.69:33975) with 1 cores

192.168.30.69:33975 with 1 cores, 4.0 GB RAM

192.168.30.68:49180) with ID 91

192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 8]

192.168.30.68:49722 with 2004.6 MB RAM, BlockManagerId(91, 192.168.30.68, 49722)

192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

192.168.30.69:33975) with 1 cores

192.168.30.69:33975 with 1 cores, 4.0 GB RAM

192.168.30.69:33975) with 1 cores

192.168.30.69:33975 with 1 cores, 4.0 GB RAM

192.168.30.68: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

192.168.30.69:45307 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 9]

192.168.30.69: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

192.168.30.69:33975) with 1 cores

192.168.30.69:33975 with 1 cores, 4.0 GB RAM

192.168.30.67:33370 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.68:49646 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.68:45033 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.68:49717 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.68: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 10]

192.168.30.67: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 11]

192.168.30.67:56622) with ID 92

192.168.30.67:50012 with 2004.6 MB RAM, BlockManagerId(92, 192.168.30.67, 50012)

192.168.30.68:45377 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.67: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

192.168.30.67:35315) with 1 cores

192.168.30.67:35315 with 1 cores, 4.0 GB RAM

192.168.30.68:42706 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.69:38654 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.68:55942 (size: 1891.0 B, free: 2004.6 MB)

192.168.30.69: java.lang.UnsatisfiedLinkError (no disni in java.library.path) [duplicate 12]

http://192.168.30.70:4040

To unsubscribe from this group and stop receiving emails from it, send an email to zrlio-users...@googlegroups.com.

Reply all

Reply to author

Forward

0 new messages