As results from tasks are progressively increased, it appears the errors such as connection refused.
31 views
Skip to first unread message
armsir33
Jul 28, 2012, 9:22:50 AM7/28/12
to spark...@googlegroups.com
Hi,
I ran an iterative program where output in the current iteration is treated as input in the next iteration. Another key matter is that the result at each iteration is expanded, say output is much larger than input. For example, if the amount of input is 1G, after several iterations, it might turn to 2G or 3G. In addition, as known that the size of result at each task should be at most a few MBs, it leads to exceptions of connection refused otherwise. However, at the context of my program, it exists such risk with high possibility. The degree of parallelism is definitely constant, for example, 72. As iterations keep advance unceasingly, it could predict, at some point, the size of result at each task will reach the upper limit. Eventually, it leads to an exception. Is there any way which could avoid this happening.
Thanks
I ran an iterative program where output in the current iteration is treated as input in the next iteration. Another key matter is that the result at each iteration is expanded, say output is much larger than input. For example, if the amount of input is 1G, after several iterations, it might turn to 2G or 3G. In addition, as known that the size of result at each task should be at most a few MBs, it leads to exceptions of connection refused otherwise. However, at the context of my program, it exists such risk with high possibility. The degree of parallelism is definitely constant, for example, 72. As iterations keep advance unceasingly, it could predict, at some point, the size of result at each task will reach the upper limit. Eventually, it leads to an exception. Is there any way which could avoid this happening.
Thanks
armsir33
Jul 28, 2012, 9:27:49 AM7/28/12
to spark...@googlegroups.com
The following is log on one slave when run a 3-iteration program.(Running one-iteration or 2-itertaion works well )
12/07/28 14:58:04 INFO spark.BoundedMemoryCache: BoundedMemoryCache.maxBytes = 5433148047 12/07/28 14:58:05 INFO spark.ShuffleManager: Shuffle dir: /tmp/spark-local-faeff06c-f1c9-46f4-baae-b1074aa02d84/shuffle 12/07/28 14:58:05 INFO server.Server: jetty-7.5.3.v20111011 12/07/28 14:58:05 INFO server.AbstractConnector: Started SelectChann...@0.0.0.0:41963 STARTING 12/07/28 14:58:05 INFO spark.ShuffleManager: Local URI: http://193.10.64.200:41963 12/07/28 14:58:05 INFO spark.Executor: Running task ID 33 12/07/28 14:58:05 INFO spark.Executor: Running task ID 30 12/07/28 14:58:05 INFO spark.Executor: Running task ID 27 12/07/28 14:58:05 INFO spark.Executor: Running task ID 28 12/07/28 14:58:05 INFO spark.Executor: Running task ID 31 12/07/28 14:58:05 INFO spark.Executor: Running task ID 32 12/07/28 14:58:05 INFO spark.Executor: Running task ID 25 12/07/28 14:58:05 INFO spark.Executor: Running task ID 26 12/07/28 14:58:05 INFO spark.Executor: Running task ID 29 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 31 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 28 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 25 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 29 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 33 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 30 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 26 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 27 12/07/28 14:58:16 INFO spark.Executor: Finished task ID 32 12/07/28 14:58:16 INFO spark.Executor: Running task ID 36 12/07/28 14:58:16 INFO spark.Executor: Running task ID 39 12/07/28 14:58:16 INFO spark.Executor: Running task ID 42 12/07/28 14:58:16 INFO spark.Executor: Running task ID 45 12/07/28 14:58:16 INFO spark.Executor: Running task ID 48 12/07/28 14:58:16 INFO spark.Executor: Running task ID 51 12/07/28 14:58:16 INFO spark.Executor: Running task ID 54 12/07/28 14:58:16 INFO spark.Executor: Running task ID 57 12/07/28 14:58:16 INFO spark.Executor: Running task ID 60 12/07/28 14:58:16 INFO spark.Executor: Running task ID 63 12/07/28 14:58:16 INFO spark.Executor: Running task ID 69 12/07/28 14:58:16 INFO spark.Executor: Running task ID 66 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 18 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 27 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 3 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 24 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 6 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.MapOutputTracker: Doing the fetch; tracker actor = 'MapOutputTracker@Node(193.10.64.107,7077) 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 15 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 9 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 0 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 12 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 21 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 33 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:16 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 30 12/07/28 14:58:16 INFO spark.MapOutputTracker: Don't have map outputs for 0, fetching them 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 51 12/07/28 14:58:18 INFO spark.Executor: Running task ID 75 12/07/28 14:58:18 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 39 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 45 12/07/28 14:58:18 INFO spark.Executor: Running task ID 78 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 63 12/07/28 14:58:18 INFO spark.Executor: Running task ID 80 12/07/28 14:58:18 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 42 12/07/28 14:58:18 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 44 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 69 12/07/28 14:58:18 INFO spark.Executor: Running task ID 92 12/07/28 14:58:18 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 56 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 57 12/07/28 14:58:18 INFO spark.Executor: Running task ID 97 12/07/28 14:58:18 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 61 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 80 12/07/28 14:58:18 INFO spark.Executor: Running task ID 100 12/07/28 14:58:18 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 64 12/07/28 14:58:18 INFO spark.Executor: Finished task ID 39 12/07/28 14:58:18 INFO spark.Executor: Running task ID 102 12/07/28 14:58:19 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 66 12/07/28 14:58:19 INFO spark.Executor: Finished task ID 97 12/07/28 14:58:19 INFO spark.Executor: Running task ID 109 12/07/28 14:58:19 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 0, reduce 73 12/07/28 14:58:19 INFO spark.Executor: Finished task ID 75 12/07/28 14:58:19 INFO spark.Executor: Finished task ID 92 12/07/28 14:58:20 INFO spark.Executor: Finished task ID 100 12/07/28 14:58:20 INFO spark.Executor: Finished task ID 109 12/07/28 14:58:24 INFO spark.Executor: Finished task ID 60 12/07/28 14:58:29 INFO spark.Executor: Finished task ID 48 12/07/28 14:58:29 INFO spark.Executor: Finished task ID 102 12/07/28 14:58:32 INFO spark.Executor: Finished task ID 36 12/07/28 14:58:32 INFO spark.Executor: Finished task ID 54 12/07/28 14:58:32 INFO spark.Executor: Finished task ID 78 12/07/28 14:59:01 INFO spark.Executor: Finished task ID 42 12/07/28 15:00:20 INFO spark.Executor: Finished task ID 66 12/07/28 15:00:22 INFO spark.Executor: Running task ID 168 12/07/28 15:00:22 INFO spark.Executor: Running task ID 169 12/07/28 15:00:22 INFO spark.Executor: Running task ID 170 12/07/28 15:00:22 INFO spark.Executor: Running task ID 171 12/07/28 15:00:22 INFO spark.Executor: Running task ID 172 12/07/28 15:00:22 INFO spark.Executor: Running task ID 173 12/07/28 15:00:22 INFO spark.Executor: Running task ID 174 12/07/28 15:00:22 INFO spark.Executor: Running task ID 175 12/07/28 15:00:22 INFO spark.Executor: Running task ID 176 12/07/28 15:00:22 INFO spark.Executor: Running task ID 177 12/07/28 15:00:22 INFO spark.Executor: Running task ID 178 12/07/28 15:00:22 INFO spark.Executor: Running task ID 179 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 39 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 37 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.MapOutputTracker: Doing the fetch; tracker actor = 'MapOutputTracker@Node(193.10.64.107,7077) 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 38 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 36 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 40 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 41 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 42 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 43 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 45 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 46 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 44 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:22 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 47 12/07/28 15:00:22 INFO spark.MapOutputTracker: Don't have map outputs for 1, fetching them 12/07/28 15:00:23 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 37 12/07/28 15:00:23 INFO spark.MapOutputTracker: Don't have map outputs for 2, fetching them 12/07/28 15:00:23 INFO spark.MapOutputTracker: Doing the fetch; tracker actor = 'MapOutputTracker@Node(193.10.64.107,7077) 12/07/28 15:00:23 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 45 12/07/28 15:00:24 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 36 12/07/28 15:00:24 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 41 12/07/28 15:00:24 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 40 12/07/28 15:00:24 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 39 12/07/28 15:00:24 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 43 12/07/28 15:00:24 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 46 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 47 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:25 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:25 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:25 INFO spark.Executor: Running task ID 206 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 73 12/07/28 15:00:25 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:760) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:25 INFO spark.Executor: Running task ID 208 12/07/28 15:00:25 INFO spark.Executor: Running task ID 209 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 75 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 76 12/07/28 15:00:25 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 42 12/07/28 15:00:25 INFO spark.Executor: Running task ID 210 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 77 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 2, reduce 38 12/07/28 15:00:25 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:25 INFO spark.Executor: Running task ID 215 12/07/28 15:00:25 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 4 12/07/28 15:00:26 ERROR spark.SimpleShuffleFetcher: Fetch failed java.io.FileNotFoundException: http://193.10.64.86:42710/shuffle/2/90/46 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1434) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:26 ERROR spark.SimpleShuffleFetcher: Fetch failed java.io.FileNotFoundException: http://193.10.64.86:42710/shuffle/2/90/43 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1434) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:26 INFO spark.Executor: Running task ID 264 12/07/28 15:00:26 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 22 12/07/28 15:00:26 ERROR spark.SimpleShuffleFetcher: Fetch failed java.io.FileNotFoundException: http://193.10.64.86:42710/shuffle/2/90/42 at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1434) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:26 INFO spark.Executor: Running task ID 265 12/07/28 15:00:26 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 23 12/07/28 15:00:26 INFO spark.Executor: Running task ID 266 12/07/28 15:00:26 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 24 12/07/28 15:00:26 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:26 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:26 INFO spark.Executor: Running task ID 286 12/07/28 15:00:26 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 43 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:646) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:27 INFO spark.Executor: Running task ID 338 12/07/28 15:00:27 INFO spark.Executor: Running task ID 339 12/07/28 15:00:27 INFO spark.Executor: Running task ID 340 12/07/28 15:00:27 INFO spark.Executor: Running task ID 341 12/07/28 15:00:27 INFO spark.Executor: Running task ID 342 12/07/28 15:00:27 INFO spark.Executor: Running task ID 343 12/07/28 15:00:27 INFO spark.Executor: Running task ID 344 12/07/28 15:00:27 INFO spark.Executor: Running task ID 345 12/07/28 15:00:27 INFO spark.Executor: Running task ID 346 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 95 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:335) at sun.net.www.protocol.http.HttpURLConnection.setNewClient(HttpURLConnection.java:650) at sun.net.www.protocol.http.HttpURLConnection.setNewClient(HttpURLConnection.java:638) at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:584) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1193) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 0 12/07/28 15:00:27 INFO spark.Executor: Running task ID 347 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 1 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 2 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 3 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 5 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 4 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 6 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 7 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 348 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 8 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 9 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 349 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 10 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 350 12/07/28 15:00:27 INFO spark.Executor: Running task ID 351 12/07/28 15:00:27 INFO spark.Executor: Running task ID 352 12/07/28 15:00:27 INFO spark.Executor: Running task ID 353 12/07/28 15:00:27 INFO spark.Executor: Running task ID 354 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 12 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 13 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 14 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 15 12/07/28 15:00:27 INFO spark.Executor: Running task ID 355 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 11 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 357 12/07/28 15:00:27 INFO spark.Executor: Running task ID 356 12/07/28 15:00:27 INFO spark.Executor: Running task ID 358 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 18 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 17 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 16 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 359 12/07/28 15:00:27 INFO spark.Executor: Running task ID 360 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 19 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 20 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 21 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 375 12/07/28 15:00:27 INFO spark.Executor: Running task ID 378 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 36 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 39 12/07/28 15:00:27 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:27 INFO spark.Executor: Running task ID 389 12/07/28 15:00:27 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 50 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 INFO spark.Executor: Running task ID 401 12/07/28 15:00:28 INFO spark.Executor: Running task ID 402 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 62 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 63 12/07/28 15:00:28 INFO spark.Executor: Running task ID 407 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 68 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 INFO spark.Executor: Running task ID 421 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 INFO spark.Executor: Running task ID 435 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 16 12/07/28 15:00:28 INFO spark.Executor: Running task ID 438 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 INFO spark.Executor: Running task ID 443 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 84 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 87 12/07/28 15:00:28 INFO spark.Executor: Running task ID 445 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 92 12/07/28 15:00:28 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 94 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:28 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:29 INFO spark.Executor: Running task ID 447 12/07/28 15:00:29 INFO spark.SimpleShuffleFetcher: Fetching outputs for shuffle 1, reduce 24 12/07/28 15:00:29 ERROR spark.SimpleShuffleFetcher: Fetch failed java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366) at java.net.Socket.connect(Socket.java:529) at java.net.Socket.connect(Socket.java:478) at sun.net.NetworkClient.doConnect(NetworkClient.java:163) at sun.net.www.http.HttpClient.openServer(HttpClient.java:394) at sun.net.www.http.HttpClient.openServer(HttpClient.java:529) at sun.net.www.http.HttpClient.<init>(HttpClient.java:233) at sun.net.www.http.HttpClient.New(HttpClient.java:306) at sun.net.www.http.HttpClient.New(HttpClient.java:323) at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970) at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911) at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1172) at java.net.URL.openStream(URL.java:1010) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply$mcVI$sp(SimpleShuffleFetcher.scala:27) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5$$anonfun$apply$1.apply(SimpleShuffleFetcher.scala:21) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:21) at spark.SimpleShuffleFetcher$$anonfun$fetch$5.apply(SimpleShuffleFetcher.scala:20) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:44) at spark.SimpleShuffleFetcher.fetch(SimpleShuffleFetcher.scala:20) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:89) at spark.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:74) at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) at scala.collection.immutable.List.foreach(List.scala:45) at spark.CoGroupedRDD.compute(CoGroupedRDD.scala:74) at spark.RDD.iterator(RDD.scala:78) at spark.MappedValuesRDD.compute(PairRDDFunctions.scala:389) at spark.RDD.iterator(RDD.scala:78) at spark.FlatMappedRDD.compute(RDD.scala:283) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.FilteredRDD.compute(RDD.scala:289) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.MappedRDD.compute(RDD.scala:273) at spark.RDD.iterator(RDD.scala:78) at spark.UnionSplit.iterator(UnionRDD.scala:12) at spark.UnionRDD.compute(UnionRDD.scala:46) at spark.RDD.iterator(RDD.scala:78) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:27) at spark.ShuffleMapTask.run(ShuffleMapTask.scala:10) at spark.Executor$TaskRunner.run(Executor.scala:74) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) 12/07/28 15:00:29 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:29 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:29 WARN io.nio: java.net.SocketException: Transport endpoint is not connected 12/07/28 15:00:29 WARN io.nio: java.net.SocketException: Transport endpoint is not connected
Reply all
Reply to author
Forward
0 new messages