Strange stack trace

[Tyson]

java.util.NoSuchElementException: key not found: Stage 0
at scala.collection.MapLike$class.default(MapLike.scala:224)
at scala.collection.mutable.HashMap.default(HashMap.scala:43)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.mutable.HashMap.apply(HashMap.scala:43)
at spark.DAGScheduler$class.runJob(DAGScheduler.scala:224)
at spark.MesosScheduler.runJob(MesosScheduler.scala:25)
at spark.SparkContext.runJob(SparkContext.scala:255)
at spark.SparkContext.runJob(SparkContext.scala:262)
at spark.SparkContext.runJob(SparkContext.scala:269)
at spark.SparkContext.runJob(SparkContext.scala:282)
at spark.RDD.count(RDD.scala:182)
at com.yahoo.SimplePageRank$.main(SimplePageRank.scala:46)
at com.yahoo.SimplePageRank.main(SimplePageRank.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:
39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:
25)
at java.lang.reflect.Method.invoke(Method.java:597)
at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run
$1.apply(ScalaClassLoader.scala:78)
at scala.tools.nsc.util.ScalaClassLoader
$class.asContext(ScalaClassLoader.scala:24)
at scala.tools.nsc.util.ScalaClassLoader
$URLClassLoader.asContext(ScalaClassLoader.scala:88)
at scala.tools.nsc.util.ScalaClassLoader
$class.run(ScalaClassLoader.scala:78)
at scala.tools.nsc.util.ScalaClassLoader
$URLClassLoader.run(ScalaClassLoader.scala:101)
at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
at scala.tools.nsc.MainGenericRunner.runTarget
$1(MainGenericRunner.scala:56)
at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:
80)
at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:
89)
at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

[Tyson]

Here is the code in question. Note: I'm using count to force the
evaluation of a single iteration. I need to measure per-iteration
times... is there a better way to do this?

for (i <- 0 until iters) {
timestamp = System.currentTimeMillis
val contribs = links.groupWith(ranks).flatMap {
case (id, (Seq(links), Seq(rank))) =>
links.map(dest => (dest, rank / links.length))
case (id, (Seq(links), Seq())) =>
links.map(dest => (dest, 0.15 / n / links.length))
case (id, (Seq(), Seq(rank))) => // Vertex had no links list
new Array[(Long, Double)](0)
}
def sum(x: Double, y: Double) = x + y
ranks = contribs.combineByKey(identity, sum, sum, parallelism)
.mapValues(sum => 0.15 / n + 0.85 * sum)

val totalRanks = ranks.count
println("Number of ranks: " + totalRanks)
println("Iteration time: " + ((System.currentTimeMillis -
timestamp) / 1000.0) + " seconds.")
/*
// Non-combiner version
ranks = contribs.groupByKey(vp).mapValues(rs => 0.15/n +
0.85*rs.sum)
*/

[Matei Zaharia]

I've seen something like this before but unfortunately I'm not sure what the cause is. Are any of your tasks failing, by any chance? That could be it (in particular if there are map fetch failures). Otherwise this requires debugging the DAGScheduler code more.

Matei