StackOverflowError when completing futures

[James Roper]

Hi all,

I have encountered a StackOverflowError when using futures, and I'm trying to work out if it's a bug in my code, or a bug in Scala futures, but I'm a bit stuck.  Firstly, this is on Scala 2.10-RC1, so if there's been any fixes relating to this in futures since RC1 please let me know.

The context is in Plays iteratees, which are heavily based on futures.  It seems to only happen when processing large streams, which potentially mean thousands or even millions of flatMap calls as the stream processing progresses.

The immediate error is this exception thrown:

java.lang.IllegalStateException: Promise already completed.

at scala.concurrent.Promise$class.complete(Promise.scala:55)

at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)

at scala.concurrent.Promise$class.failure(Promise.scala:107)

at scala.concurrent.impl.Promise$DefaultPromise.failure(Promise.scala:58)

at scala.concurrent.Future$$anonfun$flatMap$1.liftedTree3$1(Future.scala:283)

at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:277)

at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:274)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:722)

Which is a bit misleading.  The problem is here that a StackOverflowError is encountered while trying to complete the promise, leaving the promise in a bad state.  The future API tries to recover from this by trying to complete the promise with a failure, which fails with the error above.  This is probably a bug in the promise API that should be fixed, because the original error is of course lost, and the actual exception that's being handled at that point is "java.lang.IllegalStateException: problem in scala.concurrent internal callback", with many (hundreds) of chained causes, eventually leading to the StackOverflowError.  If an internal callback failed, perhaps we should treat this separately, and not try and continue to complete the promise?

So after looking at it with a debugger, The StackOverflowError is apparent.  Here is the sequence of calls that repeat themselves over and over:

at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$liftedTree3$1$1.apply(Future.scala:280)

at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$liftedTree3$1$1.apply(Future.scala:278)

at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:29)

at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:703)

at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:37)

at scala.concurrent.impl.Promise$DefaultPromise$$anonfun$tryComplete$1.apply(Promise.scala:133)

at scala.concurrent.impl.Promise$DefaultPromise$$anonfun$tryComplete$1.apply(Promise.scala:133)

at scala.collection.immutable.List.foreach(List.scala:309)

at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:133)

at scala.concurrent.Promise$class.complete(Promise.scala:55)

at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:58)

at scala.concurrent.Promise$class.success(Promise.scala:89)

at scala.concurrent.impl.Promise$DefaultPromise.success(Promise.scala:58)

at scala.concurrent.Future$$anonfun$flatMap$1$$anonfun$liftedTree3$1$1.apply(Future.scala:280)

I've tried to reproduce this in the repl, but without success, I've got no idea how to get things into this state.  Chaining many flatMap calls on a future doesn't work, since each callback is executed in a passed in execution context, which in my case is the default fork join execution context.

So first of all, does anyone know how things could get into this state?  Is there anything I should be looking in my (or Play's iteratees) code?

Cheers,

James

[James Roper]

I think I posted here too soon, I just found out that it's probably a bug in the Play iteratee API:

https://groups.google.com/forum/?fromgroups=#!topic/play-framework/oVZZl_K_dtI