Akka Future vs. Twitter's Finagle Future

7,269 views
Skip to first unread message

Michael Slinn

unread,
Feb 12, 2012, 12:29:23 AM2/12/12
to akka...@googlegroups.com
I'm comparing Akka's implementation of Future against Twitter's Finagle (API is here). Below are some impressions from a quick first pass comparing the two. I'm looking forward to our usual lively discussion :)

Both have onSuccess and onFailure; only Akka has onComplete.

Finagle has a Timer thread on which to wait if a timeout value was specified; not sure if this is a dedicated thread or a multiplexed thread. If dedicated, doesn't that inflate the thread count horribly? I looked at Akka's source and it seems that futures are interrupted periodically in order to ask "Have I timed out yet?" ... keeps the thread count down, and probably costs less than a thread context switch.

Finagle: use try/catch
Akka: various error handling constructs, including recover and the seriously under-documented recoverWith (a code example would be very nice - hint, hint!)

Finagle: "You are discouraged from creating your own Promises. Instead, where possible, use Future combinators to compose actions."
Akka: Use of Promises is encouraged and is very useful.

Finagle has Future combinators. Future.select, Future.join
Akka is missing select and join

Finagle's Future.collectO seems similar to Akka's Future.sequence() and traverse()

Finagle does not have Await.result() so many lines of code need to be written if a timeout handler is desired.

Twitter's "Effective Scala" states "vars must be declared volatile in order for them to be published to other threads"
No mention of volatile in the Akka docs. Seems like that cannot be the whole story. Too much has been left unsaid by the Akka docs and the Finagle docs.

Finagle's FuturePool seems like an overly complex alternative to Akka's Future.blocking()

Finagle's Future.proxyTo seems very nice, not aware of a similar feature in the Akka Future API.

Finagle is missing Future.withFilter(); users won't miss it, however.

Finagle requires the user to set up a j.u.c. Executor, no nice configurable Akka factory.

What have I missed or misquoted?

Mike

√iktor Ҡlang

unread,
Feb 12, 2012, 6:08:49 AM2/12/12
to akka...@googlegroups.com
On Sun, Feb 12, 2012 at 6:29 AM, Michael Slinn <msl...@gmail.com> wrote:
> I'm comparing Akka's implementation of Future against Twitter's Finagle (API
> is here). Below are some impressions from a quick first pass comparing the
> two. I'm looking forward to our usual lively discussion :)
>
> Both have onSuccess and onFailure; only Akka has onComplete.
>
> Finagle has a Timer thread on which to wait if a timeout value was
> specified; not sure if this is a dedicated thread or a multiplexed thread.
> If dedicated, doesn't that inflate the thread count horribly? I looked at
> Akka's source and it seems that futures are interrupted periodically in
> order to ask "Have I timed out yet?" ... keeps the thread count down, and
> probably costs less than a thread context switch.

The default Akka Scheduler is based on the Netty implementation of
HashedWheelTimer.

>
> Finagle: use try/catch
> Akka: various error handling constructs, including recover and the seriously
> under-documented recoverWith (a code example would be very nice - hint,
> hint!)

Finagle uses the Try construct, Akka uses Either.

>
> Finagle: "You are discouraged from creating your own Promises. Instead,
> where possible, use Future combinators to compose actions."
> Akka: Use of Promises is encouraged and is very useful.

Yes, using Promises directly means you can use the Dataflow API, which
is very nice from a syntactical perspective.

>
> Finagle has Future combinators. Future.select, Future.join
> Akka is missing select and join

I don't know what select/join does, but akka has: fallbackTo and zip

>
> Finagle's Future.collectO seems similar to Akka's Future.sequence() and
> traverse()
>
> Finagle does not have Await.result() so many lines of code need to be
> written if a timeout handler is desired.
>
> Twitter's "Effective Scala" states "vars must be declared volatile in order
> for them to be published to other threads"
> No mention of volatile in the Akka docs.

We should probably add a section about Futures and the JMM in the Akka
docs, open a ticket. We already have Actors & JMM and STM & JMM

Seems like that cannot be the whole
> story. Too much has been left unsaid by the Akka docs and the Finagle docs.

Like what? We can't fix what you don't say.

>
> Finagle's FuturePool seems like an overly complex alternative to Akka's
> Future.blocking()
>
> Finagle's Future.proxyTo seems very nice, not aware of a similar feature in
> the Akka Future API.

Promise.completeWith

>
> Finagle is missing Future.withFilter(); users won't miss it, however.
>
> Finagle requires the user to set up a j.u.c. Executor, no nice configurable
> Akka factory.

You mean like ThreadPoolConfig?

Cheers,

>
> What have I missed or misquoted?
>
> Mike
>

> --
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/akka-user/-/3VrLg855SrAJ.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to
> akka-user+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/akka-user?hl=en.

--
Viktor Klang

Akka Tech Lead
Typesafe - The software stack for applications that scale

Twitter: @viktorklang

Michael Slinn

unread,
Feb 12, 2012, 10:43:50 AM2/12/12
to akka...@googlegroups.com
Logged http://www.assembla.com/spaces/akka/tickets/1813--section-on-futures-and-the-jmm-in-the-akka-docs

I need to learn more about Finagle's timer implementation.

Rereading my posting I realized that I was not clear when I attempted to say that Akka has a nice configuration mechanism for creating threadpools, and they are enhanced beyond what j.u.c. provides. Finagle uses vanilla threadpools or the user must create their own custom threadpools. Testing custom threadpools is non-trivial.

Finagle's Future.join() docs say "Combines two Futures into one Future of the Tuple of the two results", which sounds exactly like Akka's Future.zip().

I did a Google search of the Akka docs and there is no mention of Promise.completeWith(); the Akka ScalaDoc clearly states that Promise.completeWith() is like Finagle's Future.proxyTo() From the Akka docs: "Completes this Promise with the specified other Future, when that Future is completed, unless this Promise has already been completed."

I wonder how easy it would be to integrate Finagle into existing code, or if it is better suited for new applications.

Mike

√iktor Ҡlang

unread,
Feb 12, 2012, 11:29:50 AM2/12/12
to akka...@googlegroups.com

Wdym?

Cheers,
V

>
>
> Mike
>
> --
> You received this message because you are subscribed to the Google Groups "Akka User List" group.

> To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/9RbDnkjvQiEJ.

Michael Slinn

unread,
Feb 12, 2012, 11:55:45 AM2/12/12
to akka...@googlegroups.com
Given an existing code base, if it were necessary to retrofit features that required futures, I wonder if the task might be more easily accomplished with Akka futures or Finagle futures. Integration of Akka futures is readily accomplished with Promise, and the rest of Akka does not have to be used and imposes no additional requirements.

I see now that Finagle uses the Future implementation from Twitter Utils, so Finagle is not required if you want futures but are not specifically interested in RPC. Not sure what other dependencies Twitter futures might require in order to be useful. Twitter's Scala School shows a Callback object but I don't see it in the code.

BTW, it looks like Twitter Futures does indeed require a thread for each future that wants a Timer. That is not a good use of resources. I am beginning to read about HashedWheelTimer now, looking at source, viewing presentation. I wonder if HashedWheelTimer would work with Twitter Futures.

Mike

√iktor Ҡlang

unread,
Feb 12, 2012, 12:03:43 PM2/12/12
to akka...@googlegroups.com
On Sun, Feb 12, 2012 at 5:55 PM, Michael Slinn <msl...@gmail.com> wrote:
> Given an existing code base, if it were necessary to retrofit features that
> required futures, I wonder if the task might be more easily accomplished
> with Akka futures or Finagle futures. Integration of Akka futures is readily
> accomplished with Promise, and the rest of Akka does not have to be used and
> imposes no additional requirements.

Akka Futures will essentially become the Scala standard library
Futures with SIP-14

>
> I see now that Finagle uses the Future implementation from Twitter Utils, so
> Finagle is not required if you want futures but are not specifically
> interested in RPC. Not sure what other dependencies Twitter futures might
> require in order to be useful. Twitter's Scala School shows a Callback
> object but I don't see it in the code.

Twitter's Futures and Promises are from twitter-util, and are
standalone in that sense.
I'm hoping with SIP-14 that the Scala ecosystem will gravitate towards
using the standard library,
so we don't need 12372834627834 different Futures implementations.

>
> BTW, it looks like Twitter Futures does indeed require a thread for each
> future that wants a Timer. That is not a good use of resources. I am
> beginning to read about HashedWheelTimer now, looking at source, viewing
> presentation. I wonder if HashedWheelTimer would work with Twitter Futures.

I can only speak for Akka Futures & Promises.

Cheers,

>
>
> Mike
>
> --
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/akka-user/-/GBdNYE0-6gcJ.


>
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to
> akka-user+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/akka-user?hl=en.

--

marius a. eriksen

unread,
Feb 12, 2012, 5:11:01 PM2/12/12
to akka...@googlegroups.com
Hey guys -- I'm Marius from Twitter. Nick Kallen and I have done most
of the design and implementation of Twitter's future library. It has
evolved quite drastically over time to meet our production needs.

Big disclaimer: I have not worked with Akka's futures. I only know
them from their documentation and API docs as well as from SIP-14.
Also: we've been working with Phillip, Heather, and co. to see if we
can consolidate these APIs somehow. As Viktor said, it'd be nice if
there were just one API, and maybe even just one implementation.

Akka's and Twitter's futures are fundamentally quite similar. First let me
clear up some confusion:

1) We discourage the creation of your own promises, but that
doesn't mean you can't. It's sound advice: most operations can
be phrased in terms of composition of other Futures (of course
the sources of these futures need to use Promises), and using
the various combinators generally leads to better and more
bug-free code.
2) Twitter futures don't prescribe any execution environment.
That is: they are purely a coordination mechanism, and
callbacks are run in whatever thread satisfies the promise, or
in the same thread if the promise is already complete. This
does mean that whenever you have blocking computation, you
need to take care to execute these in your own threadpool --
but again, we think of Futures as the coordination mechanism,
and if there is blocking work, we represent the result of this
as another future which can be combined in whatever ways are
approriate.

> Both have onSuccess and onFailure; only Akka has onComplete.

onComplete sounds like Twitter's "ensure"

> Finagle has a Timer thread on which to wait if a timeout value was
> specified; not sure if this is a dedicated thread or a multiplexed
> thread. If dedicated, doesn't that inflate the thread count horribly?
> I looked at Akka's source and it seems that futures are interrupted
> periodically in order to ask "Have I timed out yet?" ... keeps the
> thread count down, and probably costs less than a thread context
> switch.

You pass in a timer to Future.within -- this is typically a shared
timer. For example in Finagle we have only one timer that's shared
across all timeouts. Like akka, we use Netty's HashedWheelTimer.

> Finagle: use try/catch Akka: various error handling constructs,
> including recover and the seriously under-documented recoverWith (a
> code example would be very nice - hint, hint!)

Twitter's futures have the same rich API; "handle" is analagous to "map" for
errors, and "rescue" is analagous to "flatMap".

> Finagle does not have Await.result() so many lines of code need to be
> written if a timeout handler is desired.

Actually that's false, you can do the same with Twitter futures:

val f: Future[Int]
val x: Int = f(10.seconds)

> Twitter's "Effective Scala" states "vars must be declared volatile in
> order for them to be published to other threads" No mention of
> volatile in the Akka docs. Seems like that cannot be the whole story.
> Too much has been left unsaid by the Akka docs and the Finagle docs.

The remark on volatility is merely a general observation. However, you'll find the
same document encourages the "transformational"/declarative style (ie.
using combinators, as discussed above), since it usually avoids the
subtle corners of the memory model, and usually doesn't even require
locking.

> Finagle's FuturePool seems like an overly complex alternative to
> Akka's Future.blocking()

I'm not sure it's overly complex. It's more explicit.

*

Now, this being said, I think the biggest departure of Twitter's
futures over Akka's is the addition of cancellation, future chaining,
and "tail call optimization" (future merging). These were all added
as we gained experience with a growing codebase relying on Futures
for most of their concurrent coordination needs. I wrote the below
to the SIP-14 team in order to explain the motivaiton behind each
of these features.

CALLBACK INVOCATION ORDERING

As far as I can tell there is no way to provide callback ordering
guarantees with this SIP. That makes it quite difficult to implement
"try-finally" patterns. We do this quite often in our code:

       val resource = acquire()
       resource.operation() onSuccess { value =>
         resource.anotherOperation()
       } ensure {
         resource.release()
       }

Twitter futures support this by specifying ordering: ``respond'' (and
thus ``onSuccess'', ``onFailure'', ``ensure'', etc.) all return new
Futures that are guaranteed to dispatch only after the parent future
dispatches all of its callbacks. We relax the constraints a little bit:
if the ordering is visible according to the Java memory model, it is
also respected in callback dispatch.

CANCELLATION

I am tempted to recommend not even including cancellation as they do
much to complicate both implementation and semantics, but absent
external coordination, they are absolutely required in a well-behaved
network client. I have become convinced they are a necessary evil of
Futures. Their chief use is to avoid dogpiling. For example: in
Finagle, if a client becomes temporarily unavailable, its effective
latency will increase (due to retry policies, failure backoffs, etc.)
and new requests will begin to queue. At the same time, the server
that is issuing the request (most actors in distributed systems are
both servers and clients) may time out, and return a truncated respose
to *its* client. In this situation, most of the work that has been
queued is anyway useless. It just creates more latency, and this
snowballs.

This is akin to committing the control-theory-sin of not dropping from
back of the queue. Cancellation is the mechanism that allows us to do
just that: if queued work is anyway useless, we can avoid doing it.

All of this being said, I believe that our solution actually presents
the best of both worlds. While adding implementation complexity, it
avoids the pitfalls of cancellation: Twitter futures do not support
cancellation per sé, but rather a cancellation *signal* (level
triggered). The cancellation signal flows in the opposite direction of
values (and are similarly propagated - "linked" - by the combinators).
So for example:

       f flatMap { _ => f2 flatMap { _ => f3 } }

Calling ``f.cancel()'' will eventually make it to f3, should the
intermediate futures be successful. Cancelling a future does nothing
but set the signal. If a ``completer'' supports cancellation, they can
query and listen to the signal. Thus in the above example, whomever
controls ``f3'' may listen to cancellation:

       f3.onCancellation { /* execute some arbitrary code */ }

And that's it. Futures support nothing else. However, typically the
value updater (should they support cancellation), would complete the
promise with an appropriate exception.

In finagle, we only had to add two onCancellation clauses for this all
to work.

Crucially, doing this ensures that there's still *only one updater*.
Futures are still immutable and no third state is introduced. To see
why this is important (beyond implementation simplicity), consider
a pattern like the following:

       trait Worker {
         def work(request: Request): Future[Response]
         def release() // give it back when done
       }

       type Factory = () => Worker

       *

       val worker = factory()

       val response = worker.work(request) ensure { worker.release() }

If calling ``response.cancel()'' would mutate the future, and cancel
it right away, the ensure clause would also be executed, giving the
worker back to the pool, when it was not yet ready.

Having just one value updater path avoids these pitfalls and more.

FUTURE MERGING AND TAIL RECURSION

Another thing that comes up when using Futures is that without
implementation support, there is no "tail call optimization". Consider
the future-adapted version of a "classic" example of tail recursion:

       def compute(soFar: List[Result]): Future[List[Result]] = rpc() flatMap {
         case More(item) => compute(item :: soFar)
         case Done => soFar.reverse
       }

       compute(Nil)

A naïve implementation would simply link all of these futures
together, creating a space leak:

       outer = a flatMap b flatMap flatMap c flatMap d flatMap …

Whereas they actually all represent the same result, and so TCO
applies in the same way it does to traditional tail-recursive code.

We support this in twitter Futures by merging futures in flatMaps. We
have a bunch of code in this style, and it provides nice "expected"
and analogous behavior to other code. However, if I were to do it
again, I'm not sure I would introduce it. The implementation is quite
hairy (though I believe as simple as can be) and only a few people
understand it thoroughly.

Havoc Pennington

unread,
Feb 12, 2012, 6:55:25 PM2/12/12
to akka...@googlegroups.com
Hi,

On Sun, Feb 12, 2012 at 5:11 PM, marius a. eriksen
<marius....@gmail.com> wrote:
> 2) Twitter futures don't prescribe any execution environment.
> That is: they are purely a coordination mechanism, and
> callbacks are run in whatever thread satisfies the promise, or
> in the same thread if the promise is already complete. This
> does mean that whenever you have blocking computation, you
> need to take care to execute these in your own threadpool --
> but again, we think of Futures as the coordination mechanism,
> and if there is blocking work, we represent the result of this
> as another future which can be combined in whatever ways are
> approriate.
>

Some background on the Akka 1.2 behavior here:

https://www.assembla.com/spaces/akka/tickets/1054-complete-futures-asynchronously-when-replying-through-a-channel
http://blog.ometer.com/2011/07/24/callbacks-synchronous-and-asynchronous/

I'm not familiar with the new Akka 2.0 Future.blocking stuff, I
haven't had a chance to look at it.

As you can probably tell from my blog post above, to me the important
thing is that it has to be defined and documented whether the
callbacks are run same-stack or different-stack. Either specification
can work well but leaving it undefined in my experience creates a
dangerous (= bug-prone, hard to use and test properly) API.

Havoc

Derek Williams

unread,
Feb 12, 2012, 7:03:34 PM2/12/12
to akka...@googlegroups.com
Very interesting. I haven't yet dug very deep into Twitter's implementation, but it does look like both implementations have quite similar semantics. We must all be on the right track. I added a few comments below:

On Sun, Feb 12, 2012 at 3:11 PM, marius a. eriksen <marius....@gmail.com> wrote:
Big disclaimer: I have not worked with Akka's futures. I only know
them from their documentation and API docs as well as from SIP-14.
Also: we've been working with Phillip, Heather, and co. to see if we
can consolidate these APIs somehow. As Viktor said, it'd be nice if
there were just one API, and maybe even just one implementation.

Any comments I add here are based on current Akka, I haven't been following too closely any changes proposed for the SIP.

> Both have onSuccess and onFailure; only Akka has onComplete.

onComplete sounds like Twitter's "ensure"

I think it's identical to Twitter's 'respond', and in the scaladocs it sounds like it is the main building block of the other callback methods, just like onComplete is the base of all it's callback methods. 
 
> Finagle has a Timer thread on which to wait if a timeout value was
> specified; not sure if this is a dedicated thread or a multiplexed
> thread. If dedicated, doesn't that inflate the thread count horribly?
> I looked at Akka's source and it seems that futures are interrupted
> periodically in order to ask "Have I timed out yet?" ... keeps the
> thread count down, and probably costs less than a thread context
> switch.

You pass in a timer to Future.within -- this is typically a shared
timer. For example in Finagle we have only one timer that's shared
across all timeouts. Like akka, we use Netty's HashedWheelTimer.

That sounds just like the pattern Akka uses to create a Future from an Actor request (using '?'). I don't think we have a helper method for users to make this pattern easier, like with Future.within, but it really should have it. (I've cut myself off from adding new stuff in Akka until I get the time to finish the other things I've started, but this is one of the first things I want to do).
 
CALLBACK INVOCATION ORDERING

As far as I can tell there is no way to provide callback ordering
guarantees with this SIP. That makes it quite difficult to implement
"try-finally" patterns. We do this quite often in our code:

       val resource = acquire()
       resource.operation() onSuccess { value =>
         resource.anotherOperation()
       } ensure {
         resource.release()
       }

Unless I'm missing something, this is done in Akka like this:

val resource = acquire()
resource.operation() map { value => // or flatMap
  resource.anotherOperation()
} onComplete { _ =>
  resource.release()
}

The main difference is that Akka requires the new Future to contain the result of the callback, where I think Twitter's just returns a new Future that points at the old value.

(a side note: I might not be looking at Twitter's latest code... I can't seem to find Future.ensure?)

CANCELLATION

Akka does allow cancellation, but it is more basic then what Twitter has. In Akka the original Promise would just be completed with some cancellation exception, just like it handles timeouts. I need to dig deeper into this if I want to understand Twitter's implementation of this, as I think it is the one big different between Akka and Twitter.

FUTURE MERGING AND TAIL RECURSION

I don't think I can comment on any of this yet since I only had a quick look, and I can't decide yet if this is something that Akka's Future could benefit from or not. I think it may be due to Twitter's chaining of Futures so ensure ordering of callback's, where as Akka only ensures the order of callbacks by using methods like map/flatMap (a for-comprehension being a good example of how to ensure order).

Also, it looks like Twitter's cancellation support also means that there are references kept for each Future created in the chain, where Akka has no need to hold onto any references... so it may have to do with this as well.

Akka's Future currently only contains 2 variables: the executor (for any callbacks that need to run, can possibly be moved to the callback methods themselves as an implicit, but stored in the Future to keep the api simpler) and the current state (this variable could be a list of callbacks, or an Either[Throwable, T]), so there isn't a whole lot of state that could be shared between multiple Futures, the executor is already shared, and final result, if the same object, is already shared.

But I could be completely off here. I'll need to look into it more, but my first instinct is that this is something Twitter's Future needs due to the way it implements callback ordering and/or cancellation.

To conclude:

From my brief look at Twitter's Future, it does look like the only major difference between the two implementation is the added framework for cancellations. I honestly thought I'd find bigger differences, but other then the syntax (and all the 'behind the curtain' magic) they look very similar.

--
Derek Williams

Havoc Pennington

unread,
Feb 12, 2012, 7:13:14 PM2/12/12
to akka...@googlegroups.com
I was just reading some Akka docs and I think they aren't very crisp
about the meaning of "blocking" and "async", and I think that might
also confuse some discussions of Future.

For example at one point the docs say "synchronously (blocking) or
asynchronously (non-blocking)" but I would distinguish the two.

Here are two possible definitions:

* asynchronous: runs without the current stack frame. Because it's
either on another thread's stack, or in this same thread but "later"
(after the current stack frame has been popped), or with the current
stack frame saved as a continuation and then resumed.

* nonblocking: does not tie up a thread while failing to use CPU. (on
the OS level, does not wait on a descriptor/handle/lock or equivalent;
does not suspend the thread waiting for the OS kernel to re-awaken it)

Another example of confusing this stuff, there was some recent
anti-node.js rant blog post going around that equated "spending time
using CPU" with "blocking" and that distinction in particular is
pretty important in my opinion. If you are blocking on IO/locks, then
tying up a thread is a massive waste of memory resources. If you are
spending time using CPU, then tying up a thread (ideally one thread
out of a pool matching number of CPU cores) is exactly what you want
to do; having an unbounded number of threads using CPU is bad. So the
analysis of node.js was muddled as a result of conflating these
distinct cases.

Anyhow, for explaining something like Future.blocking, I would think
the semantics in these kind of terms have to be made crystal clear; do
I use Future.blocking for long-running CPU-bound computation? For
blocking IO only? etc.

Havoc

Michael Slinn

unread,
Feb 12, 2012, 8:40:34 PM2/12/12
to akka...@googlegroups.com
Wow, excellent discussion! I'm so glad I reached out to Twitter. Let's keep this going.

Mike

Marius Eriksen

unread,
Feb 12, 2012, 11:31:20 PM2/12/12
to akka...@googlegroups.com


On Sunday, February 12, 2012 4:03:34 PM UTC-8, Derek Williams wrote:
Very interesting. I haven't yet dug very deep into Twitter's implementation, but it does look like both implementations have quite similar semantics. We must all be on the right track. I added a few comments below:

On Sun, Feb 12, 2012 at 3:11 PM, marius a. eriksen <marius....@gmail.com> wrote:
Big disclaimer: I have not worked with Akka's futures. I only know
them from their documentation and API docs as well as from SIP-14.
Also: we've been working with Phillip, Heather, and co. to see if we
can consolidate these APIs somehow. As Viktor said, it'd be nice if
there were just one API, and maybe even just one implementation.

Any comments I add here are based on current Akka, I haven't been following too closely any changes proposed for the SIP.

> Both have onSuccess and onFailure; only Akka has onComplete.

onComplete sounds like Twitter's "ensure"

I think it's identical to Twitter's 'respond', and in the scaladocs it sounds like it is the main building block of the other callback methods, just like onComplete is the base of all it's callback methods. 

yep, you're right. i misread the docs.
Yep-- but the API is sort of unnatural, and most programmer's wouldn't be aware of the distinction. I'd argue that:

  operation() onSuccess { result =>
    doSomethingTo(result)
  } ensure {
    resource.release()
  }

is much more natural and intuitive.

The main difference is that Akka requires the new Future to contain the result of the callback, where I think Twitter's just returns a new Future that points at the old value.

(a side note: I might not be looking at Twitter's latest code... I can't seem to find Future.ensure?)

That's probably because Future extends Try, and ensure is defined there:

 

CANCELLATION

Akka does allow cancellation, but it is more basic then what Twitter has. In Akka the original Promise would just be completed with some cancellation exception, just like it handles timeouts. I need to dig deeper into this if I want to understand Twitter's implementation of this, as I think it is the one big different between Akka and Twitter.

And a very important distinction: the fact that there's always a single updater forgoes a large class of potential bugs.
 

FUTURE MERGING AND TAIL RECURSION

I don't think I can comment on any of this yet since I only had a quick look, and I can't decide yet if this is something that Akka's Future could benefit from or not. I think it may be due to Twitter's chaining of Futures so ensure ordering of callback's, where as Akka only ensures the order of callbacks by using methods like map/flatMap (a for-comprehension being a good example of how to ensure order).

Also, it looks like Twitter's cancellation support also means that there are references kept for each Future created in the chain, where Akka has no need to hold onto any references... so it may have to do with this as well.

Akka's Future currently only contains 2 variables: the executor (for any callbacks that need to run, can possibly be moved to the callback methods themselves as an implicit, but stored in the Future to keep the api simpler) and the current state (this variable could be a list of callbacks, or an Either[Throwable, T]), so there isn't a whole lot of state that could be shared between multiple Futures, the executor is already shared, and final result, if the same object, is already shared.

But I could be completely off here. I'll need to look into it more, but my first instinct is that this is something Twitter's Future needs due to the way it implements callback ordering and/or cancellation.

This space leak (analogous to stack overflow in TCO) happens even without cancellation. Consider the example given:

    def compute(soFar: List[Result]): Future[List[Result]] = rpc() flatMap {
      case More(item) => compute(item :: soFar)
      case Done => soFar.reverse
    }

    compute(Nil)

each iteration produces a new Future; expanding the each rpc():

  rpc()
  a flatMap rpc()
  a flatMap b flatMap rpc()
  a flatMap b flatMap c flatMap rpc()
  ...

The "tail call optimization" in twitter's futures collapse all the intermediate futures into one:

Marius Eriksen

unread,
Feb 12, 2012, 11:54:17 PM2/12/12
to akka...@googlegroups.com
I agree with your sentiments here. However, specifying callback semantics make it difficult to treat Futures as a pure coordination mechanism, since doing so would tie your implementation to an event loop, or an executor, or what have you. Therefore Twitter futures don't specify callback semantics, which in turn means that you should expect that your callback may be invoked directly, or it may be deferred. In practice, we use a combined approach: the first callback on a particular thread takes over the scheduler and further callbacks are deferred (unwound to the scheduler frame).


It simplifies the implementation and also the semantics somewhat (it doesn't matter where the future comes from-- they're always subject to the same execution semantics), and in practice it hasn't come up as an issue (and we do have *a lot* of code using this).

I think a few factors contribute to this:

  1. we encourage "declarative"/"data-flow" style, and the use of future combinators -- in particular this encourages designs that are pure transformations over independent operations, and so locking things like resource sharing is typically neatly encapsulated by whatever produces the futures you are operating over.
  2. locks in java/scala are by default reentrant  (which of course can be the source of even subtler bugs!)
  3. not specifying callback semantics means the programmer cannot rely on them, maybe that makes them more aware

I'm sure we have some bugs lingering, but it hasn't really come up.

Derek Williams

unread,
Feb 13, 2012, 12:23:06 AM2/13/12
to akka...@googlegroups.com
On Sun, Feb 12, 2012 at 9:54 PM, Marius Eriksen <marius....@gmail.com> wrote:
I agree with your sentiments here. However, specifying callback semantics make it difficult to treat Futures as a pure coordination mechanism, since doing so would tie your implementation to an event loop, or an executor, or what have you. Therefore Twitter futures don't specify callback semantics, which in turn means that you should expect that your callback may be invoked directly, or it may be deferred. In practice, we use a combined approach: the first callback on a particular thread takes over the scheduler and further callbacks are deferred (unwound to the scheduler frame).



We do pretty much the same thing in Akka as well, due to the problem you stated before with tail recursive algorithms:


--
Derek Williams

Marius Eriksen

unread,
Feb 13, 2012, 12:26:21 AM2/13/12
to akka...@googlegroups.com
That alone still leaves a space leak, however.

Derek Williams

unread,
Feb 13, 2012, 1:02:34 AM2/13/12
to akka...@googlegroups.com
On Sun, Feb 12, 2012 at 10:26 PM, Marius Eriksen <marius....@gmail.com> wrote:
That alone still leaves a space leak, however.

Ah, I think I get it now. Akka needs to hold a reference to each previous Future in order to have the final value returned. Definitely something to ponder.

Thanks!

--
Derek Williams

√iktor Ҡlang

unread,
Feb 13, 2012, 4:01:18 AM2/13/12
to akka...@googlegroups.com
I don't see how one can cut corners here without reducing the power of the model.
However, it could be rather easy to implement the Clojure equivalent of "transient" or the Scala "view" on Futures to coalesce such operations.
Please also note that this isn't a memory leak, since the memory will be reclaimed when the Future is not referenced anymore, and throughout the lifetime of Akka, I have had 0 issues reported regarding this.
 

Thanks!

--
Derek Williams

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

Havoc Pennington

unread,
Feb 13, 2012, 11:00:31 AM2/13/12
to akka...@googlegroups.com
Hi,

On Sun, Feb 12, 2012 at 11:54 PM, Marius Eriksen
<marius....@gmail.com> wrote:
> https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/concurrent/IVar.scala#L83

That's pretty interesting. It isn't unlike having a trivial event loop
for each thread, I guess - though each "event" just triggers
immediately and isn't waiting for anything to happen.

I may not understand it, but I think if Schedule just had a way to
queue a callback, without running, then a program could create
always-defer semantics like this (adding hypothetical API):

val schedule = Schedule.get
schedule.queue(myActualCodeThatDoesStuff) // first part of current apply()
schedule.run() // run() part of current apply()
// thread exits or does something else

That's the usual API on event loops. It seems like the existing
schedule API more or less just automatically does the run() when you
enqueue the first task, so the first task is technically not deferred
(it runs with a "queueAndRun" - calling it apply() - on the stack):

schedule.queueAndRun(myActualCodeThatDoesStuff)
// thread exits or does something else

I'm wondering if this is an "always defers" API for practical
purposes, imagining:

def myActualCodeThatDoesStuff() {
takeSomeNonReentrantLock()
addSomeCallbackToSchedule(foo) // this will always defer
outside myActualCodeThatDoesStuff
dropLock()
// callback foo is going to run outside the lock
}

What I don't understand in Schedule is when run() would be invoked
recursively (via flush()?). I guess you could un-defer some callbacks
by recursing run()? And there you'd have to be aware of what the
callbacks were and be sure you dropped locks those callbacks might
need.

Anyway if I am roughly understanding Schedule it seems like you're
defining what's async and what isn't pretty carefully and there are
predictable semantics "every callback is deferred except the outermost
one, unless you flush() and then you better know what you're doing" ?
That's pretty deterministic from a programmer perspective.

In my blog post I was bringing up the old Akka version of the "!"
method which just didn't define at all the sync/async semantics;
assuming an interface with one method !, implementations of that
interface could either defer or not. As a practical matter, using an
interface like that, you often have to add an _extra_ manual defer
just to be sure it gets deferred. So that's the kind of ambiguous
interface that can be trouble.

> doing so would tie your implementation to an event loop, or an executor

it seems OK to me to have some pluggable "trait Deferer { def
defer(work: => Unit) }"

I mean, you can always implement it as "def defer(work: => Unit) {
work }" and force non-deferral

You could also implement it as "enqueue in the Schedule" as with the
code you linked above, perhaps?

Or it can be kicked to a thread pool...

The big difference between Schedule and deferring via executor seems
like it would be "same vs. different thread" which affects whether you
can block on the deferred thing...

Akka 2's "ExecutionContext" is this "trait Deferer" thing

>   2. locks in java/scala are by default reentrant  (which of course can be
> the source of even subtler bugs!)

Where it came up for me in Akka is that actors are (as a core design
principle) not reentrant, of course.

As noted in my blog post it came up before Akka, when using
Hammersmith's netty connection and netty callbacks. Netty isn't
reentrant either presumably (I don't remember if the lock in question
was added by Hammersmith around the netty pipeline or was built-in to
netty).

In both of these cases, it was the "invoke application callback" that
needed to be deferred/async. The guideline could be reduced to: "don't
call out-of-your-own-module code with locks held." The ! method
processing messages synchronously was maybe an especially rocky case
of violating expectations, since that method is normally all about
being async.

>   3. not specifying callback semantics means the programmer cannot rely on
> them, maybe that makes them more aware

Tt isn't always clear what to do about it once you're aware - take the
case of Akka's pre-1.2 "!" method; you can fix the deadlock by
manually adding another layer of deferral, but then half the time you
have two layers of deferral. In this case as well, I doubt anyone
would be aware until they were bitten once. The "!" API is just flatly
more useful and less surprising if it has consistent semantics (= is
always async).

> I'm sure we have some bugs lingering, but it hasn't really come up.

My impression of that Schedule code is that you have pretty
well-specified behavior that almost always defers callbacks (unless
the callback is the outermost one, which is the least likely one to
hold locks or otherwise cause trouble)? I would not expect that to
cause a lot of bugs.

Code that bypassed the thread's Schedule and just ran a callback by
hand immediately, would be the more potentially troublesome case.

Sort of fun to think about Schedule as the world's tiniest event loop
implementation ;-)

Havoc

Marius Eriksen

unread,
Feb 13, 2012, 11:20:04 AM2/13/12
to akka...@googlegroups.com

On Monday, February 13, 2012 1:01:18 AM UTC-8, Viktor Klang wrote:

On Mon, Feb 13, 2012 at 7:02 AM, Derek Williams <de...@fyrie.net> wrote:
On Sun, Feb 12, 2012 at 10:26 PM, Marius Eriksen <marius....@gmail.com> wrote:
That alone still leaves a space leak, however.

Ah, I think I get it now. Akka needs to hold a reference to each previous Future in order to have the final value returned. Definitely something to ponder.

I don't see how one can cut corners here without reducing the power of the model.
However, it could be rather easy to implement the Clojure equivalent of "transient" or the Scala "view" on Futures to coalesce such operations.

This is different -- I'm talking only about merging *equivalent* futures in order to avoid space leaks. The implementation of flatMap has this outline:
  
  flatMap(f: T => Future[R]):
    result = new Promise[R]
    this.respond {
      case Return(res) => 
        val next = f(res)
        result.merge(next)
      case Throw(exc) =>
        result.setException(exc)
    }

the call to result.merge here results in the two futures being equivalent. `result' holds the state, and `next' merely links to it. This means that in a chain, all futures are merged to the root future, freeing up all intermediate futures.
 
Please also note that this isn't a memory leak, since the memory will be reclaimed when the Future is not referenced anymore, and throughout the lifetime of Akka, I have had 0 issues reported regarding this.

It is a classic space leak: you're holding onto intermediate values that will just be discarded. For large recursions, this is a real issue. This sort of pattern is used in streaming computations, for example, and is directly analogous to a "standard" FP-style recursive loop.

marius.

Havoc Pennington

unread,
Feb 13, 2012, 12:20:47 PM2/13/12
to akka...@googlegroups.com
The same-thread deferral from
https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/concurrent/IVar.scala#L83
just reminded me of a previous Akka discussion.

There's this issue that while Actor makes threads go away and
developers don't have to think about them, Akka Future makes threads
come back and developers have to think about them again.

var state
def receive = {
...
something onComplete {
// code that runs in another thread
foo(state) // bad!
}
}

AFAIK current plans include methods like pipeTo and completeWith to
avoid writing callbacks by hand; and to make the compiler get upset if
you close actor state into the onComplete callback.

I guess there was an idea before, that onComplete's callback could run
serialized on the actor. The Schedule idea from IVar.scala is the same
thing applied to threads, using a thread local.

So you could imagine a DeferSerializedContext provided as an implicit
to onComplete and friends, perhaps. It would take a runnable and run
it later, but serialized same-actor or same-thread.

Outside of actors, futures could use a thread local or just a
manually-created context; inside, they use the actor. In both cases
all callbacks run deferred-but-serialized.

Outcome: using a Future by itself never exposes concurrency to the app
developer. Future doesn't dispatch anything to another thread or
actor, keeps it all in the same one.

Closing actor state into onComplete would be just fine, and often convenient.

When you need a new thread, you'd manually send to an executor, or
you'd create a new Actor instead of using threads.

Probably too much breakage for Akka but I still think it's an interesting angle.

Havoc

Marius Eriksen

unread,
Feb 13, 2012, 12:25:26 PM2/13/12
to akka...@googlegroups.com
I had completely left out an important feature of Twitter's Futures: Locals.


These are like thread-locals, except their scope is the callback graph. In other words:

  val l = new Local[Int]
  l() = 123
  future respond { _ => assert(l() = Some(123) }
  l() = 333

We use this to implement RPC tracing, exception monitoring, and callback-tracing (which emulate stack traces for callbacks graphs to avoid debugging), amongst other things.

marius.

Marius Eriksen

unread,
Feb 13, 2012, 12:44:21 PM2/13/12
to akka...@googlegroups.com
On Monday, February 13, 2012 8:00:31 AM UTC-8, Havoc Pennington wrote:
Hi,

On Sun, Feb 12, 2012 at 11:54 PM, Marius Eriksen
<marius....@gmail.com> wrote:
> https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/concurrent/IVar.scala#L83

That's pretty interesting. It isn't unlike having a trivial event loop
for each thread, I guess - though each "event" just triggers
immediately and isn't waiting for anything to happen.

I may not understand it, but I think if Schedule just had a way to
queue a callback, without running, then a program could create
always-defer semantics like this (adding hypothetical API): 

val schedule = Schedule.get
schedule.queue(myActualCodeThatDoesStuff) // first part of current apply()
schedule.run()         // run() part of current apply()
// thread exits or does something else

The problem is that doing this necessitates a root: we don't assume anything about thread ownership and so the schedulers have to exit. On the other hand, you can always guarantee deferring by scheduling your computation via futures:

  Future.value(()) onSuccess { _ =>
    // inside of this block, deferring is guaranteed
  }

That's the usual API on event loops. It seems like the existing
schedule API more or less just automatically does the run() when you
enqueue the first task, so the first task is technically not deferred
(it runs with a "queueAndRun" - calling it apply() - on the stack):

schedule.queueAndRun(myActualCodeThatDoesStuff)
// thread exits or does something else

I'm wondering if this is an "always defers" API for practical
purposes, imagining:

 def myActualCodeThatDoesStuff() {
        takeSomeNonReentrantLock()
        addSomeCallbackToSchedule(foo) // this will always defer
outside myActualCodeThatDoesStuff
        dropLock()
        // callback foo is going to run outside the lock
 }

What I don't understand in Schedule is when run() would be invoked
recursively (via flush()?). I guess you could un-defer some callbacks
by recursing run()? And there you'd have to be aware of what the
callbacks were and be sure you dropped locks those callbacks might
need.


flush() is required in order to implement synchronous access of values:

  f respond { _ =>
    val x = f()  // (this is a blocking API)
  }

This is a common pattern when joining many futures:

  Future.join(f1, f2, f3, f4...) onSuccess { _ =>
    f1() + f2() + f3() ...
  }

flush is required here because we need to guarantee that if a future would be satisfied by a callback that was deferred, it must be available at the time we call the blocking API.
 

Anyway if I am roughly understanding Schedule it seems like you're
defining what's async and what isn't pretty carefully and there are
predictable semantics "every callback is deferred except the outermost
one, unless you flush() and then you better know what you're doing" ?
That's pretty deterministic from a programmer perspective.


Yes, in practice (especially when using something like finagle), it means that all callbacks are deferred. The only time it *isn't* is when you're registering callbacks in your own thread -- in some sense this is fair, since you're controlling the execution environment at that point, and if you want to, you could use the above trick.

In my blog post I was bringing up the old Akka version of the "!"
method which just didn't define at all the sync/async semantics;
assuming an interface with one method !, implementations of that
interface could either defer or not. As a practical matter, using an
interface like that, you often have to add an _extra_ manual defer
just to be sure it gets deferred. So that's the kind of ambiguous
interface that can be trouble.

> doing so would tie your implementation to an event loop, or an executor

it seems OK to me to have some pluggable "trait Deferer { def
defer(work: => Unit) }"

I mean, you can always implement it as "def defer(work: => Unit) {
work }" and force non-deferral

You could also implement it as "enqueue in the Schedule" as with the
code you linked above, perhaps?

Or it can be kicked to a thread pool...

The big difference between Schedule and deferring via executor seems
like it would be "same vs. different thread" which affects whether you
can block on the deferred thing...

That's the gap bridged by flush()
yes-- you are right; in practice, callbacks are always deferred, because they're almost always run in either finagle's IO workers or in a FuturePool.
 

Code that bypassed the thread's Schedule and just ran a callback by
hand immediately, would be the more potentially troublesome case.

Sort of fun to think about Schedule as the world's tiniest event loop
implementation ;-)


that pretty much describes it exactly!

marius.
 

Jacobus Reyneke

unread,
Mar 12, 2012, 4:47:17 AM3/12/12
to akka...@googlegroups.com
Hi,

I'll represent a normal user, as I'm no expert in either.

I'm using Finagle and Finagle uses Twitters Futures.

As a normal person, getting to know the in's and out's of Scala, this could become quite confusing. Using Twitter Futures (and learning about them) and learning about Akka Futures will quickly cause one to burn a couple of brain cells.

Is it sensible to create a quick comparison in on both Akka and Twitter Future docs to highlight the main differences? Even if it's just a head's up for the next unsuspecting traveler to pass this way. Maybe it makes sense to give two distinct libraries different names to avoid confusion - maybe not.

Marius mentioned that it may be sensible to migrate to a single code base. This sounds sensible.

Cheers,
Jacobus

√iktor Ҡlang

unread,
Mar 12, 2012, 5:14:00 AM3/12/12
to akka...@googlegroups.com

Hi Jacobus,

Look up Scala Improvement Proposal 14, Akka 2.0 mimicks that API. Will be in Scala Std Lib in Future.

Cheers,
V

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/4wHnIr5CrCgJ.

Jacobus Reyneke

unread,
Mar 12, 2012, 9:39:00 AM3/12/12
to akka...@googlegroups.com
Thanks V

I'm happy now :-)

Have a good one,
Jacobus

On Mon, Mar 12, 2012 at 11:14 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:
> Scala Improvement Proposal 14

Reply all
Reply to author
Forward
0 new messages