Example of sinning by accessing shared mutable state from a future

[Michael Slinn]

The docs have a warning, but do not show a concrete example of the programming sin or why it is a sin. Here is my attempt at explaining the issue. I am posting it so I can receive corrections and look at various examples of the Right Way (TM).

Futures must not access shared mutable state in an outer scope.

Here is a Scala example of what not to do:

var offset = 6

val future = Future { 

  2+3+offset // will be executed asynchronously 

}

// computations that results in changes to offset...

val result = Await.result(future, 1 second)

Here is the same programming sin, expressed in Java:

int offset = 6;

Futures.future<Integer>(new Callable<Integer>() {

  public Integer call() { 

    return 2+3+offset; // will be executed asynchronously

  }

}, executionContext);

// computations that results in changes to offset...

Integer result = 

  (Integer)Await.result(future, timeout.duration());

In both code examples, the Future is computed in an inner scope. The value of offset when the Future is first computed is passed to the Future, but when the Future later evauates, the value of offset will not be updated to the current value. 

Scala has closures, and Java wishes it did, but the error, and the correction is similar in spirit. 

Is this good so far? What are the alternatives?

Mike

[Raoul Duke]

On Tue, Jan 24, 2012 at 11:19 AM, Michael Slinn <msl...@gmail.com> wrote:
> Futures must not access shared mutable state in an outer scope.

and might there somehow miraculously be a way, at least in Scala, to
make it statically enforced?

[Michael Slinn]

The docs say 'not yet' and do not suggest that this will be possible any time soon.

Any feedback on the code examples and the comments?

Mike

[√iktor Ҡlang]

On Tue, Jan 24, 2012 at 8:19 PM, Michael Slinn <msl...@gmail.com> wrote:

The docs have a warning, but do not show a concrete example of the programming sin or why it is a sin.

Mike, I believe I have caught you with a lie:

http://akka.io/docs/akka/2.0-M2/general/jmm.html

Cheers,

√

 

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/kRtoHfgxJhAJ.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--
Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Mike Slinn]

... and well you might have, Viktor. Would you please be more specific?

Mike

[Michael Slinn]

Here is a complete code example that shows exactly what I mean:

import akka.dispatch.Future

import java.util.concurrent.Executors

import akka.dispatch.ExecutionContext

object ScopeSin extends App {

  val executorService = Executors.newFixedThreadPool(10)

  implicit val context = ExecutionContext.fromExecutor(executorService)

  var offset = 6

  

  Future { 2 + 3 + offset } onComplete { f =>

    println("Entered onComplete, offset = " + offset)

    f match {

      case Right(result)   => println("ScopeSin Scala result: " + result)

      case Left(exception) => println("ScopeSin Scala exception: " + exception)

    }

    executorService.shutdown(); // terminates program

  }

  offset = 42

  println("End of mainline, offset = " + offset)

}

Output is:

End of mainline, offset = 42

Entered onComplete, offset = 42

ScopeSin Scala result: 11

As you can see, the value of offset is changed to 42 throughout, except where it had been captured when it had the value 11.

Viktor, would you please say explicitly what meant?

Mike

[√iktor Ҡlang]

Did you see the bottom of the link I posted? :-)

On Jan 24, 2012 9:18 PM, "Mike Slinn" <msl...@gmail.com> wrote:

... and well you might have, Viktor. Would you please be more specific?

Mike

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

[Michael Slinn]

Viktor,

I am unclear as to what the 'lie' you referred to might be. Were you just trying to say that some documentation now exists? I think that the technical information that I provided is correct.

BTW, using an accessor gives the same result as a var, which is not surprising considering how def is implemented:

import akka.dispatch.Future
import java.util.concurrent.Executors
import akka.dispatch.ExecutionContext

object ScopeSin extends App {
  val executorService = Executors.newFixedThreadPool(10)
  implicit val context = ExecutionContext.fromExecutor(executorService)
  var offset = 6

  def accessor = offset

 
  Future { 2 + 3 + offset } onComplete { f =>
    println("Entered onComplete, offset = " + offset)
    f match {
      case Right(result)   => println("ScopeSin Scala result: " + result)
      case Left(exception) => println("ScopeSin Scala exception: " + exception)
    }
    executorService.shutdown(); // terminates program
  }

  Future { 2 + 3 + accessor } onComplete { f =>

    println("Entered onComplete, offset = " + offset)
    f match {
      case Right(result)   => println("ScopeSin Scala result: " + result)
      case Left(exception) => println("ScopeSin Scala exception: " + exception)
    }
    executorService.shutdown(); // terminates program
  }
  offset = 42
  println("End of mainline, offset = " + offset)
}

Result is:

End of mainline, offset = 42
Entered onComplete, offset = 42
ScopeSin Scala result: 11
Entered onComplete, offset = 42
ScopeSin Scala result: 11

The equivalent Java gives similar results.

This is a pitfall that many people are likely to fall into, and it will difficult for them to figure it out by examining variable values.

Mike

[√iktor Ҡlang]

You said:

The docs have a warning, but do not show a concrete example of the programming sin or why it is a sin.

I said:

Mike, I believe I have caught you with a lie:

http://akka.io/docs/akka/2.0-M2/general/jmm.html

See the bottom of the page above, it includes examples of what to do and what not to do, and why.

Or did I miss something?

Cheers,

√

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/i_s2HnMbhy8J.

To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[rkuhn]

Hi Michael,

I guess he meant that the page he linked to does show concrete examples of how “sins” might look like. Your example is instructive for certain points, most notably that it is not well-defined which value the Future will compute (depending on thread scheduling); this is commonly referred to as a race condition. You could make it reproducible by inserting sleep statements and showing the two possible outcomes.

What is much more difficult to demonstrate, though, are memory consistency issues which arise when unsafely passing objects between threads: since you are using an Int there cannot be any partial writes (i.e. internal inconsistency), but if you were to use an object which contains non-final fields, the code running in the Future context could see that object in an inconsistent state (i.e. partially updated) if written to from another thread.

More accessible for an example is this:

class A(val worker: ActorRef) extends Actor {
  var processed = 0
  def receive = {
    case Work =>
      worker ? Work onSuccess { result => sender ! result; processed += 1 }
  }
}

The classical error is to capture “sender” in a closure, which we battle by introducing the thread-safe “pipeTo” pattern, but this code in addition has the (obvious, once you know it) flaw that multiple increments of “processed” might run concurrently, which without synchronization leads to losses due to “last write wins”. So, a possible correct solution:

class A(val worker: ActorRef) extends Actor {
  var processed = 0
  def receive = {
    case Work =>
      worker ? Work onSuccess { _ => self ! OneDown } pipeTo sender
    case OneDown =>
      processed += 1
  }
}

Regards,

Roland

[Michael Slinn]

Roland,

Very helpful information, thank you.

1. Your example involved Actors, and a magical pipeTo solution. The biggest problem that I have encountered with regards to Actor scope issues is obtaining intermediate results from the top level actor back into the main program. Simulations and interactive games both exhibit this problem. For example, here is a little program that breaks the rules. How should it be reconstructed?
2. Similar issues exist for Futures, but guidelines are scarce. Perhaps a pipeTo analog could be utilized? I found this definition of pipeTo in the source code:

implicit def future2actor[T](f: akka.dispatch.Future[T]) = new {
    def pipeTo(actor: ActorRef): this.type = {
      f onComplete {
        case Right(r) ⇒ actor ! r
        case Left(f)  ⇒ actor ! Status.Failure(f)
      }
      this
    }
  }

I wonder how it might be repurposed for Future.

3. Similar issues exist for closures in general. The problem exists for Java and Scala, and does not arise because of some property peculiar to Akka. Are you aware of some discussion on this subject?

Thank you,

Mike

[Patrik Nordwall]

Note that you can interact with shared mutable state or external resources outside the actors/futures if the code is thread safe; falls back to things like synchronized, volatile, locks, java.util.concurrent stuff. I didn't look close at you game but it is using Swing and somewhere you must do the bridge between the Actors and Swing and that code must obey ordinary concurrency rules.

Mike

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/MhlgCA-qqqkJ.

To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Patrik Nordwall

Typesafe -  The software stack for applications that scale

Twitter: @patriknw

[Michael Slinn]

Any reason STM would not be a safe way to share mutable state?

Mike

[√iktor Ҡlang]

See Transactors. STM is only local though.

V

On Jan 25, 2012 1:35 PM, "Michael Slinn" <msl...@gmail.com> wrote:

Any reason STM would not be a safe way to share mutable state?

Mike

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/0u46RYK9StIJ.

[Michael Slinn]

1) In Jonas's 2009 JavaOne presentation he said: "My (humble) opinion on STM

>Can never work fine in a language that don’t have compiler enforced immutability

>E.g. never in Java (as of today)

>Should not be used to “patch” Shared-State Concurrency

>Still a research topic how to do it in imperative languages"

His examples addressed Clojure's STM implementation. In general, is the current Akka STM implementation suitable for accessing mutable state from Akka futures written in Scala and Java? 

2) The example code that I posted at the top of this thread shows a programming sin. In this limited context, is there any reason not to use STM?

Mike

[√iktor Ҡlang]

On Wed, Jan 25, 2012 at 3:12 PM, Michael Slinn <msl...@gmail.com> wrote:

1) In Jonas's 2009 JavaOne presentation he said: "My (humble) opinion on STM

>Can never work fine in a language that don’t have compiler enforced immutability

>E.g. never in Java (as of today)

>Should not be used to “patch” Shared-State Concurrency

>Still a research topic how to do it in imperative languages"

His examples addressed Clojure's STM implementation. In general, is the current Akka STM implementation suitable

define "suitable"

 

for accessing mutable state from Akka futures written in Scala and Java? 

2) The example code that I posted at the top of this thread shows a programming sin. In this limited context, is there any reason not to use STM?

in your example you should use "val offset" or "final int offset". Nothing in the original example needs the offset to be mutable.

You only need STM if you need atomic updates of multiple memory cells, i.e. ACI.

Cheers,

√

 

Mike

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/YhYm6rBiItsJ.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Mike Slinn]

The point of the sinning example was that the state was a var, initially
set to 6, then changed to 42. The resulting race condition caused an
undesirable result. I am exploring various means of addressing the
problem. STM is one such candidate.

Mike

[√iktor Ҡlang]

On Wed, Jan 25, 2012 at 3:25 PM, Mike Slinn <msl...@gmail.com> wrote:

The point of the sinning example was that the state was a var, initially set to 6, then changed to 42. The resulting race condition caused an undesirable result. I am exploring various means of addressing the problem. STM is one such candidate.

Question is what would be the desired behavior.

 

Mike

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To post to this group, send email to akka...@googlegroups.com.

To unsubscribe from this group, send email to akka-user+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Jonas Bonér]

On Wed, Jan 25, 2012 at 3:12 PM, Michael Slinn <msl...@gmail.com> wrote:
> 1) In Jonas's 2009 JavaOne presentation he said: "My (humble) opinion on STM
>>Can never work fine in a language that don’t have compiler enforced
>> immutability
>>E.g. never in Java (as of today)
>>Should not be used to “patch” Shared-State Concurrency
>>Still a research topic how to do it in imperative languages"

This is still my base "religious" view of things. If you don't have
enforced immutability, and even better; an effect system, then it will
always be "correctness by convention". Which is less than ideal. But
my pragmatic view is that an STM in a language without this is still
very useful since it makes it much much easier to do the right thing
than walking in to the threads & locks war zone.

>
> His examples addressed Clojure's STM implementation. In general, is the
> current Akka STM implementation suitable for accessing mutable state from
> Akka futures written in Scala and Java?
>
> 2) The example code that I posted at the top of this thread shows a
> programming sin. In this limited context, is there any reason not to use
> STM?
>
> Mike
>

> --
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/akka-user/-/YhYm6rBiItsJ.

>
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to
> akka-user+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/akka-user?hl=en.

--
Jonas Bonér
CTO

Typesafe - The software stack for applications that scale

Phone: +46 733 777 123
Twitter: @jboner

[Derek Williams]

Just an idea that popped in my head that I haven't thought through too much:

What about some kind of ThreadLocal-like class that has it's current state captured whenever a Future is created in the same thread... I haven't thought through the details yet, but it could be something to try out as long as it doesn't add much overhead. It wont help in the case of wanting to modify some value in the Actor, but it would make Mike's example work as he'd like.

It would also allow something like:

val sender: AkkaThreadLocal[ActorRef] = ...

which would automatically capture the senders current value.

It's probably crazy, but when I have a free moment I might give it a try.

--

Derek Williams

[√iktor Ҡlang]

Seems like conflation of Futures and Actors.

What problem do you mean it solves?

Cheers,

√

 

--

Derek Williams

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[Derek Williams]

That InheritedThreadLocals don't work with a thread pool. I don't think it's too important, It might bring uneeded overhead and complexity.

2012/1/25 √iktor Ҡlang <viktor...@gmail.com>

--

Derek Williams

[Scott Clasen]

Take a look in twitter-util at com.twitter.util.SavedLocal, is this kind of what you are talking about Derek?

https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Local.scala

[Derek Williams]

On Wed, Jan 25, 2012 at 12:15 PM, Scott Clasen <scott....@gmail.com> wrote:

Take a look in twitter-util at com.twitter.util.SavedLocal, is this kind of what you are talking about Derek?

https://github.com/twitter/util/blob/master/util-core/src/main/scala/com/twitter/util/Local.scala

Yep, thats what I'm thinking of, just with automatic save when a some Future related closure is created, and automatic restore when the closure is executed.

Doesn't mean I endorse the idea though, just throwing it out there.

--

Derek Williams

[Michael Slinn]

I thought that the ExecutorService returned by Executors.newFixedThreadPool() would cause Futures created from the resulting ExecutionContext to run on their own threads. Looks like I have found a circumstance where this is not true. In the following program, uncommenting all of the shutdown() statements causes an exception.

import java.util.concurrent.Executors

import scala.concurrent.stm.Ref

import akka.dispatch.ExecutionContext

import akka.dispatch.Future

object ScopeSin extends App {

  val executorService = Executors.newFixedThreadPool(10)

  implicit val context = ExecutionContext.fromExecutor(executorService)

  var offset = 6

  def accessor = offset

  val offsetRef = Ref(offset)

  

  Future { 

    2 + 3 + offset // will be executed asynchronously

  } onComplete { f =>

    println("Entered onComplete for variable, offset = " + offset)

    f match {

      case Right(result)   => println("ScopeSin Scala result: " + result)

      case Left(exception) => println("ScopeSin Scala exception: " + exception)

    }

    //executorService.shutdown(); // terminates this thread, and the program if no other threads are active

  }

  Future { 2 + 3 + accessor } onComplete { f =>

    println("Entered onComplete for accessor, accessor = " + accessor)

    f match {

      case Right(result)   => println("ScopeSin Scala result: " + result)

      case Left(exception) => println("ScopeSin Scala exception: " + exception)

    }

    //executorService.shutdown(); // terminates this thread, and the program if no other threads are active

  }

  Future { 2 + 3 + offsetRef.single() } onComplete { f =>

    println("Entered onComplete for STM Ref, offsetRef value = " + offsetRef.single())

    f match {

      case Right(result)   => println("ScopeSin Scala result: " + result)

      case Left(exception) => println("ScopeSin Scala exception: " + exception)

    }

    //executorService.shutdown(); // terminates this thread, and the program if no other threads are active

  }  

  offsetRef.single() = 42

  offset = 42

  println("End of mainline, offsetRef value = " + offsetRef.single())

  println("End of mainline, offset = " + offset)

}

The exception is:

Exception in thread "pool-1-thread-5" java.util.concurrent.RejectedExecutionException

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source)

at akka.dispatch.ExecutionContext$WrappedExecutor.execute(AbstractDispatcher.scala:106)

at akka.dispatch.Future$.dispatchTask(Future.scala:313)

at akka.dispatch.DefaultPromise.tryComplete(Future.scala:766)

at akka.dispatch.Promise$class.complete(Future.scala:620)

at akka.dispatch.DefaultPromise.complete(Future.scala:701)

at akka.dispatch.Future$$anon$2.run(Future.scala:147)

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.lang.Thread.run(Unknown Source)

Mike

[Derek Williams]

If the ExecutorService is shutdown before all the tasks have been submitted you will get this. Calling onComplete causes a new task to be submitted once the Future is complete, so it's possible for this to happen after you have shut it down.

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/qQMf834agwQJ.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Derek Williams

[Michael Slinn]

So what is required is some way for each onComplete to call  executorService.shutdown() after they have all started, or even better, after they have all finished; this pattern would facilitate the addition of more futures easily. Is a countdown latch the best approach?

The assumption is that once the Future is completed and its value is retrieved, it won't be needed again, and I want to minimize the number of idle threads.

Mike

[Michael Slinn]

Hmm, a countdown latch cannot be incremented, so another mechanism is needed.

[Derek Williams]

On Wed, Jan 25, 2012 at 5:01 PM, Michael Slinn <msl...@gmail.com> wrote:

So what is required is some way for each onComplete to call  executorService.shutdown() after they have all started, or even better, after they have all finished; this pattern would facilitate the addition of more futures easily. Is a countdown latch the best approach?

I wouldn't call it the best, but it's perhaps the most optimized (in many cases prematurely).

I consider the best approach is to avoid purely side-effecting methods (like foreach and onComplete), and use Futures.sequence to collect the final futures together. Block on that last future and you are done. Kinda like the IO monad in Haskell.

--

Derek Williams

[Michael Slinn]

You are right in a small program that fires off many futures and then stops, but a large program with collections of futures that serve different purposes would require something more substantial.

Perhaps a trait that can be mixed in with Future that maintains a count singleton?

[√iktor Ҡlang]

Hi Michael,

You shouldn't close a resource that was passed to you.


Cheers,

√

On Thu, Jan 26, 2012 at 1:21 AM, Michael Slinn <msl...@gmail.com> wrote:

You are right in a small program that fires off many futures and then stops, but a large program with collections of futures that serve different purposes would require something more substantial.

Perhaps a trait that can be mixed in with Future that maintains a count singleton?

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/Yf8uEGaxJqcJ.

To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--

Viktor Klang

Akka Tech Lead

Typesafe - The software stack for applications that scale

Twitter: @viktorklang

[rkuhn]

Am Donnerstag, 26. Januar 2012 01:21:38 UTC+1 schrieb Michael Slinn:

You are right in a small program that fires off many futures and then stops, but a large program with collections of futures that serve different purposes would require something more substantial.

Perhaps a trait that can be mixed in with Future that maintains a count singleton?

Traits being mixed in would have to reimplement all methods which produce new Futures (which are all transformations) in order to retain that mixin; I think this is impractical enough. On the other hand, you definitely don’t want anything with “single” in the name of a concept for scalable concurrent programs anyway, right?

Futures are a means to get some computation done, and they need a context in which to do that (providing access to some CPUs). This context needs to be managed by the code which wants to use it, which means that it needs to be shut down when (if ever) the computation is finished and the resources should be relinquished, and this cannot be determined automagically.

Regards,

Roland

[Roland Kuhn]

Hi Michael,

On 24 jan 2012, at 22:49, Michael Slinn <msl...@gmail.com> wrote:

Roland,

Very helpful information, thank you.

1. Your example involved Actors, and a magical pipeTo solution. The biggest problem that I have encountered with regards to Actor scope issues is obtaining intermediate results from the top level actor back into the main program. Simulations and interactive games both exhibit this problem. For example, here is a little program that breaks the rules. How should it be reconstructed?

If you have a computation which yields a single result, use "ask" and Await.result(), otherwise have an Actor receive the replies to you initial query and feed it into a BlockingQueue like the TestKit's testActor. There is a ticket for creating an AskStream pattern which would be an optimized version of the second approach. To be clear: if you must send messages to a destination outside the actor world, there will be blocking and/or polling involved, there is no way around this. 

2. Similar issues exist for Futures, but guidelines are scarce. Perhaps a pipeTo analog could be utilized? I found this definition of pipeTo in the source code:

implicit def future2actor[T](f: akka.dispatch.Future[T]) = new {
    def pipeTo(actor: ActorRef): this.type = {
      f onComplete {
        case Right(r) ⇒ actor ! r
        case Left(f)  ⇒ actor ! Status.Failure(f)
      }
      this
    }
  }

I wonder how it might be repurposed for Future.

This I don't understand: pipeTo sends the Future's contents to an actor once availabe. If you want to receive it in the current thread instead, use Await.result. 

3. Similar issues exist for closures in general. The problem exists for Java and Scala, and does not arise because of some property peculiar to Akka. Are you aware of some discussion on this subject?

No, and if I were to venture a guess: a complete compiler-verified solution needs an effect system, which is all but excluded for Java and nearly impossible for Scala (please note that this is my personal belief and I could be wrong; miracles are always welcome!). 

Regards,

Roland

Thank you,

Mike

--
You received this message because you are subscribed to the Google Groups "Akka User List" group.

To view this discussion on the web visit https://groups.google.com/d/msg/akka-user/-/MhlgCA-qqqkJ.