Future and Spray

[Maatary Okouya]

Everywhere we can read that that when performing long running operation or blocking operation it is preferable to use a special Execution context for the matter. Blocking operation like accessing the database. I understand why. It is to avoid Thread starvation. We do not want to have the "8" available threads busy with some blocking code which may eventually return or keep blocking. It either seriously slow down the application or block it indefinitely.

Meanwhile, I'm wondering how things framework Spray (or Play) are implemented. Indeed, let's take the client side. When a request is sent we get a future response. In other words, the request is executed asynchronously. This may by the way end up being a long running operation. However, there is nothing that says launching many request could lead to thread starvation in theses case. Hence i'm wondering why in that case it is not a problem. Do they have special Thread Pool.

I red in the Book "Learning concurrent programing in Scala" that using the "Blocking {}" statement block in a Future helps its scheduler to spawn more thread automatically. Could it be the way it is handled in SPRAY

The same thing could be said for receiving request, in play we get to execute an Async Action. If one wants to access a database from this action, shall one use the "Blocking {}" statement block. How to execute that action is a special threadPool/ExecutionContext.

My assumption here is that they rely on the implicit.global ExecutionContext. Maybe i'm wrong. The Bottom line is. Making request is a long operation by default, how using spray for instance in your code, would handle it such that not to create a Thread Starvation in your code ?

Is Spray using different ExecutionContext internally?

[Maatary Okouya]

Just spotted This short presentation Don't Block - How to Mess Up Akka and Spray that happens to better illustrate the problem that I came with here.

In any case i would appreciate to have other opinion

[Richard Bowker]

it's important to consider whether the long running operation is using a synchronous or asynchronous api.

with an asynchronous api, you can make a request and the thread is yielded immediately, at a later point in time the api completes the future using another thread from the pool. At no point is a thread held waiting on IO. This is the ideal case and you don't need to do anything, it makes good use of the default fork-join executor.

However if you are calling a synchronous api (i.e. it blocks the thread while fetching the result) then as you noted you have to be careful. You can wrap the request in a blocking {} but this can be dangerous as you can end up spawning more and more threads and eventually run out of memory. A more controlled approach is to create a dedicated executor using a fixed sized thread pool, and allow the futures that are being used to call the synchronous api to use that instead. this means you protect the executor that akka is using, and also bound the number of threads that can be used accessing the synchronous operation.

[Maatary Okouya]

Richard, I think i'm getting there. This would mean that somehow, the presentation that i posted ignore certain things. 

The answer is indeed here

>>>>

it's important to consider whether the long running operation is using a synchronous or asynchronous api.

with an asynchronous api, you can make a request and the thread is yielded immediately, at a later point in time the api completes the future using another thread from the pool. At no point is a thread held waiting on IO. This is the ideal case and you don't need to do anything, it makes good use of the default fork-join executor.

>>>>

I kinda like to go to the bottom of things. What can i read more to understand that, Java NIO and what ? Is there something deep in the spray or AKKA IO That's explain that. 

Because, so far with me learning concurrent programing in Scala to me, I learned that when one does a request (or any synchronous call) the following is happening:

def apply[T](body: =>T)(implicit executor: ExecutionContext): scala.concurrent.Future[T] =
{
  val runnable = new PromiseCompletingRunnable(body)
  executor.prepare.execute(runnable)
  runnable.promise.future
}

...

class PromiseCompletingRunnable[T](body: => T) extends Runnable {
val promise = new Promise.DefaultPromise[T]()

override def run() = {
  promise complete {
    try Success(body) catch { case NonFatal(e) => Failure(e) }
  }
} }

In other words there is a Thread somewhere, that dequeue that task in a Thread pool and try to complete set a promise future value with the result of the execution of that task. If that is correct, i don't understand how having a thread waiting for that network operation, is not a blocking thing ?

[Maatary Okouya]

Said differently

This to me means: there is a Thread in a ThreadPool, that dequeue that task and try to set a promise future value with the result of the execution of that task. If that is correct, i don't see how that task making an IO call does not block the run of that Thread. 

[Mathias Doenitz]

Maatary,

spray (and akka) don’t use blocking IO calls for network access.
We only access the non-blocking JDK NIO APIs underneath.

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

> --
> You received this message because you are subscribed to the Google Groups "spray.io User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to spray-user+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/spray-user.
> To view this discussion on the web visit https://groups.google.com/d/msgid/spray-user/d553bad8-b3ba-44b1-bdb9-161825d35069%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Richard Bowker]

as Mathias said, all the in built spray stuff uses NIO, which does not require consuming a thread while waiting for a result, so all of that works nicely

problems occur when you call in your own code a synchronous API (the most common culprit is JDBC to an SQL database)  in this case the thread is blocked while waiting for a result and it ruins the akka threading model. to work around this you need to provide your own execution context for these operations, but even better try to find a non blocking version of the API

[Johannes Rudolph]

Maatary,

your observations are correct. To augment what Mathias and Richard
said: The question is always what to do while waiting for external
system to deliver data. You need to:

1.) save your current state (Questions: How big is the state? Do you
keep resources not needed for continuing?)
2.) register a notification to be sent to you to resume processing
(Questions: How to register? How to wait? Where to run the
continuation?)

Every solution needs to deal with these two requirements and the
resulting questions.

Using a thread to do a "blocking call" is one solution that usually is
very expensive because the thread will likely keep lots of references
to resources and unrelated calls down in the call stack may as well.
Also, the question is in which way the thread actually does the
"blocking call" internally. Does it poll? Or even spin? Or does it
wait on an OS lock? Usually, you cannot even tell exactly what
mechanisms are used internally and how expensive they are.

Actors, on the other side, save all their state in their own instance
and have a well-known notification/wakeup protocol: waking up happens
when someone "sends a message" to this actor, i.e. a message is
enqueued to its mailbox and the actor is scheduled on its dispatcher
to run its message processing logic.

For Futures, you can register a notification function in the future to
be scheduled on the ExecutionContext once the promise is completed.
So, the state is saved as the closure you register with `onComplete`
and notification will happen by scheduling the function to run on the
implicitly passed ExecutionContext.

Java NIO, or actually most event based models work with polling (=
blocking) but allow multiplexing. This means that you have only one
single, dedicated thread which polls for new events to arrive (and
which actually *does block*). If an event arrives, the Selector thread
resumes and it can then spawn new tasks for every event that needs to
be handled. In Java NIO, it is the Selector API which allows
multiplexing. Akka IO uses the Selector in its SelectionHandler [1]
with a dedicated thread pool of size 1 on which the Selector runs and
waits for notifications. So, yes, even with Java NIO you need to block
but the single, blocking thread is amortized across all the network
events you wait for.

Regarding your questions:

Thread starvation happens if your complete (bounded) thread-pool is
waiting for blocking calls to finish. Using `blocking {}` will not
really solve the problem but will instead just create more threads
which will sooner or later lead to another resource bottleneck
(because threads usually need lots of memory etc).

Does this affect spray/akka.io? On the one hand, no, not directly,
because it uses the aforementioned mechanism where only a single
blocking thread is needed for the whole process. On the other hand,
however, if you look at point 1.) above, even with spray you need to
keep continuation state somewhere. And each kind of memory is finite.
The point of "a non-blocking approach" is that it will use much less
resources than using blocking threads.

(As a side note, this is also a general reason to use stateless
protocols instead of stateful: if a server needs to keep lots of state
between requests, it can handle usually far less peers concurrently
than it does with a stateless protocol where it needs to keep only
data to handle the immediate, concurrent requests.)

We often speak of "an asynchronous API" as if it was absolutely clear
what is meant. What is meant is usually an API that has effective
solutions for the above points 1.) and 2.). Particularly it usually
means that the API doesn't block a "full thread" while waiting for
external data but instead, it may very well use the the single,
amortized, blocking IO thread as outlined above.

(On the hardware level, the question could again be another one: use a
single polling thread or register an interrupt for the event you are
interested in...)

This was long but I hope it still makes sense :)

Johannes

[1] https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/io/SelectionHandler.scala

> --
> You received this message because you are subscribed to the Google Groups
> "spray.io User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to spray-user+...@googlegroups.com.
> Visit this group at http://groups.google.com/group/spray-user.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/spray-user/d553bad8-b3ba-44b1-bdb9-161825d35069%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net

[Maatary Okouya]

Super thanks, for taking the time to give me theses details. I'll read a bit more the Java NIO, AKKA IO and the link that you send, to get an accurate understanding. 

So far things make sense. The one point that is still not clear is what is happening to the thread that run the the request. 

Indeed when i say: 

This to me means: there is a Thread in a ThreadPool, that dequeue that task and try to set a promise future value with the result of the execution of that task. If that is correct, i don't see how that task making an IO call does not block the run of that Thread. 

If it indeed happens in the code as I mentioned in the preceding messages, then we have an issue with "Promise success (FutureTask)". If i understood well this is not asynchronous. Success will wait until FutureTask returns. 

I'm not sure, i have to read and probably get into the code, but if i understood you well, that thread block yes, but then downline it does not create another thread, it will communicate with only one thread at the selector/channel level. 

--
You received this message because you are subscribed to a topic in the Google Groups "spray.io User List" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/spray-user/avYhEFYsvcg/unsubscribe.
To unsubscribe from this group and all its topics, send an email to spray-user+...@googlegroups.com.

Visit this group at http://groups.google.com/group/spray-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/spray-user/CAKF7HneG%3Dinckw4ZbRdHNSYUzFxmt%2BbrszAkquFv1Lt1zF62zw%40mail.gmail.com.

[Maatary Okouya]

The other solution that i see is that, the actual thread that execute the  Promise success (FutureTask) is actually that unique Thread you are talking about. 

[Maatary Okouya]

A wild guess here is that i might make a confusion between a future task and a future value. Nobody says that you create a future task to send the message. However you return a future value. Hence it all depend on how you actually that that value. 

Could you give me some clue about that ?

Best, daniel