Cleaning up Akka Actors after a Spray Http Timeout

[V]

Hi,

I have a query about the best approach to cleaning up Akka Actors when a Spray Http Timeout is received.

Let's say that in your Spray HttpService you have set your request-timeout to be 3 seconds.

And you create a new DealWithRequestActor for each new Http Request in the your HttpService, passing it the RequestContext in the Actor's constructor.

If DealWithRequestActor takes 10 seconds to process before it calls ctx.complete(...), Spray will have already sent the Timeout message, and it will close the ConnectionActor.

The problems then are:

• The DealWithRequestActor eventually finishes, and tries to call ctx.complete, but the response is completed.
  
• The DealWithRequestActor will linger in the ActorSystem forever.

Are there any good suggestions for stopping the DealWithRequestActor when the there is a request timeout?

Thanks,

V

[V]

One idea I've had for the second bullet point (The DealWithRequestActor will linger in the ActorSystem forever.) is to use setReceiveTimeout on the ActorContext for DealWithRequestActor. Where the receiveTimeout is greater than the Spray Request-Timeout value, so that at some point in the future I can have the DealWithRequestActor handle the ReceiveTimeout message and stop itself.

[Age Mooij]

This is exactly the pattern I use for cleaning up request-level temporary actors. For extra bonus points make sure to set the receive timeout to a value calculated directly from the spray request timeout (I usually go for 3x).

Age

On Apr 5, 2013, at 15:08, V <c0deatt...@gmail.com> wrote:

One idea I've had for the second bullet point (The DealWithRequestActor will linger in the ActorSystem forever.) is to use setReceiveTimeout on the ActorContext for DealWithRequestActor. Where the receiveTimeout is greater than the Spray Request-Timeout value, so that at some point in the future I can have the DealWithRequestActor handle the ReceiveTimeout message and stop itself.

--
You received this message because you are subscribed to the Google Groups "spray-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spray-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[David Pratt]

Forgive my lack of Akka low-level knowledge. What happens when an actor times out, and how is that calculated? Let's say, for example, the temporary actor is starved in the ExecutionContext of it's actor system for some reason (too many requests, some other problem, etc) - does the timeout clock start from the second the message is queued up in the mailbox, or from when the actor begins to process the message?

I've been thinking about this problem a lot lately - we have some services that have the potential to cause an unbounded amount of work to show up. For example, we have (unfortunately) a part of our system that has to query a database that for various reasons doesn't hold up terribly well under load - the query times get larger and larger depending on the amount of concurrency. We've attempted to limit this by doing a few things, but what I'd ideally like is the ability to just kill a request handler entirely if a timeout happens.

One of the few nice things about the servlet/threaded model is that the entire request is managed by a single thread. You can monitor the processing of the request, and if it takes too long, you just nuke the thread. I'm trying to wrap my head around how to achieve something similar in the akka/async world.

[Mathias]

David,

> does the timeout clock start from the second the message is queued up in the mailbox, or
> from when the actor begins to process the message?

The timer starts when the actor has finished the processing of a message.
It is cancelled right before it begins processing of the next message.
The time the message is queued to the mailbox doesn't matter.

This is the relevant part of the Akka code for it:
https://github.com/akka/akka/blob/master/akka-actor/src/main/scala/akka/actor/ActorCell.scala#L454

> One of the few nice things about the servlet/threaded model is that the
> entire request is managed by a single thread. You can monitor the
> processing of the request, and if it takes too long, you just nuke the
> thread. I'm trying to wrap my head around how to achieve something similar
> in the akka/async world.

With akka this is not hard either.
Just set a timeout with `setReceiveTimeout` and react to `ReceiveTimeout` messages.
What scenario are you worried about where this simple approach would not work?

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

>> One idea I've had for the second bullet point (*The DealWithRequestActor
>> will linger in the ActorSystem forever.*) is to use* setReceiveTimeout *on
>> the ActorContext for *DealWithRequestActor*. Where the receiveTimeout is

>> greater than the Spray Request-Timeout value, so that at some point in the

>> future I can have the *DealWithRequestActor* handle the ReceiveTimeout

[Mathias]

> Are there any good suggestions for stopping the DealWithRequestActor when
> the there is a request timeout

There are several strategies.
The simplest one would be to stop the request actor after it has completed the request.
(if your logic makes sure that this will always be the case, even if the completion happens too late).

Or you handle `Timedout` messages in your main service actor (or dedicated timeout handler) and then stop/kill the respective request actor.
This would allow you to stop an ongoing request handling process right after the timeout has happened and not waste any more resources on computing a response that will arrive to late anyway…

Cheers,
Mathias

---
mat...@spray.io
http://spray.io

On 05.04.2013, at 14:09, V <c0deatt...@gmail.com> wrote:

> Hi,
>
> I have a query about the best approach to cleaning up Akka Actors when a
> Spray Http Timeout is received.
>
> Let's say that in your Spray HttpService you have set your request-timeout
> to be 3 seconds.
> And you create a new DealWithRequestActor for each new Http Request in the
> your HttpService, passing it the RequestContext in the Actor's constructor.
> If DealWithRequestActor takes 10 seconds to process before it calls
> ctx.complete(...), Spray will have already sent the Timeout message, and it
> will close the ConnectionActor.
>
> The problems then are:
>

> - The DealWithRequestActor eventually finishes, and tries to call

> ctx.complete, but the response is completed.

> - The DealWithRequestActor will linger in the ActorSystem forever.

>
> Are there any good suggestions for stopping the DealWithRequestActor when
> the there is a request timeout?
>
> Thanks,
> V
>
>
>

[V]

Thanks Mathias,

I have settled on handling a Akka Timeout to stop the Actor and it's children. The Actor always stops itself either on receiving a Akka Timeout or when the natural "task finished" Message is received.

I was worried that trying to complete a Http Request twice would cause a major problem, but it doesn't appear to.

Regards,

V

[Dave Mills]

How do I map the Timedout message received by my "main service actor" to the respective "request actor"? 

Is it just a matter of maintaining a Map[HttpRequest,ActorRef] ? 

[Johannes Rudolph]

On Wed, Oct 30, 2013 at 6:37 PM, Dave Mills <dave.a...@gmail.com> wrote:
> Is it just a matter of maintaining a Map[HttpRequest,ActorRef] ?

Yes, currently it only works this way. You could also use the `sender`
ActorRef of the HttpRequest as the map key. Maybe we should think
about an improvement in the protocol as well but that doesn't seem to
be easily achievable without introducing race-conditions. I created a
ticket to discuss it:

https://github.com/spray/spray/issues/656

--
Johannes

-----------------------------------------------
Johannes Rudolph
http://virtual-void.net