Re: [akka-user] Avoiding long mailbox queue - how to?

[Alex Cruise]

Here's what I do for an actor who gets about half a million messages in thirty seconds.  It might be totally wrong, but it doesn't OOME. :)

application.conf:

akka {

  actor {

    foo-dispatcher {

      mailbox-capacity = 1000

      mailbox-push-timeout-time = 5s

    }

  }

}

Initialization: 

val fooActor = context.actorOf(Props(createActor()).withDispatcher("akka.actor.foo-dispatcher"))

HTH!

-0xe1a

On Sat, Mar 30, 2013 at 1:42 PM, Andrew Gaydenko <andrew....@gmail.com> wrote:

I have external source of events by implementing callback method of external Java API. The method sends a message to an actor with RoundRobinRouter. Events rate is rather high resulting in very long message queues (and eating plenty of RAM). In fact 2/3 of overall test duration Akka is busy to drain queues (when all events are already happened).

How to block in sending thread to avoid long queues?

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[√iktor Ҡlang]

Hakkers,

I'd recommend having push timeout at around 0. It is extremely dangerous otherwise as:

1) self ! message  == if mailbox is full, we block for X time (as configured) then we discard the message (if unlimited this is effectively a deadlock)

2) someRemoteActor ! message == if mailbox is full, the remoting's thread will be blocked for X time (as configured) preventing messages to be received and dispatched in the mean time.

So in the end:

A) Use a pull model for flow control

or

B) discard overflow if you're using a push model

Does that help?

Cheers,

√

On Tue, Apr 2, 2013 at 9:57 AM, Andrew Gaydenko <andrew....@gmail.com> wrote:

Alex, fine, thanks! It seems to be exactly my case (probably I'd have shorter queue). At the moment that situation was resolved with using another external API (and using intermediate dispatching actor with self-spamming), but the case is rather common to meet it again. Now I see I was too lazy to read all the doc :)

      # If negative (or zero) then an unbounded mailbox is used (default)
      # If positive then a bounded mailbox is used and the capacity is set using
      # the property
      # NOTE: setting a mailbox to 'blocking' can be a bit dangerous, could lead
      # to deadlock, use with care
      # The following mailbox-push-timeout-time is only used for type=Dispatcher
      # and only if mailbox-capacity > 0
      mailbox-capacity = -1
 
      # Specifies the timeout to add a new message to a mailbox that is full -
      # negative number means infinite timeout. It is only used for type=Dispatcher
      # and only if mailbox-capacity > 0
      mailbox-push-timeout-time = 10s

It would be interesting to hear from akka developers more elaborative explanation of "can be dangerous". Concrere "dos" and "donts" would be appreciated.

--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ: http://akka.io/faq/
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Viktor Klang
Director of Engineering

Typesafe

Twitter: @viktorklang

[√iktor Ҡlang]

On Apr 2, 2013 1:43 PM, "Andrew Gaydenko" <andrew....@gmail.com> wrote:
>
> Viktor,
>
> On Tuesday, April 2, 2013 3:10:50 PM UTC+4, √ wrote:
>>
>> Hakkers,
>
>
> Am to  young to be called with this proud word "hAkker" :) Nevertheless...

>  
>>
>>
>> I'd recommend having push timeout at around 0. It is extremely dangerous otherwise as:
>>
>> 1) self ! message  == if mailbox is full, we block for X time (as configured) then we discard the message (if unlimited this is effectively a deadlock)
>
>

> Of course, it would be awful, and it isn't the usecase described in the starting message,

I was merely attempting to explain what's bad about it.

>  
>>
>> 2) someRemoteActor ! message == if mailbox is full, the remoting's thread will be blocked for X time (as configured) preventing messages to be received and dispatched in the mean time.
>
>

> ...as well as this one - messages are sent from an external (with regard to actor system) code inside the same JVM. At that use case a blocking is the aim.

Yes, I just don't like to build in assumptions at the "wrong" side.

>  
>>
>>
>> So in the end:
>>
>> A) Use a pull model for flow control
>
>

> yes, when it is possible.

>  
>>
>> or
>> B) discard overflow if you're using a push model
>
>

> Sorry, I know translation of all the words, but can not catch a sense. What does "discard overflow" mean?

Drop messages when overloaded.

Cheers!

V

>  
>>
>> Does that help?
>
>
> Any conversation with a man having sharp brain and friendly manner is helpful! :)

[Evan Chan]

I know this is not recommended, but we have a system in production that uses push timeout at -1 with a mailbox capacity at say 5000 for flow control.  There are no remote actors, and each actor just passes messages to the next actor.    We have never had a deadlock, it is brilliantly simple and processes thousands of messages a second nonstop.

On Tue, Apr 2, 2013 at 3:10 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

Hakkers,

I'd recommend having push timeout at around 0. It is extremely dangerous otherwise as:

1) self ! message  == if mailbox is full, we block for X time (as configured) then we discard the message (if unlimited this is effectively a deadlock)

If an actor is the only one that keeps sending messages to itself, and it only sends ONE message for every message that it processes, in theory it can never deadlock, because the mailbox will never grow faster than it is being consumed.

--
--
Evan Chan
Senior Software Engineer | 
e...@ooyala.com | (650) 996-4600
www.ooyala.com | blog | @ooyala

[√iktor Ҡlang]

On Tue, Apr 2, 2013 at 6:12 PM, Evan Chan <e...@ooyala.com> wrote:

I know this is not recommended, but we have a system in production that uses push timeout at -1 with a mailbox capacity at say 5000 for flow control.  There are no remote actors, and each actor just passes messages to the next actor.    We have never had a deadlock, it is brilliantly simple and processes thousands of messages a second nonstop.

Hey, I use "Unsafe" on a daily basis almost, some tools are just "on your own risk" :-)

 

On Tue, Apr 2, 2013 at 3:10 AM, √iktor Ҡlang <viktor...@gmail.com> wrote:

Hakkers,

I'd recommend having push timeout at around 0. It is extremely dangerous otherwise as:

1) self ! message  == if mailbox is full, we block for X time (as configured) then we discard the message (if unlimited this is effectively a deadlock)

If an actor is the only one that keeps sending messages to itself, and it only sends ONE message for every message that it processes, in theory it can never deadlock, because the mailbox will never grow faster than it is being consumed.

True, but that is a very specific situation :) And in that case a bounded mailbox doesn't make sense at all :)