mailbox size
G J
I'd like to know if my Actor is being swamped by messages, and is not
keeping up. I have a regular/periodic timer, and the interval at which
it gets processed is much larger than the timer period.
Thanks.
G J
One used to be able to say self.getMailboxSize, or
self.dispatcher.mailboxSize. Both are no longer supported.
Thanks.
√iktor Ҡlang
--
You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.
Viktor Klang
Akka Tech Lead
Paulicka
Do those methods become available with a bounded mailbox?
Also, what happens to new messages with a bounded mailbox when full?
> that scale
>
> Twitter: @viktorklang
√iktor Ҡlang
How does this solve the problem?
Do those methods become available with a bounded mailbox?
Also, what happens to new messages with a bounded mailbox when full?
Paulicka
The fact that the Sender gets notified if it can't process the message
seems to be a flaw in the design.
Shouldn't the Supervisor get the message? (Perhaps as well as the
Sender...)
After all, won't it be the Supervisor whose job it is to handle making
a Router or other way of distributing the load off of that Actor?
Ideally, it would be nice to know RDDTool-style how many messages have
been in the queue for the last chunk of time, so you can decide
whether your Actor is keeping up.
Ultimately, this means people are using the Actor mailbox as an Event
queue, but you should be able to introspect that queue.
Just bounding the mailbox and notifying the Sender really doesn't
enable any type of dynamic solution, unless the User of your service
knows how to ask for more help...from somewhere?
On Feb 15, 1:44 pm, √iktor Ҡlang <viktor.kl...@gmail.com> wrote:
> Hi Christopher,
>
> On Wed, Feb 15, 2012 at 10:30 PM, Paulicka
G J
Thanks for asking the question, and echoing my thoughts exactly about
the bounded mailboxes solution.
In fine tuning a real time system, I want to know if my Actor is
keeping up. If the mailbox is mostly empty then I know that it is. In
bursty situations how large does it get? etc.
Why did getMailboxSize (or equivalent) disappear?
I should do my homework, but since this is a related question, I'll
ask anyway: Is there some way to give priority to messages in a
mailbox?
If I have a periodic timer that triggers every 5 sec, then I want to
be notified every 5 secs (more or less exactly), and not have this
event be put at the end of a big mailbox and processed whenever it is
processed. Thus, the ReceiveTimeout message should be given high
priority.
Much appreciated.
Thanks.
Erik Nelson
G J
Thoughts?
Peter Vlugter
On 16/02/2012, at 2:55 PM, Erik Nelson wrote:
> For what it's worth, I recently implemented production monitoring which uses mailbox sizes, and it's been very useful to us. If that was not available in 2.0 would be a significant reason for us to not update, because we need to know more than just "is the mailbox full or not". Knowing that the actors are not able to keep up with what we're sending them is key to preventing problems, instead of just reacting to them.
We monitor mailbox sizes without using getMailboxSize. Timestamped events are generated for message send and receive (start of message processing) and many other things. Given a stream of events like this it's easy enough to calculate a mailbox size at a particular time. The send and receive events for any particular message are also connected with a unique trace id so it's also possible to say how much time messages spend in the mailbox, how long from sending to completing processing of a message, how long messages take to process on average, and so on.
Roland Kuhn
• it takes O(n) time to get some size answer from a concurrent queue
• the answer is not correct, i.e. it does not need to match the real size at neither the beginning nor the end of processing this request
• making it “more correct” involves book-keeping which severely limits scalability
• and even then the number might have changed completely by the time you receive it in your code (e.g. your thread is scheduled out for 100ms and you get 10^5 new messages during that time).
So, no mailboxSize in the default implementation. You can write you own mailbox (that's really trivial, look at https://github.com/jboner/akka/blob/master/akka-actor/src/main/scala/akka/dispatch/Mailbox.scala) and do exactly the book-keeping you like. RRDTool sounds interesting here, but be aware that this incurs a cost and ask youself if it is justified.
Switching angle: given a way for an actor to obtain its mailbox size, what would it do in case it measures that it does not keep up? Slowing down the senders—by way of a bounded queue—is basically the only thing it can do itself. Other than that only the supervisor can do something, but your application is likely not prepared to handle that, since all the supervisor can do is terminate the poor guy, invalidating all the references the senders have. There is no generic solution other than designing your application to be scalable at the required points by adding resizable routers from the beginning. Without knowing the points of contention, this is not possible.
I should probably turn this into a blog post for future reference.
Regards,
Roland Kuhn
Typesafe — The software stack for applications that scale
twitter: @rolandkuhn
√iktor Ҡlang
> It has been said before, but this seems to require re-iteration at certain intervals: mailboxSize was removed for a number of good reasons in the context of a general purpose library, you are free to add it back as an akka.actor.Extension in your specific use scenario. The reasons are:
>
> • it takes O(n) time to get some size answer from a concurrent queue
> • the answer is not correct, i.e. it does not need to match the real size at neither the beginning nor the end of processing this request
> • making it “more correct” involves book-keeping which severely limits scalability
> • and even then the number might have changed completely by the time you receive it in your code (e.g. your thread is scheduled out for 100ms and you get 10^5 new messages during that time).
>
> So, no mailboxSize in the default implementation. You can write you own mailbox (that's really trivial, look at https://github.com/jboner/akka/blob/master/akka-actor/src/main/scala/akka/dispatch/Mailbox.scala) and do exactly the book-keeping you like. RRDTool sounds interesting here, but be aware that this incurs a cost and ask youself if it is justified.
Also, to expand on Rolands argument, you have to take into
consideration, system messages that are sent by Akka to actors to
handle fault-tolerance, starting, stopping etc.
To reiterate: Actors is not about hogging a thread on one message,
it's about reactive programming.
If you want to do long running one-off tasks, that's where Futures are
really good.
Also inspecting mailboxSize from within the actor itself turns a
reactive model into an active model, which is a different paradigm.
And then we have the scenario where if you'd poll let's say a, durable
mailbox, where the actual polling of the mailbox size, could take
several hundred milliseconds, or even seconds in the case of high
load.
>
> Switching angle: given a way for an actor to obtain its mailbox size, what would it do in case it measures that it does not keep up? Slowing down the senders—by way of a bounded queue—is basically the only thing it can do itself. Other than that only the supervisor can do something, but your application is likely not prepared to handle that, since all the supervisor can do is terminate the poor guy, invalidating all the references the senders have. There is no generic solution other than designing your application to be scalable at the required points by adding resizable routers from the beginning. Without knowing the points of contention, this is not possible.
Or you create your own mailbox implementation that when the mailbox
has more than X messages, it sends the messages to some other actor
instead? (seems weird but you could do that)
>
> I should probably turn this into a blog post for future reference.
Excellent, that'd save a lot of time.
Cheers,
√
--
Viktor Klang
Akka Tech Lead
Typesafe - The software stack for applications that scale
Twitter: @viktorklang
Jonas Bonér
> On Thu, Feb 16, 2012 at 8:34 AM, Roland Kuhn <goo...@rkuhn.info> wrote:
>> It has been said before, but this seems to require re-iteration at certain intervals: mailboxSize was removed for a number of good reasons in the context of a general purpose library, you are free to add it back as an akka.actor.Extension in your specific use scenario. The reasons are:
>>
>> • it takes O(n) time to get some size answer from a concurrent queue
>> • the answer is not correct, i.e. it does not need to match the real size at neither the beginning nor the end of processing this request
>> • making it “more correct” involves book-keeping which severely limits scalability
>> • and even then the number might have changed completely by the time you receive it in your code (e.g. your thread is scheduled out for 100ms and you get 10^5 new messages during that time).
>>
>> So, no mailboxSize in the default implementation. You can write you own mailbox (that's really trivial, look at https://github.com/jboner/akka/blob/master/akka-actor/src/main/scala/akka/dispatch/Mailbox.scala) and do exactly the book-keeping you like. RRDTool sounds interesting here, but be aware that this incurs a cost and ask youself if it is justified.
>
> Also, to expand on Rolands argument, you have to take into
> consideration, system messages that are sent by Akka to actors to
> handle fault-tolerance, starting, stopping etc.
>
> To reiterate: Actors is not about hogging a thread on one message,
> it's about reactive programming.
> If you want to do long running one-off tasks, that's where Futures are
> really good.
Or Agent.sendOff
--
Jonas Bonér
CTO
Typesafe - The software stack for applications that scale
Phone: +46 733 777 123
Twitter: @jboner
Roland Kuhn
Erik Nelson
Happily, custom mailboxes also address the question I brought up in another thread. Hurray for custom mailboxes! Now I will patiently await the full release of 2.0.
erik
√iktor Ҡlang
Not that you need my approval for anything, but the ability in 2.0 to create one's own mailbox implementation sufficiently addresses the issue for me… apologies for bringing up a well trod and addressed topic.
Happily, custom mailboxes also address the question I brought up in another thread. Hurray for custom mailboxes!
Now I will patiently await the full release of 2.0.