Random thoughts on Akka actor implementation ..

[Debasish Ghosh]

Hi -

Quite some time back myself and Jonas had discussed the possibilities of having a durable mailbox for actors. This set up the context for the Issue #97 in the current tracker. We revived the discussion a couple of days back and had a few interesting mail exchanges. This is a summary of the current thoughts. Would appreciate some brainstorming over the ideas.

Currently Akka supports the let-it-crash idiom over actor restarts through supervision. But it cannot support message persistence of the actor mailboxes. The mailbox is currently implemented as a ConcurrentLinkedQueue, which is transient and falls flat over an actor's death. Can we make this durable ? Redis offers queue implementation APIs through its List data structure. In fact we have an implementation of a Transactional Queue in Akka today, which is based on Redis.

Another stream of thought was load balancing for busy actors with multiple consumers. Once the messages are pushed to the mailbox, from then onwards its FIFO. Can we improve that ? We had been discussing if we can have multiple queues for actors with multiple consumers. But we need to keep in mind about message ordering too. This leads to an idea of implementing actor pools for busy ones. Of course we need to make this configurable, so that the user can set up actor pools for specific actors declaratively.

Ideas ? Thoughts ?

Thanks.

--
Debasish Ghosh
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

[Martin Krasser]

Debasish Ghosh schrieb:

> Hi -
>
> Quite some time back myself and Jonas had discussed the possibilities
> of having a durable mailbox for actors. This set up the context for
> the Issue #97 in the current tracker. We revived the discussion a
> couple of days back and had a few interesting mail exchanges. This is
> a summary of the current thoughts. Would appreciate some brainstorming
> over the ideas.
>
> Currently Akka supports the let-it-crash idiom over actor restarts
> through supervision. But it cannot support message persistence of the
> actor mailboxes. The mailbox is currently implemented as a
> ConcurrentLinkedQueue, which is transient and falls flat over an
> actor's death. Can we make this durable ? Redis offers queue
> implementation APIs through its List data structure. In fact we have
> an implementation of a Transactional Queue in Akka today, which is
> based on Redis.

Thanks for starting this thread. I consider this as absolutely necessary
for reliable messaging in actor-based systems. Having a Redis-based
transactional queue as mailbox should be IMHO only one option for
persisting messages. Another option could be a MOM such as ActiveMQ, for
example. For reading from and writing to the mailbox the same interface
is used with different, configurable implementations (e.g. transient,
persistent-redis, persistent-jms etc.). Another aspect is to let actors
participate in transactions that are managed by the storage backend. For
example, when using ActiveMQ (or any other JMS impl), we could let
consuming actors participate in transactions started by the MOM. When
the actor dies (i.e. the transaction is not committed) or the
transaction is rolled back for some other reason, the message can be
redelivered by ActiveMQ based on a configurable redelivery policy. Using
an ActiveMQ-based approach also offers some options for load-balancing
e.g. to let actor concurrently consume messages from the same queue.
ActiveMQ also provides options for configuring distributed
store-and-forward networks for distributing load across different nodes.
These are just thoughts, I didn't implement a protoype so far (e.g.
using ActiveMQ), so there's no proof-of-concept ATM.

>
> Another stream of thought was load balancing for busy actors with
> multiple consumers. Once the messages are pushed to the mailbox, from
> then onwards its FIFO. Can we improve that ? We had been discussing if
> we can have multiple queues for actors with multiple consumers. But we
> need to keep in mind about message ordering too. This leads to an idea

> of implementing *actor pools* for busy ones. Of course we need to make

> this configurable, so that the user can set up actor pools for
> specific actors declaratively.
>
> Ideas ? Thoughts ?
>
> Thanks.
>
> --
> Debasish Ghosh
> http://manning.com/ghosh
>
> Twttr: @debasishg
> Blog: http://debasishg.blogspot.com
> Code: http://github.com/debasishg

> --
> You received this message because you are subscribed to the Google
> Groups "Akka User List" group.
> To post to this group, send email to akka...@googlegroups.com.
> To unsubscribe from this group, send email to
> akka-user+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/akka-user?hl=en.

--
Martin Krasser

Blog: http://krasserm.blogspot.com
Twitter: http://twitter.com/mrt1nz

[rossputin]

Hi.

I am totally for the durable mailbox, I believe in 'let-it-crash' but
then again I believe in 'let-it-be-resurrected' :-) For me, if my
actors fail repeatedly, I need to ensure that many thousands of
messages are not just lost, but that I can fix the issue, and continue
processing. Akka's light weight integration with Redis does seem to
fit this scenario perfectly.

Load balancing busy actors would be great too, currently we have some
load balancing machines, 'apparently' routing based on how busy the
processor is on our actor machines. This is in no way as accurate as
determining which actor has the larger message queue.

Would an actor pool not just be another way of describing the cluster
functionality? Where we could switch on extra nodes automatically to
deal with heavy load at peak times. So we would basically use the
cluster as the pool ?

Exciting things ahead for Akka.

Cheers,

-- Ross

On Mar 24, 4:49 am, Debasish Ghosh <ghosh.debas...@gmail.com> wrote:
> Hi -
>
> Quite some time back myself and Jonas had discussed the possibilities of
> having a durable mailbox for actors. This set up the context for the Issue
> #97 in the current tracker. We revived the discussion a couple of days back
> and had a few interesting mail exchanges. This is a summary of the current
> thoughts. Would appreciate some brainstorming over the ideas.
>
> Currently Akka supports the let-it-crash idiom over actor restarts through
> supervision. But it cannot support message persistence of the actor
> mailboxes. The mailbox is currently implemented as a ConcurrentLinkedQueue,
> which is transient and falls flat over an actor's death. Can we make this
> durable ? Redis offers queue implementation APIs through its List data
> structure. In fact we have an implementation of a Transactional Queue in
> Akka today, which is based on Redis.
>
> Another stream of thought was load balancing for busy actors with multiple
> consumers. Once the messages are pushed to the mailbox, from
> then onwards its FIFO. Can we improve that ? We had been discussing if we
> can have multiple queues for actors with multiple consumers. But we need to
> keep in mind about message ordering too. This leads to an idea of

> implementing *actor pools* for busy ones. Of course we need to make this

> configurable, so that the user can set up actor pools for specific actors
> declaratively.
>
> Ideas ? Thoughts ?
>
> Thanks.
>
> --

> Debasish Ghoshhttp://manning.com/ghosh

[Krishna Sankar]

1. Interesting idea. Durable queues have been a feature of the MQ systems for a long time
2. We do need a datastore for the durability – assuming durability means the queue lasts a process crash and also the machine crash
3. would be nice if could abstract the datastore interface, most probably a get/put/delete key-values would be enough. That way one can implement the durable MBs using Redis or Cassandra or even AMZ SimpleDB or SQS
4. And if we pick Cassandra or something similar, we might also get distribution and replication capability. There is the question of eventual consisteny, which we would need to turn off.
5. The ordering is a tricky business – adds the state dimension. Better to avoid it. If there is only one consumer and one actor, then the order is implicit. If there are more than one consumer, the order is pseudo implicit. With multiple consumers and actors, the order is indeterminate
6. I assume we are not making any assumptions about the processing time for each message – for example a message might take a long time while others might be processed very quickly. So better to make no assumptions – and so no guarantees on the ordering.
7. Configurable actor pools is a good idea.
8. A good  feature set from the message queuing side is the AMZ SQS.
9. Another question is the metadata – do we need to track any metadata ?
10. Would be happy to help
    

Cheers
<k/>

[Peter Veentjer]

Hi Guys,

a durable actor queue sounds nice but what should be the exact
semantics for failure atomicity and isolation? Having a lot of small
transactional data-sources (mailbox, the datastructures backed up by
some datastore) doesn't make your system transactional. So what should
be the semantics of some datastructures have been committed and others
have not (normally a rollback would solve this issue but with multiple
transactional datastructures this is going to be complicated).

And there certainly is going to be a considerable amount of overhead
when you need to persist for every message placed or removed from the
mailbox.

For the Multiverse project I'm working on a durable STM
http://pveentjer.wordpress.com/2010/03/21/simplifying-enterprise-applications-with-durable-stm

[Jan Van Besien]

Having an actor pool with automatic work balancing is exactly what I
am doing in the "workstealing" branch (http://github.com/jboner/akka/
tree/workstealing). I replaced the ConcurrentLinkedQueue with a
LinkedBlockingDeque and created an alternative dispatcher
(ExecutorBasedEventDrivenWorkStealingDispatcher) which uses work
stealing to redistribute work between actors.

All actors registered on the work stealing dispatcher are considered
to be part of the same "pool" (they are verified to be of the same
type when registering).

The main thing which remains to be done is to have a clever mechanism
to choose an actor to steal work from. I had a look at how it was done
in the fork/join framework, and apparently they do a few samples at
random, and then start going round robin.

It would be nice if a durable mailbox would be efficient for work
stealing as well (like the semantics of a Deque).

It is true that in my work stealing implementation messages are no
longer processed "in order". On the other hand, what does "in order"
mean when you have multiple threads processing messages "at the same
time"...

Jan

On Mar 24, 5:49 am, Debasish Ghosh <ghosh.debas...@gmail.com> wrote:
> Hi -
>
> Quite some time back myself and Jonas had discussed the possibilities of
> having a durable mailbox for actors. This set up the context for the Issue
> #97 in the current tracker. We revived the discussion a couple of days back
> and had a few interesting mail exchanges. This is a summary of the current
> thoughts. Would appreciate some brainstorming over the ideas.
>
> Currently Akka supports the let-it-crash idiom over actor restarts through
> supervision. But it cannot support message persistence of the actor
> mailboxes. The mailbox is currently implemented as a ConcurrentLinkedQueue,
> which is transient and falls flat over an actor's death. Can we make this
> durable ? Redis offers queue implementation APIs through its List data
> structure. In fact we have an implementation of a Transactional Queue in
> Akka today, which is based on Redis.
>
> Another stream of thought was load balancing for busy actors with multiple
> consumers. Once the messages are pushed to the mailbox, from
> then onwards its FIFO. Can we improve that ? We had been discussing if we
> can have multiple queues for actors with multiple consumers. But we need to
> keep in mind about message ordering too. This leads to an idea of

> implementing *actor pools* for busy ones. Of course we need to make this

> configurable, so that the user can set up actor pools for specific actors
> declaratively.
>
> Ideas ? Thoughts ?
>
> Thanks.
>
> --

> Debasish Ghoshhttp://manning.com/ghosh

[Jonas Bonér]

Redis has transactions and using the new DISCHARD operation we can do
a rollback. Should be a no-brainer to hook up with existing STM.

Not sure we need more backends since it is meant to be a blackbox
thing. But if we do then AMQP, JMS etc also has transactions that
should be easy to hook up with.

I'll do a JTA adaptor to the STM soon (would take less than a day).
And then we can hook into most tx systems in JEE.

--
Jonas Bonér
http://jayway.com
http://akkasource.org

[Jonas Bonér]

I have no problem too, for some cases, loosen up the ordering
garantuees (currently ordering per actor mailbox preserved). Since if
you rely on ordering for you application to work then IMO actors and
event-driven architecture in general is the wrong model.

--
Jonas Bonér
http://jayway.com
http://akkasource.org

On Mar 24, 2010, at 11:25, Jan Van Besien <janvan...@gmail.com>
wrote:

[Jonas Bonér]

Inline. 

--

Jonas Bonér 

http://jayway.com 

http://akkasource.org

On Mar 24, 2010, at 6:13, Krishna Sankar <ksan...@gmail.com> wrote:

1. Interesting idea. Durable queues have been a feature of the MQ systems for a long time
2. We do need a datastore for the durability – assuming durability means the queue lasts a process crash and also the machine crash

The idea was to at least initially use Redis. 

3. would be nice if could abstract the datastore interface, most probably a get/put/delete key-values would be enough. That way one can implement the durable MBs using Redis or Cassandra or even AMZ SimpleDB or SQS

The impl should not leak out to user. But only be a config option and/or a Trait. From user standpoint the impl should not have to change to get a durable queue. Same impl and same interface. 

I think other storages than Redis is a good idea long term. Let the user reuse existing infrastructure he already has and are administrating.  

4. And if we pick Cassandra or something similar, we might also get distribution and replication capability. There is the question of eventual consisteny, which we would need to turn off.

How would this work? I mean which semantics would a replicated and eventually consistent queue have? 

5. The ordering is a tricky business – adds the state dimension. Better to avoid it. If there is

Right. Avoid it. That's my view of the world. 

5. only one consumer and one actor, then the order is implicit.

Right. As it is now. 

5. If there are more than one consumer, the order is pseudo implicit. With multiple consumers and actors, the order is indeterminate

Right. 

6. I assume we are not making any assumptions about the processing time for each message – for example a message might take a long time while others might be processed very quickly. So better to make no assumptions – and so no guarantees on the ordering.
7. Configurable actor pools is a good idea.
8. A good  feature set from the message queuing side is the AMZ SQS.

There is one guys that have started a SQS impl about 6 months ago. But seems to have stalled. See forks on GitHub. 

9. Another question is the metadata – do we need to track any metadata ?

Such as? Sender ref? Would be nice.  

10. Would be happy to help
    

Great. I'd love to get your help. Thanks. 

Cheers
<k/>
On 3/23/10 Tue Mar 23, 10, "Debasish Ghosh" <ghosh.d...@gmail.com> wrote:

Hi -

Quite some time back myself and Jonas had discussed the possibilities of having a durable mailbox for actors. This set up the context for the Issue #97 in the current tracker. We revived the discussion a couple of days back and had a few interesting mail exchanges. This is a summary of the current thoughts. Would appreciate some brainstorming over the ideas.

Currently Akka supports the let-it-crash idiom over actor restarts through supervision. But it cannot support message persistence of the actor mailboxes. The mailbox is currently implemented as a ConcurrentLinkedQueue, which is transient and falls flat over an actor's death. Can we make this durable ? Redis offers queue implementation APIs through its List data structure. In fact we have an implementation of a Transactional Queue in Akka today, which is based on Redis.

Another stream of thought was load balancing for busy actors with multiple consumers. Once the messages are pushed to the mailbox, from then onwards its FIFO. Can we improve that ? We had been discussing if we can have multiple queues for actors with multiple consumers. But we need to keep in mind about message ordering too. This leads to an idea of implementing actor pools for busy ones. Of course we need to make this configurable, so that the user can set up actor pools for specific actors declaratively.

Ideas ? Thoughts ?

Thanks.

--

[Jonas Bonér]

Inline.

--
Jonas Bonér
http://jayway.com
http://akkasource.org

On Mar 24, 2010, at 7:53, Martin Krasser <kras...@googlemail.com>
wrote:

>
> Debasish Ghosh schrieb:
>> Hi -
>>
>> Quite some time back myself and Jonas had discussed the
>> possibilities of having a durable mailbox for actors. This set up
>> the context for the Issue #97 in the current tracker. We revived
>> the discussion a couple of days back and had a few interesting mail
>> exchanges. This is a summary of the current thoughts. Would
>> appreciate some brainstorming over the ideas.
>>
>> Currently Akka supports the let-it-crash idiom over actor restarts
>> through supervision. But it cannot support message persistence of
>> the actor mailboxes. The mailbox is currently implemented as a
>> ConcurrentLinkedQueue, which is transient and falls flat over an
>> actor's death. Can we make this durable ? Redis offers queue
>> implementation APIs through its List data structure. In fact we
>> have an implementation of a Transactional Queue in Akka today,
>> which is based on Redis.
>
> Thanks for starting this thread. I consider this as absolutely
> necessary for reliable messaging in actor-based systems. Having a
> Redis-based transactional queue as mailbox should be IMHO only one
> option for persisting messages. Another option

Agree. See my previous mail.

> could be a MOM such as ActiveMQ, for example. For reading from and
> writing to the mailbox the same interface is used with different,
> configurable implementations (e.g. transient, persistent-redis,
> persistent-jms etc.). Another aspect is to let actors participate in
> transactions that are managed by the storage backend. For example,
> when using ActiveMQ (or any other JMS impl), we could let consuming
> actors participate in transactions started by the MOM. When the
> actor dies (i.e. the transaction is not committed) or the
> transaction is rolled back for some other reason, the message can be
> redelivered by ActiveMQ based on a configurable redelivery policy.
> Using

I Iike that.

> an ActiveMQ-based approach also offers some options for load-
> balancing e.g. to let actor concurrently consume messages from the
> same queue. ActiveMQ also provides options for configuring
> distributed store-and-forward networks for distributing load across
> different nodes. These are just

Also interesting ideas.

We should collect all ideas in Assembla.

Thanks for joining in the discusion.

[Martin Krasser]

Jonas Bon�r schrieb:

> Redis has transactions and using the new DISCHARD operation we can do
> a rollback. Should be a no-brainer to hook up with existing STM.
>
> Not sure we need more backends since it is meant to be a blackbox
> thing. But if we do then AMQP, JMS etc also has transactions that
> should be easy to hook up with.
>
> I'll do a JTA adaptor to the STM soon (would take less than a day).
> And then we can hook into most tx systems in JEE.

Great! This will allow akka-camel to support transactional Camel
endpoints as well. Looking forward to play around with it!

>
> --
> Jonas Bon�r

> http://jayway.com
> http://akkasource.org
>
> On Mar 24, 2010, at 10:40, Peter Veentjer <alarm...@gmail.com> wrote:
>
>> Hi Guys,
>>
>> a durable actor queue sounds nice but what should be the exact
>> semantics for failure atomicity and isolation? Having a lot of small
>> transactional data-sources (mailbox, the datastructures backed up by
>> some datastore) doesn't make your system transactional. So what should
>> be the semantics of some datastructures have been committed and others
>> have not (normally a rollback would solve this issue but with multiple
>> transactional datastructures this is going to be complicated).
>>
>> And there certainly is going to be a considerable amount of overhead
>> when you need to persist for every message placed or removed from the
>> mailbox.
>>
>> For the Multiverse project I'm working on a durable STM
>> http://pveentjer.wordpress.com/2010/03/21/simplifying-enterprise-applications-with-durable-stm
>>
>>
>> On Wed, Mar 24, 2010 at 6:13 AM, Krishna Sankar <ksan...@gmail.com>
>> wrote:
>>> Interesting idea. Durable queues have been a feature of the MQ
>>> systems for a
>>> long time

>>> We do need a datastore for the durability � assuming durability
>>> means the

>>> queue lasts a process crash and also the machine crash
>>> would be nice if could abstract the datastore interface, most
>>> probably a
>>> get/put/delete key-values would be enough. That way one can
>>> implement the
>>> durable MBs using Redis or Cassandra or even AMZ SimpleDB or SQS
>>> And if we pick Cassandra or something similar, we might also get
>>> distribution and replication capability. There is the question of
>>> eventual
>>> consisteny, which we would need to turn off.

>>> The ordering is a tricky business � adds the state dimension. Better to

>>> avoid it. If there is only one consumer and one actor, then the
>>> order is
>>> implicit. If there are more than one consumer, the order is pseudo
>>> implicit.
>>> With multiple consumers and actors, the order is indeterminate
>>> I assume we are not making any assumptions about the processing time
>>> for

>>> each message � for example a message might take a long time while
>>> others
>>> might be processed very quickly. So better to make no assumptions �

>>> and so
>>> no guarantees on the ordering.
>>> Configurable actor pools is a good idea.
>>> A good feature set from the message queuing side is the AMZ SQS.

>>> Another question is the metadata � do we need to track any metadata ?

--

[Viktor Klang]

Thanks for opening this can of goodies!

Really looking forward to a store-and-forward type solution for making sure messages don't get dropped.
This also slightly overflows into the resend-message-if-actor-dies-while-processing-said-message territory.


Good times!

On Wed, Mar 24, 2010 at 1:09 PM, Martin Krasser <kras...@googlemail.com> wrote:

Jonas Bonér schrieb:

Redis has transactions and using the new DISCHARD operation we can do a rollback. Should be a no-brainer to hook up with existing STM.

Not sure we need more backends since it is meant to be a blackbox thing. But if we do then AMQP, JMS etc also has transactions that should be easy to hook up with.

I'll do a JTA adaptor to the STM soon (would take less than a day). And then we can hook into most tx systems in JEE.

Great! This will allow akka-camel to support transactional Camel endpoints as well. Looking forward to play around with it!

--

Jonas Bonér

http://jayway.com
http://akkasource.org

On Mar 24, 2010, at 10:40, Peter Veentjer <alarm...@gmail.com> wrote:

Hi Guys,

a durable actor queue sounds nice but what should be the exact
semantics for failure atomicity and isolation? Having a lot of small
transactional data-sources (mailbox, the datastructures backed up by
some datastore) doesn't make your system transactional. So what should
be the semantics of some datastructures have been committed and others
have not (normally a rollback would solve this issue but with multiple
transactional datastructures this is going to be complicated).

And there certainly is going to be a considerable amount of overhead
when you need to persist for every message placed or removed from the
mailbox.

For the Multiverse project I'm working on a durable STM
http://pveentjer.wordpress.com/2010/03/21/simplifying-enterprise-applications-with-durable-stm

On Wed, Mar 24, 2010 at 6:13 AM, Krishna Sankar <ksan...@gmail.com> wrote:

Interesting idea. Durable queues have been a feature of the MQ systems for a
long time

We do need a datastore for the durability – assuming durability means the

queue lasts a process crash and also the machine crash
would be nice if could abstract the datastore interface, most probably a
get/put/delete key-values would be enough. That way one can implement the
durable MBs using Redis or Cassandra or even AMZ SimpleDB or SQS
And if we pick Cassandra or something similar, we might also get
distribution and replication capability. There is the question of eventual
consisteny, which we would need to turn off.

The ordering is a tricky business – adds the state dimension. Better to

avoid it. If there is only one consumer and one actor, then the order is
implicit. If there are more than one consumer, the order is pseudo implicit.
With multiple consumers and actors, the order is indeterminate
I assume we are not making any assumptions about the processing time for

each message – for example a message might take a long time while others
might be processed very quickly. So better to make no assumptions – and so

no guarantees on the ordering.
Configurable actor pools is a good idea.
A good  feature set from the message queuing side is the AMZ SQS.

Another question is the metadata – do we need to track any metadata ?

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

[Debasish Ghosh]

Had a peek into the workstealing branch today. Implementing the work stealing we will have an actor pool with the load being shed off from the busy actors to the others. I guess if we can replace the LinkedBlockingDeque with a durable implementation having the same semantics, we will have what we are targeting as an initial implementation - durability + load balancing. As I mentioned earlier in this thread we already have a queue implementation in Redis. And we can upgrade it to a deque and add the blocking semantics. This should not be too difficult since redis has BLPOP and BRPOP already in place.

Thanks to all for chipping in .. it's turning out to be a very useful thread.

--

You received this message because you are subscribed to the Google Groups "Akka User List" group.
To post to this group, send email to akka...@googlegroups.com.
To unsubscribe from this group, send email to akka-user+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

--
Debasish Ghosh

[Odd Möller]

Hi

This is great news. Durable actor mailboxes is in my opinion one of the (few) important pieces still missing from the Akka puzzle. Just a quick observation: isn't Redis supported on POSIX-compatible systems only (i.e. not Windows). Perhaps that is a good reason to allow for alternative queue implementations.

Keep up the good work!

Best regards

Odd Möller

[Craig Blake]

On Mar 24, 2010, at 7:00 AM, Jonas Bonér wrote:
> There is one guys that have started a SQS impl about 6 months ago. But seems to have stalled. See forks on GitHub.

We've migrated from SQS to AMQP (and off of SimpleDB as well) so there's not likely to be anymore work on that fork. Sorry.

Craig

[Krishna Sankar]

Agreed. Good point. We need to make sure we have simplified the
functionality with appropriate assumptions. Arbitrary order with n:m actors
consumers is one such simplifying assumption.

Cheers
<k/>

[Krishna Sankar]

I was thinking about it as well.
Either we can do an atomic [get-delete] operation, in which case if the actor dies while processing, the message is lost.
Or we can do a [get-mark]-[process]-[delete] operation with timeout for mark.
An additional mechanism is to have an ack queue, in which case we would need a correlationID. Most probably a names paced ID is easier, avoiding the need for a Zookeeper-ish coordinator.  
Cheers
<k/>

On 3/24/10 Wed Mar 24, 10, "Viktor Klang" <viktor...@gmail.com> wrote:

[buka]

I think having the visibility timeout and auto-requeue is a necessary
characteristic of any durable queue that is predicated upon supporting
the let-it-crash mantra. Also, re the back-end impl, it might be a
good idea to make this configurable in some manner especially when it
comes to distributing messages across actors separated by arbitrary
network topologies.. I mean, it would be nice to have the same
mechanism to dispatch over a "local" Redis- TokyoCab- backed impl as
going over SQS to get to some partnering service.

On Mar 24, 8:30 am, Krishna Sankar <ksanka...@gmail.com> wrote:
> I was thinking about it as well.
> Either we can do an atomic [get-delete] operation, in which case if the
> actor dies while processing, the message is lost.
> Or we can do a [get-mark]-[process]-[delete] operation with timeout for
> mark.
> An additional mechanism is to have an ack queue, in which case we would need
> a correlationID. Most probably a names paced ID is easier, avoiding the need
> for a Zookeeper-ish coordinator.
> Cheers
> <k/>
>

> On 3/24/10 Wed Mar 24, 10, "Viktor Klang" <viktor.kl...@gmail.com> wrote:
>
> > Thanks for opening this can of goodies!
>
> > Really looking forward to a store-and-forward type solution for making sure
> > messages don't get dropped.
> > This also slightly overflows into the
> > resend-message-if-actor-dies-while-processing-said-message territory.
>
> > Good times!
>

> > On Wed, Mar 24, 2010 at 1:09 PM, Martin Krasser <krass...@googlemail.com>

[Jonas Bonér]

No worries. Thanks anyway. Inspires to continue it.

--
Jonas Bonér
http://jayway.com
http://akkasource.org

[Alex Cruise]

I apologize if this is covering well-trodden ground already, but it's a
big thread and I'm a bit busy.

One useful detail from the JMS world about when multiple receivers are
listening on the same queue is that the JMS provider is supposed to
ensure that each message is delivered *at most once* to *some* receiver;
if the queue is marked as persistent then of course there's also some
additional chance that messages will actually be received once (as
opposed to not at all :)

Some JMS providers seem to implement at-most-once by having receivers
block one another, and at least one that I've observed (MQSeries) will
essentially load-balance to every receiver.

-0xe1a

[Krishna Sankar]

Yep, good point. We need to consider "at least once" vs. "atmost once"
pattern.

I think, for a start, we should do only "at least once" with the
[get-mark]-[process]-[delete/ack] or [get-mark]-[process]-[re-queue on
timeout] until [delete-ack] semantics.

IMHO, might not be a good idea to duplicate JMS/*MQ semantics - folks can
use them independently. We should aim for an embedded, transparent and
simple functionality - other wise, things can become widely complex; at
which point it is relatively easier to use a JMS/*MQ solution.

Cheers
<k/>

[Jonas Bonér]

I like that. Thanks Alex and Krishna.

Krishna, feel free to fork off and start hacking on it. Synchronize
with Debasish. Thanks.

--
Jonas Bonér
http://jayway.com
http://akkasource.org

[Espen]

Will this effect the performance of the core Actor implementation?

We are investigating Akka actors for simplicity and performance,
remote actors and the supervision hierarchies.

-Espen

On Mar 24, 8:14 pm, Jonas Bonér <jo...@jonasboner.com> wrote:
> I like that. Thanks Alex and Krishna.
>
> Krishna, feel free to fork off and start hacking on it. Synchronize  
> with Debasish. Thanks.
>
> --

> Jonas Bonérhttp://jayway.comhttp://akkasource.org

>
> On Mar 24, 2010, at 19:04, Krishna Sankar <ksanka...@gmail.com> wrote:
>
> > Yep, good point. We need to consider "at least once" vs. "atmost once"
> > pattern.
>
> > I think, for a start, we should do only "at least once" with the
> > [get-mark]-[process]-[delete/ack] or [get-mark]-[process]-[re-queue on
> > timeout] until [delete-ack] semantics.
>
> > IMHO, might not be a good idea to duplicate JMS/*MQ semantics -  
> > folks can
> > use them independently. We should aim for an embedded, transparent and
> > simple functionality - other wise, things can become widely complex;  
> > at
> > which point it is relatively easier to use a JMS/*MQ solution.
>
> > Cheers
> > <k/>
>

[Viktor Klang]

On Wed, Mar 24, 2010 at 11:15 PM, Espen <espen....@gmail.com> wrote:

Will this effect the performance of the core Actor implementation?

I believe the idea was to have it configurable per actor, so if you don't want to use it, it won't affect the performance.
 

For more options, visit this group at http://groups.google.com/group/akka-user?hl=en.

[Jonas Bonér]

On 24 March 2010 23:22, Viktor Klang <viktor...@gmail.com> wrote:
>
>
> On Wed, Mar 24, 2010 at 11:15 PM, Espen <espen....@gmail.com> wrote:
>>
>> Will this effect the performance of the core Actor implementation?
>
> I believe the idea was to have it configurable per actor, so if you don't
> want to use it, it won't affect the performance.

Right. Not in any way. Will probably just be a new dispatcher. Orthogonal.

--
Jonas Bonér

twitter: @jboner
blog: http://jonasboner.com
work: http://scalablesolutions.se
code: http://github.com/jboner
code: http://akkasource.org
also: http://letitcrash.com

[Debasish Ghosh]

On Thu, Mar 25, 2010 at 4:13 AM, Jonas Bonér <jo...@jonasboner.com> wrote:

On 24 March 2010 23:22, Viktor Klang <viktor...@gmail.com> wrote:
>
>
> On Wed, Mar 24, 2010 at 11:15 PM, Espen <espen....@gmail.com> wrote:
>>
>> Will this effect the performance of the core Actor implementation?
>
> I believe the idea was to have it configurable per actor, so if you don't
> want to use it, it won't affect the performance.

Right. Not in any way. Will probably just be a new dispatcher. Orthogonal.

>> and optionally a new durable Queue implementation ..

--
Debasish Ghosh

[Jeppe Nejsum Madsen]

Craig Blake <craig...@gmail.com> writes:

Any specific reasons you can share for the move away from SQS and AMPQ?

/Jeppe

[Jonas Bonér]

--

Jonas Bonér 

http://jayway.com 

http://akkasource.org

On Mar 25, 2010, at 9:53, Debasish Ghosh <ghosh.d...@gmail.com> wrote:

On Thu, Mar 25, 2010 at 4:13 AM, Jonas Bonér <jo...@jonasboner.com> wrote:

On 24 March 2010 23:22, Viktor Klang <viktor...@gmail.com> wrote:
>
>
> On Wed, Mar 24, 2010 at 11:15 PM, Espen <espen....@gmail.com> wrote:
>>
>> Will this effect the performance of the core Actor implementation?
>
> I believe the idea was to have it configurable per actor, so if you don't
> want to use it, it won't affect the performance.

Right. Not in any way. Will probably just be a new dispatcher. Orthogonal.

>> and optionally a new durable Queue implementation ..

Right. Of course. 

[Craig Blake]

The max message size is a little too small for some of our messages, for one, and it's too much overhead to push that data temporarily into S3 or another object store. We also need to have real pub/sub support for a few of our systems. AMQP is a better fit for those needs.

We're also moving from SimpleDB to CouchDB using the scouchdb library (thanks Debasish!), mostly due to concerns about performance consistency and replication.

To add my two cents to the thread, though, it would be great to have durable mailboxes, and it makes sense to only aim for "at-least-once" guarantees initially.

[Jonas Bonér]

Thanks for your comments. I agree that "at least once" is the right
default.

--
Jonas Bonér
http://jayway.com
http://akkasource.org