Message ack-ing

[Lee Henson]

Hi

A question about message acking. If I want my agent to return the
message to the queue if it failed to successfully process it (e.g. an
exception was thrown), how do I go about setting that up. From trying
to put the pieces together from various docs, it looks like you call
the ack method once you have finished processing a message, but in
agent.rb it looks like the ack happens before the message is
processed. What am I missing?

Cheers
Lee


def setup_queue
if amqp.respond_to?(:prefetch) && @options.has_key?(:prefetch)
amqp.prefetch(@options[:prefetch])
end
amqp.queue(identity, :durable => true).subscribe(:ack => true)
do |info, msg|
begin
info.ack <- acking here
receive(serializer.load(msg)) <- before processing here
rescue Exception => e
Nanite::Log.error("RECV #{e.message}")
end
end
end

[Lee Henson]

I've had a play around and got near the behaviour that I want
(exception in the actor to requeue the message unack-ed):

http://github.com/leemhenson/nanite/commit/c42ae9fd7ae86bf599df4ce3960b87e115343865#diff-0

Am I barking up the wrong tree?

[Mathias Meyer]

On Fri, Oct 29, 2010 at 7:38 PM, Lee Henson <lee.m....@gmail.com> wrote:
> I've had a play around and got near the behaviour that I want
> (exception in the actor to requeue the message unack-ed):
>
> http://github.com/leemhenson/nanite/commit/c42ae9fd7ae86bf599df4ce3960b87e115343865#diff-0
>
> Am I barking up the wrong tree?
>

That depends. That change assumes low-level AMQP things to deal with
messages that caused errors. From what I can see it'll basically put
the message on the same queue again, ending up on the same agent.
There's other things to consider than simply ack'ing the message after
the actor method was executed. Assume for example that an actor method
uses EventMachine to run some tasks asynchronously or schedules a
repeated task with a timer. Plus, if the actor method runs a longer
piece of code, the message may be delivered again at a later point.

I've thought about this problem a bit, and came to my personal
conclusion that just requeueing may not fully solve the problem, and
is probably just too low-level for a framework like Nanite. What I'd
say should happen instead is the message should be put back on the
request queue, where it's going to be picked up by a mapper again,
signaled by either the user (e.g. by calling a method) or
automatically when an exception is raised (like in your example).

For full reliability in terms of message delivery and processing, some
extra care has to be taken. I'm just not fully sure yet how that's
going to look. Some sort of delayed ack'ing may be part of it though.

Cheers, Mathias
--
http://paperplanes.de
http://twitter.com/roidrage

[Lee Henson]

> That depends. That change assumes low-level AMQP things to deal with
> messages that caused errors. From what I can see it'll basically put
> the message on the same queue again, ending up on the same agent.

Yes, this is exactly what I want in my situation. Obviously it places
a burden on the agent developer to have some sort of monitor on the
number of attempts there have been to process a message, and once a
threshold is breached it should be moved to a dead-letter queue. The
RabbitMQ blog suggests they are planning to add integrated dead-letter
queueing at some point, but it's not there at the moment.

> There's other things to consider than simply ack'ing the message after
> the actor method was executed. Assume for example that an actor method
> uses EventMachine to run some tasks asynchronously or schedules a
> repeated task with a timer. Plus, if the actor method runs a longer
> piece of code, the message may be delivered again at a later point.

Agreed, in this situation the current implementation makes sense as a
default.

> I've thought about this problem a bit, and came to my personal
> conclusion that just requeueing may not fully solve the problem, and
> is probably just too low-level for a framework like Nanite. What I'd
> say should happen instead is the message should be put back on the
> request queue, where it's going to be picked up by a mapper again,
> signaled by either the user (e.g. by calling a method) or
> automatically when an exception is raised (like in your example).

I think I would like to have the option. You could have the current
behaviour as the default, but with the capability within the agent to
specify explicit ack/nack-ing:

class ExplicitAckingAgent
expose :email_client, :explicit_ack => true

def email_client
...
end
end

What do you think? Would you agree to an api along those lines?

Cheers
Lee

[Mathias Meyer]

On Mon, Nov 1, 2010 at 5:53 PM, Lee Henson <lee.m....@gmail.com> wrote:

> I think I would like to have the option. You could have the current
> behaviour as the default, but with the capability within the agent to
> specify explicit ack/nack-ing:
>
> class ExplicitAckingAgent
>    expose :email_client, :explicit_ack => true
>
>    def email_client
>        ...
>    end
> end
>
> What do you think? Would you agree to an api along those lines?
>

That looks decent. I'm gonna give it some more thought over the next
couple of days to get a general idea of where this stuff should go and
how it should work and be exposed to the user.

[Lee Henson]

Ok, you can see where I'm at on the temp branch on my fork. I'll
continue working on it tomorrow. Hopefully it'll be something
contributable in the end.

github.com/leemhenson/nanite

On Nov 1, 9:27 pm, Mathias Meyer <pomonra...@googlemail.com> wrote: