[rabbitmq-discuss] adjusting timeout for non-acked messages to be requeued
vishnu
Alexis Richardson
On Tue, Dec 29, 2009 at 4:37 AM, vishnu <pat...@gmail.com> wrote:
> Hi
> I'm trying to use rabbitmq for an web application thats close to
> synchronous. The main reason I'm looking at rabbitmq is for the reliability
> and transactionality. The idea is for the webapplication to pump messages
> into the queue and for a down stream application to subscribe in
> acknowledgement mode and for each message start a transaction, attempt to
> process the message and if it fails for technical reasons, to rollback the
> transaction.
OK.
> However, I would like to be able to configure the time for non-acked
> messages to get requeued to be picked up. Where can this be done?
This and other 'time based' behaviours are frequently requested, but
*alas* are not yet built-in to RabbitMQ.
However, don't give up. You could create a consumer (in any language)
which managed this for you, eg enforcing a disconnect after some time
has passed. Or, if you are feeling brave you could create a plugin to
do this 'inside the broker'. Plugins have to be written in erlang.
I hope that helps. Perhaps the list will have more immediately useful
suggestions.
Cheers,
alexis
> thanks
> Vishnu
> _______________________________________________
> rabbitmq-discuss mailing list
> rabbitmq...@lists.rabbitmq.com
> http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
>
>
_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
http://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss
vishnu
Vishnu
In what way?
On Tue, Dec 29, 2009 at 2:57 PM, vishnu <pat...@gmail.com> wrote:
> hi, sorry Im a bit confused. But the behavior of moving the message from the
> unacknowledge queue to the ready queue is already a part of rabbitmq.
If there were then it need not be time-based.
> There
> must be some magic constant somewhere which governs this musn't there?
alexis
Alexis Richardson
I am not the best person to answer this question, but I can tell you
that your behaviour can be achieved without requiring timers.
Assuming you are unwilling to simply close the channel and force the
requeue, then:
1. You could explicitly requeue by re-publishing the message. No need
to use acks here.
2. You could look at what one of the many task management systems do.
For example celery or quebert.
http://ask.github.com/celery/introduction.html
http://bytebucket.org/adroll/quebert/
Or look at the list here: http://delicious.com/alexisrichardson/rabbitmq+work
3. Other ways to force a requeue. I am not going to try and describe
these yet, in case I say something misleading.
Let me know what you think.
alexis
vishnu
Matthew Sackman
On Wed, Dec 30, 2009 at 09:07:53AM +0530, vishnu wrote:
> thanks, I think Im getting a better understanding now. Initially I wanted
> to look at auto requeueing for circumstances when my client subscribing to
> events went down or got disconnected. Now my understanding is that in those
> cases, messages get auto requeued.
Correct.
> However, if I choose not to autoack, then it looks like the
> responsibility of requeuing a client, forcing a time delay on the delivery
> and also maybe forcing a limit on the number of passes through the queue is
> going to have to become subscriber responsibility. I was hoping to handle
> that as a configuration on the queue itself outside of code, but I guess
> that isn't possible.
>
> (the use case we have requires us to make sure that every message is
> processed as far as possible. Meaning actual failures and successes are sent
> onward, but failures for technical reasons require us to try again later, by
> which time the technical issue might have been resolved).
Rabbit does eventual delivery. Thus eventually, the message will be
delivered. However, it may be delivered to more than one consumer. A
message that is not ack'd will, when the channel on which it was
delivered closes (or the connection, or some other condition that I
always forget, but which Matthias pointed out within about the last
month on this list), be reinserted into its queue to be redelivered.
Thus if you set qos.prefetch to 1, don't auto-ack, and then your client
goes down, you have the following set of properties:
1) Your client will only receive the next message when it's successfully
ack'd the current message.
2) If the client crashes, at most 1 message (i.e. the message that it's
currently processing) will be requeued.
3) If you want to do timeouts, if depends where you wish to put the
timeout - kill clients if they're taking too long (this seems odd) - or
drop messages you receive (i.e. ack them but do no work) if they're
deemed too old. But yes, that's all client-side logic - we don't do TTL
in the server yet.
Matthew
Uwe Kubosch
> delivered. However, it may be delivered to more than one consumer. A
> message that is not ack'd will, when the channel on which it was
> delivered closes (or the connection, or some other condition that I
> always forget, but which Matthias pointed out within about the last
> month on this list), be reinserted into its queue to be redelivered.
Where in the queue is the message requeued on disconnect?
If there is only one client, and prefetch is set to 1, and this client does not ack a message and disconnects, will the same message be delivered first on reconnect, or is there a chance of out-of-order delivery?
--
Uwe Kubosch
Systems Developer
Datek Wireless AS
u...@datek.no
http://datek.no/
Matthew Sackman
> > Rabbit does eventual delivery. Thus eventually, the message will be
> > delivered. However, it may be delivered to more than one consumer. A
> > message that is not ack'd will, when the channel on which it was
> > delivered closes (or the connection, or some other condition that I
> > always forget, but which Matthias pointed out within about the last
> > month on this list), be reinserted into its queue to be redelivered.
>
> Where in the queue is the message requeued on disconnect?
It's not defined in the spec. RabbitMQ puts it at the tail of the queue,
but there is no guarantee that that behaviour will not change in the
future. Do not rely on this.
> If there is only one client, and prefetch is set to 1, and this client does not ack a message and disconnects, will the same message be delivered first on reconnect, or is there a chance of out-of-order delivery?
As above - it's currently the case that the message will be put at the
end of the queue and will be the last message delivered. The spec does
not rule either way on this. You should be able to verify this behaviour
yourself from testing.
Matthew