How to pause/resume message delivery to queue on the server side

[Peter Long]

Hi,

I am new to RabbitMQ and may just not know the right terminology, and consequently am not finding the answer to this question when I search. Forgive me if this has been asked and answered.

I was wondering if there is a way to instruct the RabbitMQ server to temporarily pause delivery of new messages queued on a particular queue, allowing the queue depth to grow until instructed to let delivery resume. I am thinking of a scenario where some maintenance needs to be done downstream of the consumer(s), say the database used by the consumers needs to be restarted. I know that one solution is simply to stop the consumers. Unfortunately that would require some code refactoring which I would like to delay for now. At present my producers and consumers are running in the same process. I would like to keep the producers publishing to the queue, I just want to halt consumption. 

Thanks in advance,

--

Peter Long

[Michael Klishin]

On 20 January 2015 at 19:26:14, Peter Long (peter...@gmail.com) wrote:
> I was wondering if there is a way to instruct the RabbitMQ server
> to temporarily pause delivery of new messages queued on a particular
> queue, allowing the queue depth to grow until instructed to let
> delivery resume. I am thinking of a scenario where some maintenance
> needs to be done downstream of the consumer(s), say the database
> used by the consumers needs to be restarted. I know that one solution
> is simply to stop the consumers. Unfortunately that would require
> some code refactoring which I would like to delay for now. At present
> my producers and consumers are running in the same process. I
> would like to keep the producers publishing to the queue, I just
> want to halt consumption.

Peter,

Simply cancel your consumer with the basic.cancel protocol method. However exactly that
is done slightly varies between clients but typically method name is basic_cancel or basicCancel
and it is a part of the channel API.

When, when you want to resume consumption, re-add your consumer, e.g. the same consumer instance
if your client is object-oriented, or begin consuming with the same function.

Note that _some_ consumers may be stateful and cannot be reused but as far as RabbitMQ is concerned,
you just cancel and register consumers. There is no pause operation.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Peter Long]

MK,

Thanks for the quick reply. As I said in my original post, I understand that I can cancel from the consumer end. Unfortunately implementing that would require refactoring my code so my consumer and producer are not running in the same process OR adding a way to signal the process to cancel the consumer and later resume. I am asking if there exists a way to use a server side control to pause message delivery instead, thus saving me the time and expense of refactoring my code base.

--

Peter Long

[Michael Klishin]

On 20 January 2015 at 19:40:00, Peter Long (peter...@gmail.com) wrote:
> I am asking if there exists a way to use a server side control to
> pause message delivery instead, thus saving me the time and expense
> of refactoring my code base.

I'm afraid there is no.

[Peter Long]

Thanks. I was afraid that would be the answer.

On Tuesday, January 20, 2015 at 11:41:15 AM UTC-5, Michael Klishin wrote:

[V Z]

+10

We need to syspend delivery of messages when we know that consumers will not be able to process them due to temporary, but sometimes longer lasting, environmental conditions, like backend maintenance. Otherwise, consumers are actively rejecting and requeuing messages keeping themselves, Rabbit, and network busy.

We would love to be able to register a trigger/flag/semaphore with Rabbit, then associate it with one or more queues (usually many more), then be able to notify Rabbit given the signal/flag/semaphore name, and have all deliveries from those queues suspended until trigger/flag/semaphore is reset.

What do you think?

[dfed...@pivotal.io]

If you know that consumers are unable to process messages you can cancel this consumers and don't start them until conditions change. I understand that you would like to have broker control over consumers, but it seems more like client-side responsibility, since it's up to consumers to know if they can process messages or not.

[V Z]

True. You can. Operationally, though, when there are a lot of consumers, it becomes difficult to manage who to shut down, when to restart, who restarted, verify everything, etc. It's a lot easier to flip a single flag.

[Luigi Berrettini YNAP]

I found the "cancel consumer" answer on many posts on the Internet, but unfortunately it is only a "your consumer can cancel itself".

Since there is an extension "in which the broker will send to the client a basic.cancel in the case of such unexpected consumer cancellations" I understand that the broker has a way to cancel consumers, therefore is there a management API route that allows to cancel consumers (e.g. after we have listed the via the /api/consumers/vhost route)?

Thanks,

Luigi.

[Michael Klishin]

There is no HTTP API endpoint for cancelling a consumer. Consumers can be cancelled using any RabbitMQ client. 

On 18 May 2016 at 13:14:02, Luigi Berrettini YNAP (lby...@gmail.com) wrote:
> > I found the "cancel consumer" answer on many posts on the Internet,
> but unfortunately it is only a "your consumer can cancel itself".
>

> Since there is an extension(http://www.rabbitmq.com/consumer-cancel.html)

> "in which the broker will send to the client a basic.cancel in
> the case of such unexpected consumer cancellations" I understand
> that the broker has a way to cancel consumers, therefore is there
> a management API route that allows to cancel consumers (e.g.
> after we have listed the via the /api/consumers/vhost route)?
>

[imbacen]

Another +10 from me, this would be a very nice enhancement. A single flag that would suspend sending messages to (all?) consumers would be very useful when you have to do backend maintenance. While consumers should in theory be made transactional so they can be taken offline at any moment and message not being acked, sometimes you have other async tasks inside that transaction and graceful shutdown becomes pain in the ass. If you could simply suspend the consumption then you could safely destruct the consumers no matter what. I'm not sure how the protocol would handle this or if any extensions would be needed. A simple solution would be to lie to consumers that no messages are available when the flag is set. I think there is a use case for such a feature.

Current solution requires you to have extra management logic for consumers to cancel and resume them on demand.

Dne torek, 20. januar 2015 17.26.13 UTC+1 je oseba Peter Long napisala:

[Bert Speckels]

This post is quite old but it asks for something that I am looking for right now.

In our situation a user can decide to stop consuming messages via web ui. The problem is that we have a clustered backend. So we have to tell all nodes of the cluster to cancel their channel, then wait for their response and finally respond to the user that no more messages will be consumed.

A second problem is that we want to stop
consuming messages on another application which is not controlled from the mentioned UI/backend.

Having a pause and resume feature on queues in rabbitmq would be much easier and does not need any cluster or remote orchestration on the consuming side.

Much more: pause and resume can be integrated into management UI of rabbitmq.

[Luke Bakken]

Hi Bert,

Just as a reminder ... if you reply to an old thread your message may never be noticed. We prefer to have new discussions opened with links to old (more than a month or so) threads.

Currently, your only option server-side is to force-close a connection.

One issue that I can think of with server-side cancellations is, how does your consumer know that it shouldn't immediately try to resume? The server-initiated basic.cancel method does not support a "reason".

Thanks,

Luke