RabbitMQ: Auto-clean \ Deleting queues after consuming message

[ajmal rahman]

0down votefavorite 1

My application would need to connect to a RabbitMQ queue and consume the message in it (only 1) . And I must also make sure that the queue is deleted right after the message is consumed. I was looking for a way to do this, with simple configurations. Is that possible? If yes, how? How do I create a consumer and configure such that it gets cancelled\ the queue gets deleted after the message is consumed? I was thinking of using auto-delete, but doesn't this mean I would need to cancel the consumer explicitly? And I would need to set the consumer  tag for the same? I could also delete the queue explicitly, but then the consumer listening to the queue would throw an exception. Are there better alternatives?

[Michael Klishin]

See http://www.rabbitmq.com/queues.html.

If there ever was a consumer, it doesn't have to be cancelled explicitly (and even if that was the case, what's the big deal?)

A channel closure or connection termination would suffice.

Exclusive queues might work as well but, of course, they can only be used by the declaring connection

so they cannot exist prior to consumer connection. Although for response collection and temporary consumer state

they make most sense.

Consumer tag generation is entirely orthogonal to all of the above. Some clients generate them for you, or you can pass

a blank string and the server will generate one. In most clients it is rarely needed to provide a consumer tag if you don't care what it is.

Server-generated consumer tags are returned with the basic.consume method response: basic.consume-ok. E.g. in the Java client

this translates to a string returned by (some) Channel#basicConsume overloads.

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[ajmal rahman]

Hi MK,

Thanks for the response,

It is important for us to delete the queues since there would be 1000's of them created. Another application creates the queues and hence I believe exclusive is not an option (not the same connection)  . We just create consumers for the queues. The queue is not required after the message is consumed. How best to delete the queue in this case? 

Regards,,

Ajmal

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Use auto-delete queues and set a TTL on them in case you intentionally

or not begin consuming with basic.get at some point in the future:
http://www.rabbitmq.com/ttl.html.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ajmal rahman]

If I set a TTL (for example 10 seconds) and auto-delete (10 seconds) - does that mean the queue gets deleted even if there are consumers for it (not cancelled) ?

Also - do I get the same behavior wit TTL + Expires - the only difference being it gets deleted only after the period defined in the xpires parameter?

[Michael Klishin]

The question is answered in http://www.rabbitmq.com/ttl.html:

«This controls for how long a queue can be unused before it is automatically deleted. Unused means…»

Auto-delete is a boolean property, not an expiration setting, see http://www.rabbitmq.com/queues.html.

Queue TTL should not be confused with *message* TTL. I recommended the former in this thread.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Just in case: I recommended using auto-delete queues (since exclusive aren't an option per OP's comment).

It should be sufficient in most cases, primarily when there actually are consumers.

Queue TTL was recommended as a guardrail for when there will be no consumers. And yes, there will be

such cases even if you don't expect that as consumers can fail, refuse to start, undergo planned maintenance,

have queue names misconfigured, switched to basic.get without understanding of the effect it makes on the auto-delete
behavior and so on.

[Luke Bakken]

Ajmal,

Your questions are answered in the documentation to which Michael linked:

http://www.rabbitmq.com/ttl.html#queue-ttl

You should try the following scenarios in your environment:

* Run a consumer that declares a queue with auto-delete set to true and x-expires set to 10 seconds.

* Open the management UI and go to the queues page to observe.

* Start a basic.consume operation on the queue, and sleep for longer than 10 seconds. What happens with the queue?

* Cancel the consume operation, close the channel and connection. Is the queue still there?

Now, re-run the above scenario but do not create the queue as auto-delete.

The following golang code can be used to demonstrate these scenarios - https://gist.github.com/lukebakken/4c390610ed44594d5be5450373a54504

Thanks,

Luke

[Michael Klishin]

And to complete the picture, replace basic.consume (a consumer) with consuming using basic.get

(which doesn't register a consumer) to compare.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ajmal rahman]

Thanks Luke and Michael for the suggestions.

I tried both:

Creating a queue with auto-delete set as true and  x-expires 10 seconds.

Creating a queue with auto-delete set as false and  x-expires 10 seconds.

In both cases, the queue is not deleted until the consumer is cancelled \ connection is closed - the only difference being it is deleted immediately when auto-delete is set as true.

Does this mean that I need to cancel the consumer explicitly and there is no other way? (I would like to use basic.consume as opposed to basic.get as I read it is faster)

Regards,

Ajmal

[Michael Klishin]

Ajmal,

Sorry but I feel we keep walking around in circles in this thread.

From http://www.rabbitmq.com/ttl.html:

«This [queue TTL] controls for how long a queue can be unused before it is automatically deleted.
Unused means the queue has no consumers, the queue has not been redeclared, and basic.get has not been invoked for a duration of at least the expiration period…»

You *don't have to* cancel the consumer, although that's quite trivial to do anyway.
It's sufficient to just close the channel or connection. The above sentence explains the conditions under which

the queue TTL will kick in.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ajmal rahman]

Thanks, I do get your point.

I was looking for a delete a queue without cancelling the consumer or closing the channel\connection. Looks like there isn't any!

Regards,

Ajmal

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

Why would a queue that still has consumers be automatically deleted? What should then happen to those

consumers? The only option that makes sense to me is that they would have to be cancelled [1]. Which would

be no meaningfully different from cancelling the consumer in your own code except it would be much harder to reason about.

1. http://www.rabbitmq.com/consumer-cancel-notifications.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[ajmal rahman]

The queue is created by another application for which our application creates the consumer. The queue would contain only 1 message after which there is no use of it. What I need is to consume it and then delete the queue. But then the consumer has to be cancelled before deleting the queue I believe. Cancelling the consumer seems to be a bit tricky in our code, since we need to do it based on the consumer tag.

Is there a way :

I can delete the queue using deleteQueue - also making sure the consumers are cancelled?

Can I get the consumer of a queue based on the queue name?

Do I always need a consumer tag to cancel a consumer?

Regards,

Ajmal

[Michael Klishin]

Cancelling a consumer is trivial. If it's not then you have some code to refactor.

Consumer tags are consumer identifiers. Can you delete a database record without a primary key of some kind?

It works the same way for consumer cancellation in RabbitMQ.

I linked to a protocol extension that notifies consumers when their queue is deleted "from underneath them" (e.g. by a different app).

Instead of dragging on with the assumption that cancelling a consumer is hard and there has to be a magical

feature, make it easy to cancel consumers and do that.

You can list consumers of a queue over HTTP API but if you absolutely have to do that even though your app

adds the consumer, it sounds like a major red flag to you. Don't do that, refactor your code instead of piling on hacks.

Lastly, the auto-delete + TTL combination recommendation still stands.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Michael Klishin]

That should read "it sounds like a major red flag to me (and should to you)".

[Michael Klishin]

The link I provided is incorrect, the correct one is https://www.rabbitmq.com/consumer-cancel.html.

[ajmal rahman]

Sure, let me do that then.

To confirm, auto-delete + TTL combination works only if the consumer is cancelled \ connection or channel  is closed?

[Michael Klishin]

Answers to those questions were provided earlier in the thread and in the docs.

I'm locking this thread because it keeps going in circles.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.