Purging a large queue is slow

[Raymond Rizzuto]

When I purge a large queue via the web admin or via the easy net q admin api, it takes so long that sometimes the request times out, even though it succeeds. Is there any way to make this operation more efficient?

[Michael Klishin]

How many messages are there? Purging is potentially a very I/O heavy operation.

MK

> On 1/7/2015, at 16:01, Raymond Rizzuto <ray.r...@gmail.com> wrote:
>
> When I purge a large queue via the web admin or via the easy net q admin api, it takes so long that sometimes the request times out, even though it succeeds. Is there any way to make this operation more efficient?
>

> --
> You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
> To post to this group, send an email to rabbitm...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ray Rizzuto Jr]

100k or more.

[Michael Klishin]

On 1 July 2015 at 16:24:58, Ray Rizzuto Jr (r.j.r...@ieee.org) wrote:
> 100k or more.

I don't know what default .NET HTTP API client timeouts are like but
removing 100K+ of rows in a data store usually takes some time as well. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Ray Rizzuto Jr]

Right, but dropping and recreating a table is fast.  I would hope it might be able to optimize file deleted vs message deleted similarly.

[Michael Klishin]

 On 1 July 2015 at 18:33:41, Ray Rizzuto Jr (r.j.r...@ieee.org) wrote:
> Right, but dropping and recreating a table is fast. I would hope
> it might be able to optimize file deleted vs message deleted similarly.

In relational databases rows are not shared between tables. In RabbitMQ, a message
may be in N queues at once, so there's a separate index just for that, and even when
you delete a queue, it's more often than not does not come down to deleting one file.

[Alvaro Videla]

Hi,

While there seems to be room for improvement according to this: https://github.com/rabbitmq/rabbitmq-server/blob/master/src/rabbit_variable_queue.erl#L554

Keep in mind that queue.purge is not the same as deleting a DB table. Queue purge only removes messages that haven't been fetched from the queue, but it doesn't delete the queue. If a consumer fetched a message from queue A, then we purge queue A, that consumer expects queue A to exist, it even will issue an ack back to queue A about said message. Even more, other publishers could be sending messages to that queue.

Regards,

Alvaro

[Raymond Rizzuto]

That improvement sounds like it would be useful in my case.  I'm not sure if it matters, but all of the messages are embedded in the index since they are under the 4k limit. 

[Raymond Rizzuto]

One other option that might be useful is to make the purge from the web asynchronous.  Currently it appears to be synchronous, and the only reason appears to be to show a popup that the queue has been purged. 

[Alvaro Videla]

I started working on queue purge performance improvements. If there are no pending acks, this branch can purge 100.000 msgs in around 240 ms:

https://github.com/rabbitmq/rabbitmq-server/tree/rabbitmq-server-295

You can follow progress on this ticket: https://github.com/rabbitmq/rabbitmq-server/issues/295

To post to this group, send email to rabbitm...@googlegroups.com.