default/lazy limited queues performance

[Semen Vishniakov]

Hi all,

I tested lazy queues on our configuration (45 queues; fanout exchange; transient messages) 

and found out that they perform great on unlimited queues, however if I limit queue size to 20 messages, the result becomes completely different:

the RAM consumption grows rapidly up and it takes up to 7 times longer to enqueue all sent messages, than in the case of default queues.

Default no limit (test not finished):

Lazy no limit:

Default limit 20:

Lazy limit 20:

Could somebody explain this performance difference on limited queues?

I was also wondered why the RAM usage in case of default limited queues stays low and completely unchanged during the whole testing?

Thanks,

Sem

[Michael Klishin]

On 22 January 2016 at 17:10:30, Semen Vishniakov (vishny...@gmail.com) wrote:
> Could somebody explain this performance difference on limited
> queues?

Probably not without investigating first, and we are quite busy at the moment.

To clarify, how exactly did you limit them? Using max-length?

> I was also wondered why the RAM usage in case of default limited
> queues stays low and completely unchanged during the whole testing?

Because messages are dropped at roughly the same rate as they enter the queue.
Lazy queues are not really meant to be used with limits (they target exactly the opposite
use case), there are side effects that were not accounted/tested for. 
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Semen Vishniakov]

Probably not without investigating first, and we are quite busy at the moment. 

To clarify, how exactly did you limit them? Using max-length? 

Yes, using max-length.

 

Lazy queues are not really meant to be used with limits

What does removing default queue mode mean for those, using "max-length"?

Sem

[Michael Klishin]

Nothing: it won't ship until the end of 2016. By then

this and a lot of other things can be improved.

[Alvaro Videla]

Lazy queues work by pushing everything to disk unless there are consumers attached to the queues. This means when you add a limit of say 20, what you are doing while publishing is this:

- message is accepted by the queue and pushed to disk.

- message limit is checked, you have less than 20 msgs, so no problem

- repeat that process until there are 20 messages in the queue.

- message 21st arrives, queue length is over the limit, this means we need to drop the head of the queue, to do that, the lazy queue has to load whatever is on disk (20 msgs) in memory, and just drop the head. 

- message 22nd arrives, queue length is over the limit (and this will be the case until messages are consumed faster than they are published, or the queue is purged). The message on the head of the queue is dropped. Nothing is loaded from disk this time since the lazy queue still has 20 msgs in RAM.

- repeat the previous process until those 20 msgs are dropped due to the max-length settings.

- Once those 20 msgs have been dropped, the lazy queue will need to load again from the file system, just to drop 1 message from the head of the queue, and so on.

It's easy to see how this will present a performance problem sooner or later.

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To post to this group, send email to rabbitm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.