Message pile up

[Amit Khosla]

Hi,

We are currently  using RabbitMQ version 3.6.10 (though planning to upgrade soon as well). Currently using classic queues only.

We often face a challenge that some times, some consumer is slow or is disconnected and due to this, we find that messages are piled up in the system.

As we read that RabbitMQ works best in case of low message count in queue (Please correct me if it is incorrect).

My question is, if there is a pileup in a queue whose consumer is not available what all problems we can observe due to this till the time the consumer is not attached?

And in case of slow consumer, what kind of challenges occur at RabbitMQ side?

Is there any setting in RabbitMQ which can minimize this impact?

Also, is there any way other than config(if not available), to handle this scenario?

Thanks & Regards

Amit Khosla

[Alex K]

On Thu, Aug 19, 2021, 08:06 Amit Khosla <amitkho...@gmail.com> wrote:

Hi,

We are currently  using RabbitMQ version 3.6.10 (though planning to upgrade soon as well). Currently using classic queues only.

We often face a challenge that some times, some consumer is slow or is disconnected and due to this, we find that messages are piled up in the system.

As we read that RabbitMQ works best in case of low message count in queue (Please correct me if it is incorrect).

My question is, if there is a pileup in a queue whose consumer is not available what all problems we can observe due to this till the time the consumer is not attached?

It depends from the amount of messages being buffered, number of queues and consumers. Generally you would see a slight increase of memory usage on the rmq server. You can observe this metric easily with any monitoring tool. 

And in case of slow consumer, what kind of challenges occur at RabbitMQ side?

As said, mostly memory usage.

Is there any setting in RabbitMQ which can minimize this impact?

In case you observe high memory usage you can try lazy queues. Or set a limit at the number of messages that can be queued, in case you can afford to loose some messages. 

Also, is there any way other than config(if not available), to handle this scenario?

Thanks & Regards

Amit Khosla

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CADGMEYUirfsgydW1OKCmhb2iG5KzPxYo2F6RmfFunhg-HR83GQ%40mail.gmail.com.

[Ken Peng]

You might need to upgrade the software and use the latest hardware.
I tune my rabbitmq cluster by reading this:
https://www.cloudamqp.com/blog/part2-rabbitmq-best-practice-for-high-performance.html

regards.

[Amit Khosla]

Thanks Alex and Ken for replies.

As mentioned in the document shared by Ken, queues works best when messages in queue is close to 0. So, what all implication occurs in case of higher values like may be a million messages piled up.

We do have around 1600 queues and have ttl set as well (3 node cluster of 16 core each having 64 GB memory as well). 

My question is regarding what else can be the concern other than memory usage? And due to one queue having many messages, will other queues also be affected? Or the queue having those messages is impacted?

Thanks & Regards

Amit

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA1D6YumW0OEf6BeLdonraeh6kohHBUo0mme6vuPjs7_zbvpfg%40mail.gmail.com.

--

[Amit Khosla]

Thanks Alex and Ken for replies.

As mentioned in the document shared by Ken, queues works best when messages in queue is close to 0. So, what all implication occurs in case of higher values like may be a million messages piled up.

We do have around 1600 queues and have ttl set as well (3 node cluster of 16 core each having 64 GB memory as well). 

My question is regarding what else can be the concern other than memory usage? And due to one queue having many messages, will other queues also be affected? Or the queue having those messages is impacted?

Thanks & Regards

Amit

On Fri, Aug 20, 2021 at 2:06 AM Ken Peng <kenn...@gmail.com> wrote:

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAA1D6YumW0OEf6BeLdonraeh6kohHBUo0mme6vuPjs7_zbvpfg%40mail.gmail.com.

--