Can we stop flow control using lazy queue?

[Musfiqur Rahman]

Hi,

We are using RabbitMQ 3.6.6. We have ~200 exchanges, ~800 queues in 2 servers(128 GB Ram,2.5 TB Disc). We have observed that sometimes our queue goes into "flow" mode during heavy load(too many messages from producers). This makes the queue unresponsive. Sometime it even destabilizes the full cluster. Some of us are thinking of moving to Kafka so that producer can happily publish message without being blocked and also not destabilizing the system. But I would prefer a solution with RabbitMQ itself. 

My question is: Will lazy queue solve/alleviate the issue?

-Musfiqur     

[Lutz Horn]

This version of RabbitMQ was released on Aug 19, 2016. You will probably not get any support for such an old version.

Lutz

---

Von: Musfiqur Rahman
Gesendet: ‎22.‎01.‎2020 16:32
An: rabbitmq-users
Betreff: [rabbitmq-users] Can we stop flow control using lazy queue?

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/f7cc47fc-4e6c-411f-9f95-1a7d3cae45d4%40googlegroups.com.

[Musfiqur Rahman]

Hi,

I am actually asking a question.If we upgrade and start using lazy queue will the problem be solved?

-Musfiqur

On Wednesday, January 22, 2020 at 10:38:37 AM UTC-5, Lutz Horn wrote:

This version of RabbitMQ was released on Aug 19, 2016. You will probably not get any support for such an old version.

Lutz

---

Von: Musfiqur Rahman
Gesendet: ‎22.‎01.‎2020 16:32
An: rabbitmq-users
Betreff: [rabbitmq-users] Can we stop flow control using lazy queue?

Hi,

We are using RabbitMQ 3.6.6. We have ~200 exchanges, ~800 queues in 2 servers(128 GB Ram,2.5 TB Disc). We have observed that sometimes our queue goes into "flow" mode during heavy load(too many messages from producers). This makes the queue unresponsive. Sometime it even destabilizes the full cluster. Some of us are thinking of moving to Kafka so that producer can happily publish message without being blocked and also not destabilizing the system. But I would prefer a solution with RabbitMQ itself. 

My question is: Will lazy queue solve/alleviate the issue?

-Musfiqur     

--
You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitm...@googlegroups.com.

[Luke Bakken]

Hello,

There's really no way we can answer that question with the information provided. Do you monitor your system? Are you hitting memory alarms? Are your consumers keeping up with producers? Anything in the logs? You mention a cluster, but not how many nodes nor if your queues are mirrored.

What you should do is ensure your publishers are using publisher confirms correctly and have them rate-limit themselves based on the confirm latency over time. If the latency is increasing, you know load is increasing and the publishers can scale back.

We have a tutorial in Java that demonstrates how to use publisher confirms correctly:

https://www.rabbitmq.com/tutorials/tutorial-seven-java.html

In addition, the code in the PerfTest tool also implements a window of outstanding confirms. It does not throttle itself back based on latency, you would have to write that code:

https://github.com/rabbitmq/rabbitmq-perf-test/

Thanks,

Luke

[Ayanda Dube]

With such memory at hand, I suggest you also consider bumping up your credit-flow[1] settings from default {400,200} by factor of, for example 3, to {1200,600}.  This should allow

your queues to sustain/accept more load before getting into flow. Also use PerfTest as suggested to find the precise setting for your case.

[1] https://www.rabbitmq.com/blog/2015/10/06/new-credit-flow-settings-on-rabbitmq-3-5-5/

Best regards,

Ayanda

Erlang Solutions Ltd.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ff927119-56fc-4ee4-b0f8-06066de3e787%40googlegroups.com.

Code Sync & Erlang Solutions Conferences

Lambda Days - Kraków: 13-14 February 2020

Code BEAM SF - San Francisco: 5-6 March 2020

Code BEAM Lite ITA - Bologna: 6 April 2020

ElixirConf EU - Warsaw: 29-30 April 2020

Code BEAM STO - Stockholm: 28-29 May 2020

Erlang Solutions cares about your data and privacy; please find all details about the basis for communicating with you and the way we process your data in our Privacy Policy. You can update your email preferences or opt-out from receiving Marketing emails here.

[Musfiqur Rahman]

Thanks for the information. It is little complicated. I see in the description of lazy queue, it says that it can handle millions of messages in queue. How can you explain that with the flow-control not being active?

To my understanding, the producers in a persistent queue will be blocked (in a flow state) if the consumers are down/slow. How does being lazy helps in that case?

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ff927119-56fc-4ee4-b0f8-06066de3e787%40googlegroups.com.

[Ayanda Dube]

Lazy queue is known to handle more/millions of messages from it's significantly lower memory utilization during operation. See the numbers [1].

So basically publishing blocked due to your Rabbit instances running out of memory will take much longer to be experienced with a lazy queue than with a default queue.

But. Since they use more disk resources than RAM during operation, publishing could still be eventually blocked if you run out of disk space - with consumers being unavailable

for prolonged periods of time. But disk is cheaper and That's how lazy queues help - with memory utilization and when/when not publishers are blocked. They don't help in the context of flow-control. That's now something else which applies regardless of whether the queue is lazy or not.

Flow control determines/controls how many messages RabbitMQ's internal messaging components on the critical path can handle before they throttle / enter into "flow" state to avoid

being overwhelmed by, e.g. a burst of traffic outpacing the rate of message processing. Regardless of whether consumers are available or not, if a queue can't process messages fast

enough to keep up with the incoming traffic, it's likely to throttle and enter into that "flow" state. Lazy queues won't protect you from this (actually, they operate slower and could even

enter into "flow" faster than default queue under the same conditions). Traversal of these inflight messages on the critical path is in RAM, in the erlang process message queues[2] before

they are queued on disk/ram if you're persisting and/or using lazy queues. 

As mentioned in previous response, you control "flow" with credit-flow configuration. This would be in the advanced.config[3] file as follows:

{credit_flow_default_credit, {400, 200}}

The more RAM you have the more internal messaging components can handle, allowing you leeway to bump up these defaults. Model your traffic with PerfTest to get the ideal and precise

settings for these credit configs for an acceptable QoS before "flow" is engaged, if ever.

[1] https://www.rabbitmq.com/lazy-queues.html#performance

[2] https://erlang.org/doc/getting_started/conc_prog.html#message-passing

[3] https://github.com/rabbitmq/rabbitmq-server/blob/master/docs/advanced.config.example

Best regards,

Ayanda

Erlang Solutions Ltd.

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/ffafac8e-721a-442a-bb17-c424ebe79227%40googlegroups.com.

[Michael Klishin]

Mustiqur,

Your question is a bit like showing up to a doctor and telling them "if I take this pill, would the pain go away?"

without ever mentioning any symptoms.

Lazy mode can reduce paging pauses and/or GC activity of a given queue but we have no idea what's the bottleneck.

In addition, first release of lazy queues actually increased GC churn unintentionally. I don't remember what version

that was addressed in but even 3.6.16 has been out of support since May 201 [4]8. You'd have to dig in the change log [1]

to find that out. We offer no guidance for 3.6 at this point.

A much better option is to consider upgrading and if your queues are durable, trying quorum queues which have some downsides

but generally offer a lot more predictable resource usage [2].

Having extensive monitoring is essential in understanding how your system behaves and why. For all we know it can be disk I/O

that's the bottleneck, it is certainly not unheard of [3].

Lazy queue mode can be enabled and disabled at runtime, so it's really easy to try then.

When in doubt, do your own benchmarks and see what monitoring data tells you.

1. https://www.rabbitmq.com/changelog.html

2. https://www.rabbitmq.com/quorum-queues.html

3. https://www.rabbitmq.com/monitoring.html

4. https://www.rabbitmq.com/versions.html

To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/2be7c092-7b33-4308-bbe6-3593ed689616%40googlegroups.com.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Musfiqur Rahman]

Hi,

Thanks for the information. Can you pls confirm either of the the following in general?

- There is no way in rabbitmq that can decouple the publishers' speed and consumers' speed because of the inherent flow control.

- In a persistent queue, the limit of the publisher is determined by disk and memory. In a lazy queue, it is blocked by the disk only. There may be other consumers but that will only slow down the connection between queue and that particular consumer. the queue will go into a flow state if either if the connection is blocked.

-Musfiqur

On Jan 24, 2020, at 11:31 AM, Michael Klishin <mkli...@pivotal.io> wrote:



You received this message because you are subscribed to a topic in the Google Groups "rabbitmq-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rabbitmq-users/xKUXWnRU5eo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rabbitmq-user...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/CAGcLz6VZwUrEnAns1sDYa%2B-zhDzztLEUY14sx9OqVgoKg6khHw%40mail.gmail.com.

[Michael Klishin]

There are ways to decouple publisher throughput from that of the consumer *for a period of time*, and lazy queues were introduced for that particular reason.

The behavior of disk paging is more complex than a single sentence may explain. Quorum queues and classic queues in lazy mode are primarily

bound by disk I/O but that's not the only factor. Classic queues have to do a fairly involved balancing due to a protocol feature where clients can dictate whether

a message should be kept in RAM or not (the transient property). The docs provide a general overview [1] with a few key configurable elements.

In the long term if your publishers consistently outpace consumers then you either have to use queue length limits or message TTL, or you will run out of resources,

with any messaging tool.

1. https://www.rabbitmq.com/memory.html

To view this discussion on the web, visit https://groups.google.com/d/msgid/rabbitmq-users/8FE460D7-49E2-4E60-B709-ADF03D69F9DA%40gmail.com.