When Prefetch-Size of Qos can be implemented?

[jeff...@lcecorp.com]

Hi,

We have a queue which has many messages, each time our consumer (a windows service) runs, all the messages in that queue will be in Unacked status and can only be consumed by that consumer, we would like be able to run multiple instance of that consumers to consume the messages faster, but since all the messages are in unacked status by the first one, all the later consumers couldn't get more messages.

We tried to set the prefetch-size to control the limit of the unacked messages, but we were told that parameter is not implemented by RabbitMQ, just want to know when that parameter can be implemented? Or is there any other ways to allow us scale out our consumer windows service to consume that queue faster?

Thanks,

[Luke Bakken]

Hello -

Prefetch is supported by RabbitMQ - https://www.rabbitmq.com/consumer-prefetch.html

[jeff...@lcecorp.com]

No, prefetch size is not implemented:

basic.qos

The server supports per-consumer and per-channel limits. The global flag is given different semantics from those in the specification. See consumer prefetch for more information. Prefetch size limits are not implemented.

https://www.rabbitmq.com/specification.html

The prefetch-size is the parameter to control unack messages size,  not prefetch-count.

[Luke Bakken]

Hi Jeff,

Thanks for pointing that out. I didn't realize that prefetch size (vs count) is part of the spec. I doubt that prefetch size will be implemented since prefetch count is what pretty much everyone should use anyway.

However, you want to use prefetch count (also referred to as "QoS") to accomplish what you're trying to do - distribute work among several consumers. Please see the "Work Queues" tutorial. This is a link to the Python version but others exist in different programming languages - https://www.rabbitmq.com/tutorials/tutorial-two-python.html

Thanks,

Luke

[jeff...@lcecorp.com]

Thanks Luck, appreciate that.

I have read that article, the prefetch-count only defines how many unacked messages deliver to the consumer, but it doesn't control how many messages get into the unacked status. By AMQP definition, the prefetch-size control that limits, right now we can only set it as "0" which means unlimited in RabbitMQ, so when my consumer service runs, all the Ready status messages will become Unacked status, so all the other consumers service come later couldn't get any messages.

That feature is very important for the consumer scale out, I want to ask the RabbitMQ development team when that parameter can be implemented or is there any other solutions for that purpose: only certain number of Ready messages become Unacked status, not all messages.

Thanks,

[Luke Bakken]

Hi Jeff,

Let's say you have a queue with 100 messages in the Ready state. You start 5 consumers, and each consumer sets a prefetch (QoS) count of 5. That means that 25 messages will be delivered (5 messages to each of the 5 consumers) and those 25 messages move to the Unacked state. The remaining 75 will be in the Ready state. When a consumer acknowledges a message, the message is removed from the queue and that consumer will be delivered another message out of the 75 in the Ready state. If the consumer acknowledges all 5 messages at once, they will be delivered another 5 since that is the value of their prefetch count.

This is my understanding of your use-case. If that's not correct, you'll have to clarify further.

Thanks,

Luke

[jeff...@lcecorp.com]

No, I did run the test. What happened is: no mater how many the prefetch-count is set, when the first consumer comes, all the 100 messages will become Unacked status and consumed by the first consumer, all the cosumers later will get no messages.

[Luke Bakken]

Hi Jeff,

You have an error in your code. If you can share your code, we may be able to assist. The scenario I described is a core use-case for RabbitMQ that we know works successfully.

Thanks,

Luke

[jeff...@lcecorp.com]

Thanks, Luke. One point might relate to that is: In my windows service, each of them will create a channel, and connect to the queue by that channel, if I run multiple of the consumer windows service, there will be multiple channels running at the same time and connected to the same queue, in that case,  I tried to make each of the connected channel can only get certain number of messages, but that couldn't done by prefetch-count. I hope that can be done by prefetch-size, but that parameter hasn't been implemented yet.

[Luke Bakken]

Hi Jeff,

Again, "get a certain number of messages" is exactly what prefetch count is for. At this point providing code would be the best way to determine what is happening.

Thanks,

Luke

[Michael Klishin]

Prefetch and QoS are documented in http://www.rabbitmq.com/confirms.html.

Consumer prefetch is rarely necessary and was primarily added to implement queue federation,

--

You received this message because you are subscribed to the Google Groups "rabbitmq-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rabbitmq-users+unsubscribe@googlegroups.com.
To post to this group, send email to rabbitmq-users@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

MK

Staff Software Engineer, Pivotal/RabbitMQ

[Arnaud Cogoluègnes]

As suggested by Luke, the code would be useful. A common mistake is to register the consumer and then set prefetch on the channel, this should be done the other way around.

On Tue, Apr 24, 2018 at 9:37 PM, Luke Bakken <lba...@pivotal.io> wrote:

--