Dynamically vary prefetch counts

[Alfredo Di Napoli]

Hello folks!

First of all I wanted to seize the chance to thanks the team & community behind RabbitMQ: I think it's a truly impressive piece of software.

Now let's get to the meat of my question:

At work I have a cluster doing video transcoding, with different Rabbit queues tailored to job sizes (e.g. 1 queue for vids < 25mb, 1 queue for vids > 25 && < 150mb, etc).

We rely on Amazon's Auto Scaling Group (ASG) feature to handle scaling of the cluster according to the CPU utilisation on each machine.

In order to avoid loading each machine of my cluster beyond a capacity it can handle, I have empirically derived certain prefetch counts to assign to each of my

queues; example: a machine cannot take simultaneously more than 2 jobs > 500MB.

Of course my aim would be to maximise the CPU as well the overall utilisation of each machine in a way the money we are paying to Amazon would be well spent.

This is notoriously difficult to do and it's an open research area:

http://web.stanford.edu/~cdel/2014.asplos.quasar.pdf

What I'm asking is somewhat orthogonal: pretend I have a super algorithm which can reply to the questions "Can this machine accept more load?" and "Is this machine bearing too much load"? What I would like to do is to tweak the prefetch counts accordingly - and when the system is running! - to let the system reach an optimum. We are of course talking about "local"

ones and not "global" ones (aka shared for all consumers).

My questions for you folks:

1. Despite in the docs in not explicitly mentioned, is it possible to vary dynamically the prefetch count of a queue? Do you think this is a good idea at all?

   - If yes, will Rabbit enforce its consistency properties to make sure no jobs will get lost during the "shrinking"/"expansion"?

   - If no, do you guys have experience in this area and can give me pointers on how I could solve my problems in an alternative way?

Thanks a lot!

Alfredo

[Michael Klishin]

On 16 March 2015 at 16:42:13, Alfredo Di Napoli (alfredo....@gmail.com) wrote:
> My questions for you folks:
>
> 1. Despite in the docs in not explicitly mentioned, is it possible
> to vary dynamically the prefetch count of a queue? Do you think
> this is a good idea at all?

Prefetch is configured per-channel. It's a reasonable  idea. Keep in mind that
scaling up is trivial but down is a lot trickier: if you already have N messages
delivered and pending acknowledgement, and change prefetch to M where M < N,
existing consumer load won't go away until your consumers are sophisticated
enough to get a notification of some sort from your scaling tool and drop some work.

> - If yes, will Rabbit enforce its consistency properties to make
> sure no jobs will get lost during the "shrinking"/"expansion"?

There is a race condition between basic.qos being sent to the server and incoming deliveries
(if channel is allowed to send more messages to consumers). That and the above are perhaps the
only non-obvious cases.

> - If no, do you guys have experience in this area and can give me
> pointers on how I could solve my problems in an alternative way?

Prefetch can be altered by apps largely for the reason you're describing.
--
MK

Staff Software Engineer, Pivotal/RabbitMQ

[Alfredo Di Napoli]

Hey Michael,

thanks for the blazing fast reply, it's pure gold!

Alfredo