Per-Connection Flow Control -- The Case Against (Try #2)

[Chip Salzenberg]

The fundamental paradox of per-connection flow control is that it holds up the stop sign just when progress becomes possible.  It is backwards and unhelpful.  Consider:

1. A client is publishing 1.5K/sec to each of four exchanges, each of which has a queue.

2. There are no consumers.  Therefore the queue is growing.

3. RMQ does not stop this.  "Flow control" does not trigger.

4. The consumers appear to begin to tear down the backlog.

5. RMQ per-connection flow control suddenly decides that now there are some consumers, now it has a reason to throttle the sender.

6. If the consumers had not shown up, the producer would not have been blocked.

Therefore, it is only when the backlog can go DOWN that the broker decides to throttle the sender.  Not when the backlog was GROWING, but when it could be SHRINKING, that's when RMQ decides to stop accepting.

This is not acceptable.

[Pablo Molnar]

I'm also experiencing the flow control to be unhelpful using spring-amqp libraries. Is there a way to just turn it off?

Thanks,

Pablo

_______________________________________________
rabbitmq-discuss mailing list
rabbitmq...@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Chip Salzenberg]

An untested patch to disable it:

diff --git a/src/rabbit_reader.erl b/src/rabbit_reader.erl

index 5acf6ac..8b65821 100644

--- a/src/rabbit_reader.erl

+++ b/src/rabbit_reader.erl

@@ -368,7 +368,7 @@ terminate(_Explanation, State) ->

 control_throttle(State = #v1{connection_state   = CS,

                              conserve_resources = Mem}) ->

-    case {CS, Mem orelse credit_flow:blocked()} of

+    case {CS, Mem} of

         {running,   true} -> State#v1{connection_state = blocking};

         {blocking, false} -> State#v1{connection_state = running};

         {blocked,  false} -> ok = rabbit_heartbeat:resume_monitor(

Pablo

rabbitmq-discuss@lists.rabbitmq.com
https://lists.rabbitmq.com/cgi-bin/mailman/listinfo/rabbitmq-discuss

[Reverend Chip]

On 5/26/2012 1:08 AM, Tony Garnock-Jones wrote:
> On 25 May 2012 18:57, Chip Salzenberg <rev....@gmail.com
> <mailto:rev....@gmail.com>> wrote:
>
> This is not acceptable.
>
>
> Why not? I'd be interested to learn what problems this is causing for you.

That's very funny.

If you're serious, then I can't take you seriously.

[Reverend Chip]

On 5/29/2012 6:21 AM, Tony Garnock-Jones wrote:

On 28 May 2012 19:12, Reverend Chip <rev....@gmail.com> wrote:

That's very funny.

 If you're serious, then I can't take you seriously.

I suspect you're just being rude....

I explained a scenario, as produced by the current flow control code, that is ridiculous on its face: That growing queues are fine, until consumers appear, at which point growing queues are not fine.  That's a paradoxical result that at the very least should prompt reevaluation of the flow control algorithm.  To ask what the practical bad results are in my specific case is ridiculous misdirection.

Either growing queues (up to disk and memory limits) are acceptable -- my position, and the position of RMQ without consumers -- or growing queues are not acceptable -- the position of RMQ with consumers.  This paradox is quite simple to write and to read.  The behavior of the current code is absurd, as anyone can see if they were to choose to look.

So no, I was not being rude, specifically.  But I did ridicule your comment, which well deserved it.  Per Thomas Jefferson: "Ridicule is the only weapon which can be used against unintelligible propositions. Ideas must be distinct before reason can act upon them."

PS: You may not have thought about the further wrinkle that the queues do not accept new messages until they are EMPTY.  Which is doubly ridiculous.

, but since it's also possible that the message you're replying to didn't get through to you in its entirety (because of broken "From" quoting at some stage in the mail delivery pipeline), I'll reproduce the rest of what I wrote here in case it helps advance the discussion:

 

>From one point of view, any time consumers exist but are not keeping up with offered traffic is a good time to throttle the producers. Perhaps call this the "steady state" point of view - I'm unsure if it has a better name already.

>From another point of view - call this the "bursty" point of view, perhaps - the growth of queues is perfectly acceptable, whether consumers exist at the time or not.

ISTM Rabbit makes a guess about whether the application is "steady state" or "bursty", choosing "steady state" all the time. Rabbit team, is this the case? Is there a way of configuring the flow control to permit unbounded queue growth to support the "bursty" case? Or do I have the wrong end of the stick entirely?

 

1. A client is publishing 1.5K/sec to each of four exchanges, each of which has a queue.

2. There are no consumers.  Therefore the queue is growing.

3. RMQ does not stop this.  "Flow control" does not trigger.

4. The consumers appear to begin to tear down the backlog.

5. RMQ per-connection flow control suddenly decides that now there are some consumers, now it has a reason to throttle the sender.

6. If the consumers had not shown up, the producer would not have been blocked.

7. The consumers drain some of the messages from the queues.

8. The producers are unblocked.

9. Profit!!!

Regards,

  Tony

--
Tony Garnock-Jones
tonygarn...@gmail.com
http://homepages.kcbbs.gen.nz/tonyg/