BBR internet drafts

[Neal Cardwell]

We just posted some internet drafts on BBR, to present at the next IETF in a few weeks, in Prague:

   https://tools.ietf.org/html/draft-cardwell-iccrg-bbr-congestion-control-00
   https://tools.ietf.org/html/draft-cheng-iccrg-delivery-rate-estimation-00

Comments/suggestions welcome.

Happy 4th of July!

neal

[David Sze]

Hi Neal, Yuchung,

I have a couple questions about the delivery rate estimation draft and its use in BBR.

1. Consider an application that has 3 distinct "phases":

(a) Sends a large amount of data (enough to get BBR out of Startup state)

(b) Alternates between long periods of idle time and sends of small amounts of data (e.g. 1xSMSS)

(c) Sends a large amount of data again

If I'm understanding correctly, none of the sends in phase (b) would be considered "app limited", since they don't satisfy the "<1xSMSS" condition.

Meanwhile, the delivery rate estimates during phase (b) could drastically underestimate the actual BtlBw.  e.g.

Real BtlBw: 10 Mbps

Real RtProp: 150 ms

@ t=0

P.size=1460

P.delivered=0

P.delivered_time=0

P.is_app_limited=0

@ t=150

rs.delivery_rate=(C.delivered - P.delivered)/(C.delivered_time - P.delivered_time)=(1460-0)/(0.15-0) = ~76 Kbps

Given a long enough phase (b), eventually BBR will update its BtlBw estimate to be equal to ~76 Kbps.

When phase (c) starts, the application throughput will behave something like "slow start", since the TargetCwnd and PacingRate will initially be based on the BtlBw of ~76 Kbps.

Is this behaviour known/intentional?  An application that loops between phases (b) and (c), where (b) is long but (c) is relatively short, will get sub-optimal throughput during (c).

Would it make sense to tighten the "app limited" constraints such that the small sends in (b) would be considered "app limited", so that BBR doesn't update BtlBw if the channel never gets filled?

2. Section 4.2 of the draft talks about ACK losses and BtlBw underestimation.  If ACKs are reordered/delayed, isn't BtlBw overestimation also possible?  e.g.

(a) @ t=0 (P1 and P2 transmitted)

P1.delivered=0

P1.delivered_time=0

P2.delivered=0

P2.delivered_time=0

(b) @ t=150 (ACK for P1 delayed, ACK for P2 received)

rs2.delivery_rate=(1460-0)/(0.15-0)

(c) @ t=150 (P3 transmitted)

P3.delivered=1460

P3.delivered_time=150

(d) @ t=300 (ACK for P1 received)

rs1.delivery_rate=(2920-0)(0.3-0)

(e) @ t=300 (ACK for P3 received)

rs3.delivery_rate=(4380-1460)/(300-150)

In step (e), the delivery_rate numerator includes the byte count for P1, but P1 actually traversed the bottleneck link long before t=150.  Meaning the numerator accounts for P1, but the denominator does not, resulting in an overestimate.

Is my understanding correct?  If so, is the assumption just that delays of this magnitude are rare enough in practice that it's OK to ignore them?

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neal Cardwell]

On Tue, Jul 25, 2017 at 2:08 PM, David Sze <sze....@gmail.com> wrote:
> Hi Neal, Yuchung,
>
> I have a couple questions about the delivery rate estimation draft and its
> use in BBR.
>
>
> 1. Consider an application that has 3 distinct "phases":
>
> (a) Sends a large amount of data (enough to get BBR out of Startup state)
> (b) Alternates between long periods of idle time and sends of small amounts
> of data (e.g. 1xSMSS)
> (c) Sends a large amount of data again
>
> If I'm understanding correctly, none of the sends in phase (b) would be
> considered "app limited", since they don't satisfy the "<1xSMSS" condition.

Actually, the sends in phase (b) would be considered "app-limited".
This is covered in section 3.4 of the draft:

https://tools.ietf.org/html/draft-cheng-iccrg-delivery-rate-estimation-00#section-3.4

The check in OnApplicationWrite() is executed "when the sending
application asks the transport layer to send more data". Since the app
is idle at the time of the write, "C.write_seq - SND.NXT < SND.MSS"
and all the other conditions are true, so the connection is marked as
application-limited.

We will try to update the draft to make it more clear that these
checks run before the data is enqueued in TCP, rather than after.

> 2. Section 4.2 of the draft talks about ACK losses and BtlBw
> underestimation. If ACKs are reordered/delayed, isn't BtlBw overestimation
> also possible? e.g.
>
> (a) @ t=0 (P1 and P2 transmitted)
>
> P1.delivered=0
> P1.delivered_time=0
> P2.delivered=0
> P2.delivered_time=0
>
> (b) @ t=150 (ACK for P1 delayed, ACK for P2 received)
>
> rs2.delivery_rate=(1460-0)/(0.15-0)

In TCP, ACKs are cumulative, so if a sender has received an ACK for P2
it knows that P1 was delivered as well. So this particular issue
should not be a problem, if I understand your scenario.

Also keep in mind that the app or congestion control will generally be
applying a filter to the delivery rate samples, so if there are
transient effects that cause the delivery rate samples to occasionally
be low, those samples should be ignored (e.g. see the BtlBw filter in
the BBR draft).

neal

[David Sze]

Hi Neal,

Thanks so much for the quick response - just a couple followups in-line:

On Tue, Jul 25, 2017 at 2:19 PM, Neal Cardwell <ncar...@google.com> wrote:

On Tue, Jul 25, 2017 at 2:08 PM, David Sze <sze....@gmail.com> wrote:

> 2. Section 4.2 of the draft talks about ACK losses and BtlBw
> underestimation.  If ACKs are reordered/delayed, isn't BtlBw overestimation
> also possible?  e.g.
>
> (a) @ t=0 (P1 and P2 transmitted)
>
> P1.delivered=0
> P1.delivered_time=0
> P2.delivered=0
> P2.delivered_time=0
>
> (b) @ t=150 (ACK for P1 delayed, ACK for P2 received)
>
> rs2.delivery_rate=(1460-0)/(0.15-0)

In TCP, ACKs are cumulative, so if a sender has received an ACK for P2
it knows that P1 was delivered as well. So this particular issue
should not be a problem, if I understand your scenario.

Section 3.3 talks about TCP SACK - doesn't the same scenario apply if you replace ACK with SACK in the above example?

 

Also keep in mind that the app or congestion control will generally be
applying a filter to the delivery rate samples, so if there are
transient effects that cause the delivery rate samples to occasionally
be low, those samples should be ignored (e.g. see the BtlBw filter in
the BBR draft).

Right, but the BtlBw filter in the BBR draft is a max filter, which is good for filtering underestimates.  In the above scenario it's an overestimate, which would sit in the filter for BtlBwFilterLen round trips.

[Neal Cardwell]

On Tue, Jul 25, 2017 at 2:40 PM, David Sze <sze....@gmail.com> wrote:

Hi Neal,

Thanks so much for the quick response - just a couple followups in-line:

On Tue, Jul 25, 2017 at 2:19 PM, Neal Cardwell <ncar...@google.com> wrote:

On Tue, Jul 25, 2017 at 2:08 PM, David Sze <sze....@gmail.com> wrote:

> 2. Section 4.2 of the draft talks about ACK losses and BtlBw
> underestimation.  If ACKs are reordered/delayed, isn't BtlBw overestimation
> also possible?  e.g.
>
> (a) @ t=0 (P1 and P2 transmitted)
>
> P1.delivered=0
> P1.delivered_time=0
> P2.delivered=0
> P2.delivered_time=0
>
> (b) @ t=150 (ACK for P1 delayed, ACK for P2 received)
>
> rs2.delivery_rate=(1460-0)/(0.15-0)

In TCP, ACKs are cumulative, so if a sender has received an ACK for P2
it knows that P1 was delivered as well. So this particular issue
should not be a problem, if I understand your scenario.

Section 3.3 talks about TCP SACK - doesn't the same scenario apply if you replace ACK with SACK in the above example?

Yes, I believe that scenario would apply if we replace ACK with SACK.

  

Also keep in mind that the app or congestion control will generally be
applying a filter to the delivery rate samples, so if there are
transient effects that cause the delivery rate samples to occasionally
be low, those samples should be ignored (e.g. see the BtlBw filter in
the BBR draft).

Right, but the BtlBw filter in the BBR draft is a max filter, which is good for filtering underestimates.  In the above scenario it's an overestimate, which would sit in the filter for BtlBwFilterLen round trips.

Right. In practice we do not see this kind of extreme reordering as a problem in the traffic we have seen.

That said, there can be over-estimates from just using the raw ACK rate, as discussed in the draft and in the presentation at IETF. The send_rate filter can filter out most of these cases. I have not worked through your example to see if the send_rate help in that particular case. That said, the current send_rate filter is simple, but not perfect - it misses some cases. But we are working on improving the filtering for delivery rate estimation. If you have specific ideas for improving it, please feel free to share them.

Thanks!

neal