bbr vs ecn

[Dave Taht]

BBR appears to be at least a bit broken with an ecn-enabled qdisc
trying to regulate things.

Here's graphs of a test with two simultaneous BBR flows, with ecn
enabled and not enabled, over a 48ms RTT, at 20Mbit. Queue depth
settles down nicely in the packet drop case to about 10 packets, the
ecn one works ok with one ecn flow, but goes to hell with 2, hitting
100 packets, and tons of marks that have no apparent affect.

I can supply a great deal more flent test data than this (I can put a
tarball somewhere).

--
Dave Täht
Let's go make home routers and wifi faster! With better software!
http://blog.cerowrt.org

[Neal Cardwell]

Hi Dave,

By design, BBR does not use ECN as a signal. We can discuss the design
rationale for that, but that's its own topic. :-)

For now, let's stick with your test.

In the current BBR design, when there are multiple BBR flows sharing
the bottleneck BBR bounds the queue to roughly the BDP. The BDP here
in this test is 20000000 / (1514*8) * .048 ~= 79 packets. And BBR
bounds the queue to 100 packets, which is roughly the BDP. So that
behavior is expected, based on the current design.

In our tests with the parameters you cite (2 flows, 48ms RTT, 20
Mbit/sec link) we see that the BBR flows coordinate well, maintain
their goal of bounding the queue to 1*BDP, and consequently packets
have a typical RTT of:

(two-way propagation delay) + (queue delay)
~= 110ms
~= 2*(two-way propagation delay)

In your e-mail you mention the queue depth "hitting 100 packets" for
the 2-flow ECN case, but the graph seems to show a number of packets
little over 300. Can you elaborate?

I am a little surprised that you are seeing such different behavior
with drops or ECN. BBR responds to the delivery rate and two-way
propagation delay, which, if I understand correctly, should be roughly
the same in your drop and ECN test cases. I will take you up on your
offer - if you would be able to place a tarball somewhere with packet
traces (just headers is fine), that would be great.

Thanks!

neal

> --
> You received this message because you are subscribed to the Google Groups "BBR Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dave Taht]

On Wed, Sep 21, 2016 at 7:35 AM, Neal Cardwell <ncar...@google.com> wrote:
> Hi Dave,
>
> By design, BBR does not use ECN as a signal. We can discuss the design
> rationale for that, but that's its own topic. :-)

I'm too blurry to take apart the captures, but if it isn't going to
pay attention to CE, it should (somehow) not negotiate on, when
enabled via sysctl, (if it isn't already - but seemed like it was
enabling it, but I'd need to look at the cubic+bbr caps, maybe it's
struggling to control the cubic). That would allow the aqm to drop at
the point of congestion, BBR flows.

Discussing what CE "should" do to BBR is something I presently have
insufficient coffee and BP medication for.

>
> For now, let's stick with your test.
>
> In the current BBR design, when there are multiple BBR flows sharing
> the bottleneck BBR bounds the queue to roughly the BDP. The BDP here
> in this test is 20000000 / (1514*8) * .048 ~= 79 packets. And BBR
> bounds the queue to 100 packets, which is roughly the BDP. So that
> behavior is expected, based on the current design.
>
> In our tests with the parameters you cite (2 flows, 48ms RTT, 20
> Mbit/sec link) we see that the BBR flows coordinate well, maintain
> their goal of bounding the queue to 1*BDP, and consequently packets
> have a typical RTT of:
>
> (two-way propagation delay) + (queue delay)
> ~= 110ms
> ~= 2*(two-way propagation delay)

Yep. Looks pretty good. How about aiming for 1/2 BDP? :-P

> In your e-mail you mention the queue depth "hitting 100 packets" for
> the 2-flow ECN case, but the graph seems to show a number of packets
> little over 300. Can you elaborate?

The graph with "300" was the number of ECN marks happening over the
sampling interval, not the number of packets. The other graph was the
number of backlogged packets.

It's easier for me if you just browse the data and plot types with flent. :)

I also stuck in that last test series outrageous amounts of buffering
(like pfifo_1000 with offloads) to see what would happen.

> I am a little surprised that you are seeing such different behavior
> with drops or ECN. BBR responds to the delivery rate and two-way
> propagation delay, which, if I understand correctly, should be roughly
> the same in your drop and ECN test cases. I will take you up on your
> offer - if you would be able to place a tarball somewhere with packet
> traces (just headers is fine), that would be great.

I'll get to caps later? I have had a long week... I disabled taking
caps early on as I was running low on disk space and didn't want to
permute the tests.

[Dave Taht]

I put 4 tests up, with captures, at

http://blog.cerowrt.org/flent/bbr-ecncaps.tgz

Each has 2 cubic and 2 bbr flows in them

ecn-cake-flowblind: single queue codel derived aqm with ecn enabled on the tcp
noecn-cake-flowblind: single queue codel derived aqm without ecn
enabled on the tcp
ecn-cake: fq'd aqm with ecn enabled on the tcp
noecn-cake: fq'd aqm without ecn enabled on the tcp

There are corresponding flent test data showing queue depth,
bandwidth, flow sharing, and other behaviors, like number of ecn
marks/drops, etc.

There's also some pics, like

http://blog.cerowrt.org/flent/bbr-ecncaps/backlog-ecn.svg
http://blog.cerowrt.org/flent/bbr-ecncaps/backlog-noecn.svg

I took the caps apart with tcptrace and xplot for a few minutes...

http://blog.cerowrt.org/flent/bbr-ecncaps/alotofcestonoeffect.png

[Kevin Bowling]

I'm curious, can you elaborate your thoughts on internal ECN, Internet ECN, and congestion control?

Regards,

Kevin Bowling - Systems Software Team - P: 480-227-1233

> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

The information in this message may be confidential.  It is intended solely for

the addressee(s).  If you are not the intended recipient, any disclosure,

copying or distribution of the message, or any action or omission taken by you

in reliance on it, is prohibited and may be unlawful.  Please immediately

contact the sender if you have received this message in error.

[Eric Dumazet]

Small reminder about ECN : linux does not implement rfc3540 , and so far I am not aware of any plan about this.

> To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "BBR Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bbr-dev+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dave Taht]

On Thu, Sep 22, 2016 at 1:50 PM, Kevin Bowling <kbow...@llnw.com> wrote:
> I'm curious, can you elaborate your thoughts on internal ECN, Internet ECN,
> and congestion control?

Those debates have raged across many, many mailing lists, for many years.

Of 6 times I've quit smoking, 4 of the times I resumed were triggered
by an ECN debate on the ietf AQM mailing list. I've had a piece long
in draft on my blog attempting to summarize the various viewpoints,
and inject mine, but I'm on my 7th attempt at giving up cigs at the
moment.

One point I've always tried to make is that "ECN has mass" - that a
preponderance of ECN marked packets can (and does) squeeze out other
valuable packets on that path, including (ultimately) very valuable
things like arp and routing packets. fq_codel with ecn enabled (by
default) - moderates that effect to such an extent that I can live
with it. Otherwise the natural game theory result is that everything
(voice, video, gaming, tcp, acks) will have to be ecn marked in order
to survive.

In other circumstances ECN is intensely desirable - uninterrupted
delivery of an iframe in a low latency videoconference/VR session
being my favorite example - and over very long, or very short RTT
links as another.

I would not mind if it was not a global option you set with sysctl but
something you selected via sockopt for those needy applications. ECN
is also intensely valuable under well-controlled circumstances.

ECN is the wet paint of the congestion control universe.

...

At the moment, given that BBR does not seem to (presently!) do
anything that I'd consider sane with CE marks, I'd merely like it to
not negotiate ecn at all - even when configured - to give an innocent
AQM along the path a chance to drop at its point of congestion. I plan
to rework my testbed so I can test both cubic with ecn and bbr
without, on a path with fq_codel/cake/pie on it.

In my early tests...

With ecn enabled, for both cubic and bbr competing, we get a packet
backlog that codel struggles wildly to control, hurting, and bbr
reverts to it's 1BDP mode.

http://blog.cerowrt.org/flent/bbr-ecncaps/backlog-ecn.svg

It gets so bad that ping comes to a halt, acks get starved out, etc.

http://blog.cerowrt.org/flent/bbr-comprehensive/bbr_ecn_eventually_starving_ping.png

(please note I can demonstrate this result with any ecn enabled but
non-ecn respecting sender, and it's why I gave up on single queue'd
AQM designs)

without ecn, things are much better:

http://blog.cerowrt.org/flent/bbr-ecncaps/backlog-noecn.svg
(that big spike is a later started cubic fighting it out with bbr

I've got caps and more data on this behavior in the bbr-ecncaps.tgz

Reading the tea leaves (fq_codel has a mysterious option for it) , I
suspect google's doing their own thing with ECN, and that's not in the
current BBR patch. I will probably lose endless hours of my life to
trying to divine what the "right thing" is, and be surprised and full
of admiration for it when it actually appears.

In the meanwhile:

There has been some promising progress forward in resolving the ecn
debates on the ietf tcp-prague mailing list. There's been another
project, which I've been rather dubious about, trying to get DCTCP to
work over the broader internet, called l4s. In both cases, and
possibly now, BBR's, they repurpose the ECT(1) bit to "do something
different", and now, everybody can go and debate how to use that
single bit remain to mean something meaningful, and I'll need to
nicotine patches before paying further attention.

>> > an email to bbr-dev+u...@googlegroups.com.

>> > For more options, visit https://groups.google.com/d/optout.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "BBR Development" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to bbr-dev+u...@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
>
>
> The information in this message may be confidential. It is intended solely
> for
> the addressee(s). If you are not the intended recipient, any disclosure,
> copying or distribution of the message, or any action or omission taken by
> you
> in reliance on it, is prohibited and may be unlawful. Please immediately
> contact the sender if you have received this message in error.
>

[Neal Cardwell]

On Wed, Sep 21, 2016 at 8:30 PM, Dave Taht <dave...@gmail.com> wrote:
>> For now, let's stick with your test.
>>
>> In the current BBR design, when there are multiple BBR flows sharing
>> the bottleneck BBR bounds the queue to roughly the BDP. The BDP here
>> in this test is 20000000 / (1514*8) * .048 ~= 79 packets. And BBR
>> bounds the queue to 100 packets, which is roughly the BDP. So that
>> behavior is expected, based on the current design.
>>
>> In our tests with the parameters you cite (2 flows, 48ms RTT, 20
>> Mbit/sec link) we see that the BBR flows coordinate well, maintain
>> their goal of bounding the queue to 1*BDP, and consequently packets
>> have a typical RTT of:
>>
>> (two-way propagation delay) + (queue delay)
>> ~= 110ms
>> ~= 2*(two-way propagation delay)
>
> Yep. Looks pretty good. How about aiming for 1/2 BDP? :-P

Yes, the current behavior where of creating a ~1*BDP queue when bulk BBR flows
share a bottleneck is something we are actively working on improving. The
current behavior is not the end goal, but rather the place where the current
code arrives due to some subtle trade-offs. We want the number of packets in
flight to be allowed to potentially be ~1*BDP beyond the pipe BDP, in order to
handle delays that are due to link-layer or receiver effects that cause
delayed, stretched, or aggregated ACKs (these are very common for wifi,
cellular, and cable systems). But ideally we don't want more packets in flight
if the delays are simply due to competing flows, since that can lead to the
~1*BDP of queue you can see in these tests. So we are working on improving the
behavior (reducing the queue) in the case with competing BBR flows.

neal

[chrom...@gmail.com]

On Monday, 26 September 2016 21:59:07 UTC+3, Neal Cardwell wrote:

We want the number of packets in
flight to be allowed to potentially be ~1*BDP beyond the pipe BDP, in order to
handle delays that are due to link-layer or receiver effects that cause
delayed, stretched, or aggregated ACKs (these are very common for wifi,
cellular, and cable systems).

This is good behaviour for a dumb FIFO at the bottleneck.  Those are still common.

However, when you start seeing CE marks, you can be *certain* that there is an AQM on the bottleneck.  The AQM typically knows more about the bottleneck link's characteristics and loading than the endpoints do, and has picked a target queue fill level with that in mind.  The correct response to CE, per RFC, is identical to packet loss - except that retransmission/recovery is not necessary.

I've just skimmed through the BBR Linux code to verify that that is also true for BBR, since BBR does scan for loss patterns indicating a "policer" or a short FIFO at the bottleneck.  The simplest way to add CE response would appear to be to add recognition of CA_EVENT_ECN_IS_CE in bbr_cwnd_event(), halting any window-growth phase and setting rs->losses.  I think this will be sufficient to make BBR RFC compliant wrt ECN.

 - Jonathan Morton