On Sun, Jul 15, 2018 at 12:11 PM Jonathan Morton <
chrom...@gmail.com> wrote:
>
> > On 15 Jul, 2018, at 9:29 pm, 'Neal Cardwell' via BBR Development <
bbr...@googlegroups.com> wrote:
> >
> > A) tailoring BBR's ecn response to fq_codel's onset of CE.
> >
> > We are also working on BBR's response to ECN (specifically, adding such a response). There, too, it would be interesting to see traces showing the ECN mark patterns from your tests.
>
> Given that BBR currently disables advertisement of ECN capability (correctly, since it isn't),
It does now? It didn't used to, enabling ecn for tcp and turning on or
off bbr were entirely independent last i checked (and at the time, I
had hoped, they would become co-dependent until such time they were
both supported at the same time, so as to minimize user error). I'm
setting up some tests this week, will look into it.
>getting such traces may be tricky. Would ECN responses to other TCPs, such as NewReno and Cubic, be helpful? Or mangling ECT(0) into a BBR flow and watching the fireworks? (Since BBR should be able to adapt to the path bandwidth - eventually - without ECN assistance, that might not be as bad as it seems.)
It was a matter of hooking it up. There has been some published work
at the ietf on applying l4s's notions to BBR, but it was
insufficiently detailed to readily duplicate. Still:
https://datatracker.ietf.org/meeting/101/materials/slides-101-iccrg-bbr-congestion-control-with-l4s-support-02
Those results seemed terrible enough to try and tackle with something
attuned to fq-ing and codel-ling. Unlike some, I'd be totally happy
with some bandwidth sacrifice, 0 queuing delay (with fq_codel staying
in it's "new" flow phase consistently), and finding some way to
observe when it slips into "old flow" phase, to stay around that.
Since that's a truly tiny number of ms I imagine it's currently
measured in the "noise" that BBR currently rejects, but since we
obsoleted the field of LPCC and there's kind of a need for consistent,
if lower than max, bandwidth for many apps, I'd like to try.
I had three basic approaches in mind. 1) force a probeRTT phase on
receipt of CE 2) decrease the RTT estimate by somewhere between 5 and
20ms, reducing cwnd to suit something less agressive than cubic but
harsher than reno 3) observe the frequency of CE marks and drops in
general, and buffer depths, and observed RTTs, with various sorts of
tcp related traffic using flent's toolset.
Please note my principal objective is to beat up the "final" version
of sch_cake (after the 19 not so thoroughly evaluated revisions prior
to upstreaming last week) so I'm going to hit it with cdg, vegas,
cubic, and dctcp, and it's just that I'm sufficiently annoyed by BBR
lacking ecn still to want to wrap my hands around it and shake it hard
- but i do got other things on my plate and won't mind if someone
produces patches....
I was pleased to see a couple very good improvements to dctcp land in
net-next recently.
> In general, Codel's behaviour is as follows:
This is a nice summary below, although it's preaching to the largely
converted. :)
>
> - when packet sojourn in queue is less than target (typically 5ms), no marks are made.
>
> - when packet sojourn is observed over target *continuously* for at least the interval, the first mark is made. At this point the marking frequency is equal to the interval. Interval is typically 100ms, and is intended to match a prior estimate of the path RTT.
>
> - as long as packet sojourn remains above target, the marking frequency increases linearly per time. To achieve this, the interval between marks decreases on a inverse-square-root schedule per mark. The second mark will therefore, by default, occur about 70ms after the first.
>
> - when packet sojourn falls below target, marking ceases.
>
> - when packet sojourn remains below target, the marking frequency is reduced according to some rule. The rule for reference Codel is a bit non-intuitive and, I suspect, has not been carefully thought out. Cake's version decreases it linearly, by running the inverse-square-root-schedule in reverse (marks are scheduled but not actually made during this time); eventually it returns to the baseline interval.
One thing from netdevconf was the concept of ABC in the cellular
network, where permission to accellerate is granted by the aqm in the
middle. A lot of the data these folk are dealing with is flawed (they
*start* with traces that have 150-250ms delay already in them before
applying their "solutions") which makes me want to tune out...
https://netdevconf.org/0x12/session.html?congestion-control-for-cellular-wireless-networks
An idea towards "permission to accellerate" is to try and get into the
new/old flow bifircation fq_codel has. One means is "hard", I think -
as BBR seems (?) to pace a minimum of two packets at a time which will
tend to force things too often into the old flows before being acked
> - if packet sojourn rises above target very soon after it fell below, the delay before marking resumes, and the marking frequency, may therefore be shorter/faster than the initial condition. This helps Codel to adapt to RTTs shorter than estimated.
>
> The potential positions of marks can also be inferred from packet losses from the same AQM action on a non-ECT flow, which would result in SACKs in roughly the same pattern as ECE flags arriving at the sender. This is true because Codel both marks and drops packets from the head of the queue, immediately prior to delivery. In general, Codel drops non-ECT packets at the same times as it would mark ECT ones, but it delivers *some* packet in both cases; in case of a drop, it'll be the following packet in the same flow (ensuring fast recognition that a drop has occurred).
Um, er,
"in case of a drop, it'll be the following packet in the same flow
(ensuring fast recognition that a drop has occurred)"
is only true on a fq_codel derived algorithm. In the case of an aqm of
any sort, that does not do fq, it's anybody's guess as to what packet
from the flow will arrive next in time. (win!, for fq_codel/fq_pie,
for stability of rate and markings, again).
>If the queue is configured deeply, it won't show overflow loss as a FIFO would, and in a lab environment it's also possible to assert that no random loss occurs.
>
> An exception to the mark-loss equivalence rule is that Cake won't drop the last remaining packet in the queue, but might still mark it. Codel and fq_codel might still drop a tail packet, if it's been sitting in the queue long enough.
i note that (completely other, and often heated thread on the cake
list), that not dropping the last packet in a queue when things are
overloaded is *not* something I agree with at lower bandwidths. It is
better to force an RTO when things are that congested, and/or rely on
other packets already in the pipe to arrive to replace it, in order to
retain low queuing latency, store stuff "in the network and not the
queue". That observation is what led to fq_codel's current, deployed
"drop even if it's the last packet" behavior.
(I don't think that the BBR list is particularly the right place for
furthering this but I did want to make it clear that I think it's
controversial)
Also, since we got cake to 50gbit seeing what happens with > 1024
flows should be "interesting"
>
> - Jonathan Morton
>
--
Dave Täht
CEO, TekLibre, LLC
http://www.teklibre.com
Tel:
1-669-226-2619