Subject: Re: [tcpm] [tcpPrague] L4S status update

From: hiren panchasara <[email-masked]>
Date: Tue, 1 Nov 2016 15:36:22 -0700

Hi Bob,
A tiny correction in-line:

On 11/01/16 at 12:02P, Bob Briscoe wrote:
[snip]
>   * Data Centre TCP (DCTCP) for
>       o Linux (in the mainline kernel <https://www.kernel.org/>)
>       o FreeBSD patch
>         <http://lists.freebsd.org/pipermail/freebsd-net/2014-February/037915.html>

https://svnweb.freebsd.org/base?view=revision&revision=r277054
It's been integrated in mainline FreeBSD for quite some time now. :-)

Thanks for the whole writeup, btw.

Cheers,
Hiren

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Mon, 21 Nov 2016 15:27:57 +0100

Hi Bob and all,

see below.

Am 01.11.2016 um 01:02 schrieb Bob Briscoe:
> A few people have been working away to specify and document all the
> aspects of the new Low Latency, Low Loss, Scalable throughput (L4S)
> service, which held a successful BoF in Berlin. As the decision was to
> try to work across multiple WGs, I thought it would be useful to give ...

Thanks for putting this together.

>   * Dual Queue Coupled AQM
>       o With Curvy RED for Linux (access available shortly)
>       o With PI2 for Linux <https://github.com/olgabo/dualpi2> [*UPDATED*]

I'll repeat my concerns that I already expressed at the L4S BOF in Berlin:

While I agree that we probably need to separate low-delay congestion
control schemes from traditional "queue-filling" congestion schemes,
I strongly suggest to avoid putting a congestion control-specific
coupling scheme into the network (a classic case for applying the
"end-to-end arguments in system design").
The current Dual queue coupled AQM proposal has got a coupling based on
a congestion control specific dropping law p_C=(p_L/2)². So if
congestion control schemes change then this coupling needs to be
adapted. For example, the currently proposed scheme may fail if that
vast majority of TCP traffic is using BBR other some other forthcoming
CC scheme instead of Cubic, Reno, Compound etc. The same applies to
draft-briscoe-tsvwg-ecn-l4s-id, section 2.5, where the dropping
likelihood is defined.

Regards,
 Roland

---

From: Ingemar Johansson S <[email-masked]>
Date: Tue, 22 Nov 2016 14:35:05 +0000

Hi Roland + others.

As regards to comments around other new congestion control algorithms and that they may need adapted dropping likelihood relation between a classic queue and L4S queue.
I have not tried out but I suspect that BBR may get an unfair treatment, at the same time it is possible that other delay based CCs may suffer. 

They question however if this is a major problem?, one may as well see this as an incentive to switch over to scalable congestion controls and L4S ? There is after all no requirement to stick to a particular congestion control no matter what. ?

The question whether or not endpoint dependency should be built into the networks is of course  a valid question but given that the state of the art congestion controls like Reno and Cubic rely on a 1/sqrt(p) function then that is perhaps OK ?. 
There will for a foreseeable time come updated endpoint based congestion control algorithms that are optimized  for one thing or the other (I am guilty too :-). However if one can distinguish between two classes (classic and L4S) where classic belong to the 1/sqrt(p) family then I believe that it is possible to solve the problem. If we try find a solution where classic = "all sorts of non-scalable non-L4S dependent congestion controls" then I believe that we easily end up in big problems.

/Ingemar

> -----Original Message-----
> From: Bless, Roland (TM) [mailto:[email-masked]]
> Sent: den 21 november 2016 15:28
> To: Bob Briscoe <[email-masked]>; tsvwg IETF list <[email-masked]>;
> TCP Prague List <[email-masked]>; AQM IETF list <[email-masked]>; tcpm
> IETF list <[email-masked]>
> Subject: Re: [tcpPrague] [aqm] L4S status update
> 
> Hi Bob and all,
> 
> see below.
> 
> control schemes from traditional "queue-filling" congestion schemes, I
> strongly suggest to avoid putting a congestion control-specific coupling
> scheme into the network (a classic case for applying the "end-to-end
> arguments in system design").
> The current Dual queue coupled AQM proposal has got a coupling based on a
> congestion control specific dropping law p_C=(p_L/2)². So if congestion
> control schemes change then this coupling needs to be adapted. For
> example, the currently proposed scheme may fail if that vast majority of TCP
> traffic is using BBR other some other forthcoming CC scheme instead of
> Cubic, Reno, Compound etc. The same applies to draft-briscoe-tsvwg-ecn-
> l4s-id, section 2.5, where the dropping likelihood is defined.
> 
> Regards,
>  Roland
>

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Tue, 22 Nov 2016 17:45:57 +0100

Hi Ingemar,

my point was that it's probably better to refrain from
building CC-specific behavior into network elements as the
CC algorithms may evolve faster and in more flexible ways
than we can foresee. Thus, it would be good to have a separation
(or coupling) scheme that actually doesn't depend on the
1/sqrt(p) dropping law.

Am 22.11.2016 um 15:35 schrieb Ingemar Johansson S:
> As regards to comments around other new congestion control algorithms
> and that they may need adapted dropping likelihood relation between a
> classic queue and L4S queue. I have not tried out but I suspect that
> BBR may get an unfair treatment, at the same time it is possible that
> other delay based CCs may suffer.

It would be interesting to see what happens if BBR is sorted into
the L4S queue in comparison to what happens if BBR is sorted into
the classic queue (BBR isn't reacting to loss according to 1/sqrt(p)).

> They question however if this is a major problem?, one may as well
> see this as an incentive to switch over to scalable congestion
> controls and L4S ? There is after all no requirement to stick to a
> particular congestion control no matter what. ?

Yes, and that's why I find that built-in coupling law too limiting.

> The question whether or not endpoint dependency should be built into
> the networks is of course  a valid question but given that the state
> of the art congestion controls like Reno and Cubic rely on a
> 1/sqrt(p) function then that is perhaps OK ?. There will for a
> foreseeable time come updated endpoint based congestion control
> algorithms that are optimized  for one thing or the other (I am
> guilty too :-). However if one can distinguish between two classes
> (classic and L4S) where classic belong to the 1/sqrt(p) family then I
> believe that it is possible to solve the problem. If we try find a
> solution where classic = "all sorts of non-scalable non-L4S dependent
> congestion controls" then I believe that we easily end up in big
> problems.

I'm not sure. Maybe we have a class of "queue-filling" CCs and
a class of low-delay targeted CCs.

Regards,
 Roland

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 22 Nov 2016 19:09:39 +0000

Roland,

I share your concern about cc-specific AQMs. But that is not a good 
characterization of what we're doing.

On the current Internet, everything is meant to be somewhat "friendly" 
to the original TCP cc (now spec'd in RFC5681). All sorts of cc's work 
alongside that, with slightly different "fairness" properties, and only 
one AQM is needed to cover them all. Nonetheless, Reno is the "lamest", 
so everyone has to try to "Do (not much) Harm" to the lamest.

*Is any AQM CC-neutral?**
*Note rule 5 <https://tools.ietf.org/html/rfc7567#section-4.5> in the 
AQM Guidelines [RFC7567]
       "AQM algorithms SHOULD NOT interpret specific transport protocol 
behaviors."
In general, the advice in that section is sound, but I don't think we 
realized at that time just how subtle this issue is.

Since then, I discovered that the autotuning parameter table in the PIE 
algorithm is designed very precisely around the 1/sqrt(p) rule of Reno 
(see Fig 5 in the PI^2 paper <http://www.bobbriscoe.net/pubs.html#PI2>). 
Similarly, the sqrt control law in Codel claims to be dependent on Reno 
{Note 1}.

The point is that these AQMs still work fine with Cubic, Compound, 
Westwood, etc, because all these ccs were designed to interwork with 
Reno. {Note 2}

The idea of L4S (and specifically the DualQ Coupled AQM and the L4S ID 
spec) is to enable a shift to a completely different "norm", but still 
coexist with all the 'Classic' cc's that coexisted around the old 
"norm". The new norm is intended to be just as fuzzy as the old norm 
{Note 3}. The idea is two fuzzy clouds of congestion controls, around an 
old and a new norm that are related together.

*BBR**
*I believe BBR attempts to be 'friendly' to loss-based flows when 
competing in the same queue. But it's still research, and we don't yet 
know how good it is at that in all scenarios, although we do have code 
to test now. Given BBR currently sets Not-ECT, it would classify itself 
into the Classic queue of a DualQ AQM, and if it coexists with Reno it 
/should/ coexist with L4S traffic in the other queue. See Koen's recent 
posting 
<https://www.ietf.org/mail-archive/web/tsvwg/current/msg14771.html> 
about this.

There would be nothing to stop someone designing a variant of BBR that 
coexisted in the L4S queue with Scalable CCs like DCTCP (the point being 
that if the bottleneck was not DualQ it would keep delay low and if if 
the bottleneck was DualQ it would benefit even more from the lower 
queuing delay there). However, it would have to be a bit more careful 
about its whole round trip of queue probing, to avoid increasing the 
delay in the L4S queue. You'll see that I suggested to Neil Cardwell 
that they consider probing with a few packets rather than a whole 
window, e.g. the chirping 
<http://www.bobbriscoe.net/pubs.html#chirp_impl> technique that Mirja 
and I looked into back in 2010 was designed to find the same knee 
between rate increase and delay increase, with far fewer packets. I 
thought of a better way of using chirping a few weeks ago, so I will be 
returning to that too.

*Specs**
*There is no statement that all L4S cc's MUST adhere to a 1/p rule. The 
L4S ID draft says:
   "The inverse proportionality requirement above is worded
    as a 'SHOULD' rather than a 'MUST' to allow reasonable flexibility
    when defining these specifications."

I hope that 'SHOULD' is fuzzy enough - I suspect adding more words would 
make it less fuzzy. But I would welcome wording to make it even more 
fuzzy if you would like to engage in wordsmithing.

Nonetheless, we are trying to steer a path between a rock and a hard 
place. Because, to shift to a much calmer waters beyond the rocks, we 
have to define some number to relate L4S to Classic. I am wary because 
when ECN was specified, there were attempts to define ECN as different 
to drop. However, ECN originally ended up "the same as drop" because 
no-one could muster enough backing behind any particular number to 
relate the two, so the number '1' won by default (ie. ECN = 1 * drop^1 ).

Bob

{Note 1} I have never got a good answer to my questions on aqm@ietf as 
to why a sqrt that controls the shrinkage of the spacing between dropped 
packets has something to do with the steady state law of Reno, 
particularly because the law leads to linear growth in p over time.

{Note 2} Actually, I don't believe PIE would work that well with Cubic 
at v high BDP, once it was far from Reno-friendly, but that's only 
intuition from the stability analysis, not from actual testing.

{Note 3} Indeed, at the moment, when DCTCP is on its own in the L4S 
queue of the DualQ AQM as coded now, it hits up against a step 
threshold, which makes it behave as 1/p^2, not 1/p. For now, that's just 
because we didn't want to change too much about DCTCP at one time. But 
it's also got some nice properties. This will all need to be discussed 
as the DualQ AQM is specified more deeply.

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 22 Nov 2016 22:37:20 +0200

> On 22 Nov, 2016, at 21:09, Bob Briscoe <[email-masked]> wrote:
> 
> {Note 1} I have never got a good answer to my questions on aqm@ietf as to why a sqrt that controls the shrinkage of the spacing between dropped packets has something to do with the steady state law of Reno, particularly because the law leads to linear growth in p over time.

If you have intervals between events which follow a 1/sqrt(N) sequence, where N is the number of preceding events, you get an event frequency which increases linearly with time.  This applies directly to Codel’s signalling strategy, which is to start at one mark per (assumed) RTT, and to increase the marking frequency if that was insufficient to control the queue.

When you have multiple Reno flows sharing a single queue, there is a sqrt(N) factor in several of the characteristics, where N is the number of flows.  When such a shared link becomes saturated, all of the flows must be signalled to slow down, but for stochastic reasons it’ll probably take more than N signalling events to do so.  Increasing the signalling frequency while the queue remains insufficiently controlled has a good chance of quickly finding all the flows, while dropping relatively few packets (of non-ECN flows).

I’m reasonably convinced that Codel is a near-optimal solution to congestion signalling on TCP-friendly flows.  Regrettably, the latter are not the only type of traffic actually found on the Internet.

Really, the central assumption of Codel is that each flow requires only one congestion signal event per RTT to cause it to back off.  Codel stops working well on traffic which doesn’t obey that assumption (a linear increase in drop frequency is inadequate to mitigate a flood - you need to work with drop probabilities for that), but it *does* work acceptably well with multiple flows sharing a queue, due to this operating-point search.

 - Jonathan Morton

---

From: Mario Hock <[email-masked]>
Date: Thu, 24 Nov 2016 17:57:14 +0100

Hello Bob and Roland,

I followed your discussion and want to share my opinion, here. (Comments 
inline).

> Westwood, etc, because all these ccs were designed to interwork with

The reason that the AQMs also work with these CCs is because the CCs 
still have a lot in common with TCP Reno, like that they use a 
congestion window, that they increase the CWnd up until packet loss etc. 
Also the 1/sqrt(p)-law is still, more or less, applicable to them.

For newer CCs like BBR or PCC, the 1/sqrt(p) does not apply. They are 
based i.a. on throughput vs. loss/delay probing. BRR, for example, does 
not even use a congestion window (at least not in the same way as the 
other CCs do).

> old and a new norm that are related together.

I agree with the general idea. But from that reasoning newer CCs, like 
BBR, belong into the "new" queue, where L4S lives in!

If they don't belong in the same queue as L4S, then the queues should 
not be coupled in the way you propose. This coupling prevents any 
innovation in the non-L4S queue.

> thought of a better way of using chirping a few weeks ago, so I will be

I think L4S is not studied well enough, yet to be standardized in a way 
that other low delay approaches must adjust to the L4S concepts.

There is at least BBR, PCC, nv (New Vegas) currently under development 
that all bring in fresh ideas how congestion control could be made in 
the future. Standardizing L4S as "the" new concept without intensively 
study other approaches is just too soon.

If we want to act now, we should focus on a solution that enables new 
approaches in congestion control but is not tailored to either 
Reno/Cubic or L4S/DCTCP. Otherwise more research is required how we want 
the future of congestion control look like.

Best regards,

Mario Hock

---

From: "Bless, Roland (TM)" <[email-masked]>
Date: Thu, 24 Nov 2016 21:48:02 +0100

Hi Bob,

see comments inline, please.

> characterization of what we're doing.

Yep, but it's part of what you're proposing.

> so everyone has to try to "Do (not much) Harm" to the lamest.

Yes, and that's actually a deployment problem we agree on. Congestion
control (CC) innovation is obstructed by the old loss-based CC so some
separation mechanism would be good to let room for low-delay CCs.

> realized at that time just how subtle this issue is.

I think that AQMs could be in fact designed CC neutral to a large
extent, but every CC reacts probably different to the drop/ECN signals.
So the dropping strategy implemented by the AQM will always cause a
CC-specific reaction. Some AQMs therefore may achieve better results if
they are tuned to specific CC behavior (but my e2e argument would also
apply here), however, right now it's not that easy to change AQM
implementations, esp. those implemented in hardware.

> old and a new norm that are related together.

Yes, and I support that basic idea to introduce network support for
separating flows with different CC schemes. However, I'd like to have a
solution that doesn't build in a specific coupling law, otherwise we
will end up with having either TCP friendly or DCTCP friendly CC schemes.

> know how good it is at that in all scenarios, although we do have code

I'm not sure how friendly/fair BBR is actually to other CCs,
but L4S is also still research...

> <https://www.ietf.org/mail-archive/web/tsvwg/current/msg14771.html>

Hmm, from what I understood so far BBR reacts differently to packet loss
than Reno/Cubic etc. So that coupling by dropping probability may not
work correctly in this case, because BBR will not react according to
1/sqrt(p) (cf. the FAQ from the BBR slide set presented at IETF97,
https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf)

> thought of a better way of using chirping a few weeks ago, so I will be

That would be interesting...

>    when defining these specifications."

I found two MUSTs in these contexts:

draft-briscoe-tsvwg-aqm-dualq-coupled-00:
   In order to prevent
   starvation of Classic traffic by scalable L4S traffic (e.g.  DCTCP)
   the drop probability of Classic traffic MUST be proportional to the
   square of the marking probability of L4S traffic, In other words, the
   power to which p_L is raised in Eqn. (1) MUST be 2.

https://tools.ietf.org/html/draft-briscoe-tsvwg-ecn-l4s-id-02
2.5:
   The likelihood that an AQM drops a Not-ECT Classic packet (p_C) MUST
   be roughly proportional to the square of the likelihood that it would
   have marked it if it had been an L4S packet (p_L).  That is

      p_C ~= (p_L / k)^2

> relate the two, so the number '1' won by default (ie. ECN = 1 * drop^1 ).

That's a different discussion.

> particularly because the law leads to linear growth in p over time.

Yep. However, according to our experiments Codel's dropping law seems to
achieve a quite reasonable loss desynchronization.

> intuition from the stability analysis, not from actual testing.

We tested PIE at 10GB/s and it worked with Cubic similarly as at
low speeds.

> as the DualQ AQM is specified more deeply.

So probably it would be good to find out what is the common denominator
of _new_ L4S-like CCs (not only considering DCTCP) and what is a common
denominator for CCs in the other class.

Regards,
 Roland

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 25 Nov 2016 14:23:26 +0000

Jonathan,

On 22/11/16 20:37, Jonathan Morton wrote:
> If you have intervals between events which follow a 1/sqrt(N) sequence, where N is the number of preceding events, you get an event frequency which increases linearly with time.
Yes, I pointed that out originally 
<https://www.ietf.org/mail-archive/web/aqm/current/msg00376.html>. And 
Toke confirmed in response that it did indeed happen in practice.
> This applies directly to Codel’s signalling strategy, which is to start at one mark per (assumed) RTT, and to increase the marking frequency if that was insufficient to control the queue.
>
Just throwing a square root in "somewhere", doesn't mean it is the 
correct "somewhere".

A stated goal of the sqrt in the CoDel control law is to match the 
1/sqrt(p) in TCP Reno's window formula. Quite aside from whether that is 
a correct goal, it isn't even doing that:
* ACK-clocked load (number of flows, N) is proportional to 1/cwnd, ie. 
proportional to sqrt(p) [see 2nd para of section 4 of PI2 paper 
<http://www.bobbriscoe.net/pubs.html#PI2>]
* because CoDel applies a sqrt to the interval between drops, it results 
in a linear increase in p with time (because the sqrt effectively gets 
squared - see the simple maths in my original posting about this 
<https://www.ietf.org/mail-archive/web/aqm/current/msg00376.html>).

So, the question is: "Why is a linear increase in p (starting from 0) 
good for controlling load from N flows, where N is proportional to sqrt(p)?"

>
> I’m reasonably convinced that Codel is a near-optimal solution to congestion signalling on TCP-friendly flows.
The word "optimal" has a precise meaning. I think you mean simply "I am 
predisposed to CoDel."

Whether the control law increases p linearly with time or by the sqrt, 
or by any function of time, is not the point anyway. Time is not the 
correct unit for this control law. The CoDel control law has no 
variables in it (other than the point at which it starts) that depend on 
any feature of the traffic. Once the CoDel control law starts, it just 
blindly increases until it reaches a high enough drop level to control 
the traffic. So the higher p needs to be, the longer it takes. And the 
lower p needs to be, the more it will overshoot within a round trip.

A constant increase in p with time, and no dependency on the traffic is 
just plain wrong.

No way will that result in anything that anyone could prove was 
"optimal" even if you put caveats around it like "near-optimal" or 
"reasonably convincingly near-optimal".

Perhaps this helps you to see why claims of near-optimality say more 
about the political or religious zeal of the person making the claim, 
than they do about CoDel itself.

> Regrettably, the latter are not the only type of traffic actually found on the Internet.
>
Similarly, isn't the phrase "acceptably well" another way of saying "We 
don't need to consider any other AQM that might be better at handling a 
wider range of load scenarios"? Even though there is already an 
alternative available that increases the drop level dependent on how 
fast the queue is growing.

This religious belief in a particular technology is not healthy. Please 
can we have some objectivity here.

Regards

Bob
>
>   - Jonathan Morton
>
> _______________________________________________
> tcpPrague mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/tcpprague

---

From: "De Schepper, Koen (Nokia - BE)" <[email-masked]>
Date: Fri, 25 Nov 2016 18:34:10 +0000

Hi Roland, Ingemar, Bob, Dave,

I see congestion control mainly as an inter-flow-protocol that regulates how to share the bandwidth. Up to now it was agreed to adapt the rate to 1/sqrt(p). 

Any congestion control deviating from this rule has been proven to fail. Ignoring drop will get you either pushed away, or you push away other traffic. I think most people agree with this?

Also BBR will face the same problem if it keeps on ignoring drop. I tried a few long BBR flows next to long Cubic flows on our L4S testbed and currently (depending on the RTT, rate and queue size) it either starves itself, starves Cubic, or oscillate between the 2 previous cases over 20 second intervals. I also tried only BBR flows on a PIE AQM. Same story there, either high drop (15%), no drop or oscillations, also depending on the same parameters. Try it yourself and see... I see a lot of value in the other BBR mechanisms though, but not the drop ignoring aspects.

I don't see solutions without network support when new congestion controls ignore drop and still need to support other tcp flows which are deployed now...... Up to now there are at least 2 network supported solutions:  DualQ Coupling that is an inter-protocol-translator (and therefor needs to know both protocols) and complete flow isolation like FQ which is the inter-flow-protocol inhibitor (for all flows) and takes over the end-systems role to share the BW (so not really according to the end-to-end principle).

For L4S, our main goal should be to define a new inter-flow-protocol between L4S flows (L4S drafts assumes rate adapted based on ect(1) marking probability). Ones this is decided, we can couple it to the Classic drop/mark law. The network needs to support the coupling, as it is the only one that knows the drop and the marking of both classes. It depends on the end-systems, but it is according to the end-to-end principle, because it is the least and most simple role that the network can have.

If a delay based protocol can be defined (like rate = F(queue delay)? ) then it could also be coupled to a classic drop probability with network support. But I don't think there is a proposal available. Also I don't think a delay based protocol can achieve the same low latency results as an explicit signal.

Koen.

> From: aqm [mailto:[email-masked]] On Behalf Of Bless, Roland (TM)
> Sent: dinsdag 22 november 2016 17:46
> To: Ingemar Johansson S <[email-masked]>; Bob Briscoe
> <[email-masked]>; tsvwg IETF list <[email-masked]>; TCP Prague List
> <[email-masked]>; AQM IETF list <[email-masked]>; tcpm IETF list
> <[email-masked]>
> Subject: Re: [aqm] [tcpPrague] L4S status update
> 
> Hi Ingemar,
> 
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm

---

From: Bob Briscoe <[email-masked]>
Date: Mon, 28 Nov 2016 23:07:34 +0000

Mario,

On 24/11/16 16:57, Mario Hock wrote:
> Hello Bob and Roland,
>
> I followed your discussion and want to share my opinion, here. 
> (Comments inline).
>
> congestion window, that they increase the CWnd up until packet loss 
> etc. Also the 1/sqrt(p)-law is still, more or less, applicable to them.
>
> based i.a. on throughput vs. loss/delay probing. BRR, for example, 
> does not even use a congestion window (at least not in the same way as 
> the other CCs do).
>
>
[BB] That's not for you (or I) to say.

The designer of a new CC can choose to innovate within Classic queues or 
L4S. Whichever they choose, they would set the identifier that 
classifies their packets into the appropriate queue, and they would have 
to coexist with the other behaviours in that queue.

We defined L4S as a framework to encourage innovation with new CCs in 
the L4S queue (because it's easier to deploy very nice properties), but 
please don't assume we were trying to mandate that! Also, if considering 
defining a CC with respect to L4S, bear in mind that L4S is only aiming 
for experimental status in the IETF at present.

L4S is not defined as "anything new must go here".

> innovation in the non-L4S queue.
[BB] Not at all.

> I think L4S is not studied well enough, yet to be standardized in a 
> way that other low delay approaches must adjust to the L4S concepts.
[BB] They don't have to. L4S is aiming for experimental status at 
present. Strictly, other low delay experimental protocols only have to 
prove they coexist with standards track CCs (ie. Reno). In practice, any 
new experimental CC also has to prove it co-exists with the deployed 
base, whether standards track or not (e.g. Cubic, Compound etc). If L4S 
becomes part of the deployed base, it will have got there on its own 
merits, then other experiments that come along later will certainly have 
to interwork.

But anyway, as explained above, this should be no more difficult than 
interworking with Reno in the Classic queue, as if L4S was not there. 
Then interworking with L4S should come for free.

> study other approaches is just too soon.
[BB] I hope I've convinced you that L4S doesn't preclude any of these.

L4S is certainly different in that it is not a CC, rather it is a new 
space for easier CC experimentation. Another difference from all the CCs 
you mention above was the goal of persuading network operators of the 
benefit of creating such a space; because L4S is about improving the 
information flow between network and hosts. But none of that stops other 
CCs using the old space for experimentation.

> Reno/Cubic or L4S/DCTCP. Otherwise more research is required how we 
> want the future of congestion control look like.
[BB] No-one is stopping you proposing something concrete.

You will, however, need an implementation, and proof that app 
performance has significant benefits, and a group of people interested 
in working on the idea, and .... This all takes work and time. Work on 
what has become L4S first started in 2012, see:
1) Wu, H., Ju, J., Lu, G., Guo, C., Xiong, Y. & Zhang, Y., "Tuning ECN 
for Data Center Networks," In: Proceedings of the 8th International 
Conference on Emerging Networking Experiments and Technologies CoNEXT 
'12 pp.25-36 ACM (2012)
2) http://www.bobbriscoe.net/present.html#1207tsvarea-dctcp

The choice of 1/p as a definition of the new L4S experimentation space 
was not arbitrary. There are numerous benefits to the number of 
congestion signals per RTT being * much higher and
* invariant with flow rate.
You will see further research that exploits these properties soon...

Cheers

Bob

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 29 Nov 2016 00:45:26 +0000

Roland,

> implementations, esp. those implemented in hardware.
I am particularly worried about embedding fq in the Internet. That is 
far worse than embedding a subtly different performance improvement for 
certain congestion controls. With fq, the network determines the precise 
departure time of each packet, completely overriding the host's choice, 
without any understanding of what the applications is trying to achieve.

Even worse, in Jan 2017, I am told that fq_CoDel will become hard coded 
into the Linux WiFi drivers, without even a framework to dynamically 
load any alternative(s). Of course, we can add such a framework, but we 
are seeing Linux become the next major middlebox problem. It might be 
excusable if there were not sound alternatives available,... but there are.
> will end up with having either TCP friendly or DCTCP friendly CC schemes.
And your proposal is...?

Also, what exactly is your rationale for not wanting a coupling law? It 
feels nice not to tie anything down. However, when hosts have fuzziness 
around both norms, additional fuzziness between the norms would just add 
an unnecessary dimension of uncertainty for no benefit.

As I just explained to Mario, the choice of "inversely proportional to 
marking probability" as the principle behind the new L4S space was not 
arbitrary.

     Congestion signals per round trip = probability of congestion 
signal * packets per round trip
        = p * W
where W is the window, and p is marking probability.

L4S is defined for
     W ~= k/p
where k is a constant.

So, L4S Congestion signals per round trip = p * W
                                                                  ~= k

That is, L4S is defined so that congestion signals per round trip is 
invariant as flow rate scales. That's an extremely important property..

> I'm not sure how friendly/fair BBR is actually to other CCs,
See Koen's recent post.
> but L4S is also still research...
Well, L4S is more a space in which research can flourish. And also a 
space where there's an initial CC (DCTCP) already deployed in 3 OSs, 
with a lot of deployment experience in controlled environments, and it's 
showing pretty cool results over the Internet, even though DCTCP wasn't 
designed for the Internet.

I call L4S and incrementally deployable clean-slate.
> https://www.ietf.org/proceedings/97/slides/slides-97-iccrg-bbr-congestion-control-02.pdf)
Well, it's BBR's problem to show that it can interwork with the existing 
Internet first (and I am not including L4S in "the existing Internet" yet).
>     have marked it if it had been an L4S packet (p_L).  That is
They are deliberate MUSTs, linking together the meaning of the two 
congestion signals that a network queue applies. Neither constrains 
hosts; the SHOULD in the host behaviour is deliberately more liberal to 
hosts.

The network is given freedom to flex the constant of proportionality (k 
in the above equation), but not the long-term scaling exponent (the 
square). That allows networks and equipment vendors to differentiate 
themselves, without compromising long term scaling.

Given hosts have flex around both "norms", if the network flexed the 
scaling relationship between the "norms" as well, it would add another 
dimension of uncertainty to the system, with no benefit.

> That's a different discussion.
Well, no. Think of it as push-back against your earlier sentence: "I'd 
like to have a solution that doesn't build in a specific coupling law".

We are proposing the square coupling law at the IETF precisely because, 
without a standard, no-one would be able to design a CC and be able to 
say anything about how it shares out capacity.
> We tested PIE at 10GB/s and it worked with Cubic similarly as at
OK.
> denominator for CCs in the other class.
The text proposal to answer this is in the ecn-l4s-id draft.

Pls feel free to suggest specific text improvements.

Cheers

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 29 Nov 2016 03:20:28 +0200

> On 29 Nov, 2016, at 02:45, Bob Briscoe <[email-masked]> wrote:
> 
> I am particularly worried about embedding fq in the Internet. That is far worse than embedding a subtly different performance improvement for certain congestion controls. With fq, the network determines the precise departure time of each packet, completely overriding the host's choice, without any understanding of what the applications is trying to achieve.

You don’t seem to be very fond of *either* component of fq_codel, then.  A shame, since it works very well in practice.

What fq_codel does is identify latency-sensitive flows by the fact that they are not taking up their fair share of the bandwidth.  Packets belonging to these flows are typically delivered *immediately* and *without loss*.  This is a far cry from “completely overriding the hosts’s choice” of timing, as you claim.

In fact, I would actually be grateful if you retracted that particular claim.

It must be emphasised, though, that flow-isolating AQMs are not designed for the Internet core - there are just too many individual flows to manage.  Their place is at the edge, on either end of last-mile links, where the bottlenecks are most apparent.  For core, backhaul and peering links, plain AQM is sufficient and much easier to implement.

> Even worse, in Jan 2017, I am told that fq_CoDel will become hard coded into the Linux WiFi drivers, without even a framework to dynamically load any alternative(s). Of course, we can add such a framework, but we are seeing Linux become the next major middlebox problem. It might be excusable if there were not sound alternatives available,... but there are.

This tight integration is because it was necessary to solve some serious, long-standing problems with Linux wifi, which couldn’t be solved satisfactorily at the qdisc layer because information about wifi-specific things was needed - and there were *no* practical alternatives which actually solved the problem, otherwise we’d have used them.

Wifi is also a last-mile technology, and it is often the bottleneck in several types of practical deployment.  Large conferences are a particular example.  I’m rather looking forward to seeing the first large conference to deploy the new Linux wifi stuff, and seeing whether it has made the typical load there easier to cope with.  It probably has.

 - Jonathan Morton

---

From: Matt Mathis <[email-masked]>
Date: Mon, 28 Nov 2016 18:55:28 -0800

Bob's point is that fq_anything forfeits any mechanism for an application
or user to imply the value of the traffic by how much congestion they are
willing to inflict on other traffic.

This concept is the foundation of ConEx and related technologies which
could move the capacity allocation problem into the economic domain.  This
actually makes a lot of sense.    However, after investing a lot of time in
it over the years (in collaboration with Bob etal), I have come to believe
that this only makes sense for enforcing financially equitable use of a
deliberately undersized network.    Building capacity is generally cheaper
than building the congestion instrumentation.....

That said, fq_anything does not work at core router scale.

Note that there are two views, each of which is self consistent:

1) You need fq_* to isolate flows; prioritization must be done with
IP/TOS/DSCP bits; aggressive flows can't hurt other flows; low delays
require that flows sharing a Q to be nice to each other and respond to AQM
2) Uniform AQM/drop/mark per packet permits shared economic view of the
value of the traffic (e.g. a price) ; traffic is prioritized by how
aggressive of CC you choose; low delay [is/should be] a design property of
the shared CC and AQM algorithms.

If you have a way to create proper incentives about congestion (e.g. price
and chargeback), #2 is probably a strong system; if that fails #1 is
probably stronger.

Note that half solutions or solutions split between the models don't work.
period.  Arguing about incomplete systems that are missing some of the
parts is pointless because they don't work at some level (often layer 8 or
9).

Drop tail counts as "neither", so even partial deployment of anything can
be an improvement.

Thanks,
--MM--
The best way to predict the future is to create it.  - Alan Kay

Privacy matters!  We know from recent events that people are using our
services to speak in defiance of unjust governments.   We treat privacy and
security as matters of life and death, because for some users, they are.

On Mon, Nov 28, 2016 at 5:20 PM, Jonathan Morton <[email-masked]>
wrote:

> You don’t seem to be very fond of *either* component of fq_codel, then.  A
> shame, since it works very well in practice.
>
> What fq_codel does is identify latency-sensitive flows by the fact that
> they are not taking up their fair share of the bandwidth.  Packets
> belonging to these flows are typically delivered *immediately* and *without
> loss*.  This is a far cry from “completely overriding the hosts’s choice”
> of timing, as you claim.
>
> It must be emphasised, though, that flow-isolating AQMs are not designed
> for the Internet core - there are just too many individual flows to
> manage.  Their place is at the edge, on either end of last-mile links,
> where the bottlenecks are most apparent.  For core, backhaul and peering
> links, plain AQM is sufficient and much easier to implement.
>
> into the Linux WiFi drivers, without even a framework to dynamically load
> any alternative(s). Of course, we can add such a framework, but we are
> seeing Linux become the next major middlebox problem. It might be excusable
> if there were not sound alternatives available,... but there are.
>
> This tight integration is because it was necessary to solve some serious,
> long-standing problems with Linux wifi, which couldn’t be solved
> satisfactorily at the qdisc layer because information about wifi-specific
> things was needed - and there were *no* practical alternatives which
> actually solved the problem, otherwise we’d have used them.
>
> Wifi is also a last-mile technology, and it is often the bottleneck in
> several types of practical deployment.  Large conferences are a particular
> example.  I’m rather looking forward to seeing the first large conference
> to deploy the new Linux wifi stuff, and seeing whether it has made the
> typical load there easier to cope with.  It probably has.
>
>  - Jonathan Morton
>

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 29 Nov 2016 05:42:11 +0200

> On 29 Nov, 2016, at 04:55, Matt Mathis <[email-masked]> wrote:
> 
> Bob's point is that fq_anything forfeits any mechanism for an application or user to imply the value of the traffic by how much congestion they are willing to inflict on other traffic.

Yes, it does.

I actually consider that a good thing, because most applications will, given the choice, choose to inflict more congestion on other traffic in order to boost their own performance.  There are honourable exceptions, but it’s not a behaviour we can solely rely on.

For example, Steam uses between four and eight parallel TCP streams (I can’t figure out what the number depends on) to receive game updates, when one or two would already saturate most domestic Internet connections.  This magnifies the impact on other things the user might be doing with that connection, such as - ironically enough - playing multiplayer games.  You’d think Valve, of all companies, would keep that in mind.

> This concept is the foundation of ConEx and related technologies which could move the capacity allocation problem into the economic domain.

“Economic domain” only works if there is a financial cost borne by the causer of the congestion.  Good luck making that work, in a world where IoT device manufacturers don’t bear the costs of DDoSes launched through them.

> That said, fq_anything does not work at core router scale.

Correct.

> 1) You need fq_* to isolate flows; prioritization must be done with IP/TOS/DSCP bits; aggressive flows can't hurt other flows; low delays require that flows sharing a Q to be nice to each other and respond to AQM

…or that different flows are carefully kept in different queues.

Cake uses set-associative flow hashing to achieve flow isolation much more reliably than the current version of fq_codel.

Cake also applies Codel and BLUE in parallel, each covering a different AQM regime - BLUE takes over if and when Codel fails to control a particular queue.  If both Codel and BLUE fail to control the queue, Cake uses head-drops from the longest queue to remain within a total memory budget.  All of these avoid penalising well-behaved traffic whenever possible.

Cake also has mechanisms which consider “per host” fairness simultaneously with “per flow” fairness.  Incidentally, the Linux wifi got “per station airtime fairness” along with the fq_codel upgrade, achieving a similar aim by different means.

SFB uses a Bloom filter to a similar end, though that’s not strictly an FQ qdisc.

> 2) Uniform AQM/drop/mark per packet permits shared economic view of the value of the traffic (e.g. a price) ; traffic is prioritized by how aggressive of CC you choose; low delay [is/should be] a design property of the shared CC and AQM algorithms.

It’s fair to say that “uniform AQM” is the only mechanism available at the core level, mainly because there are too many flows there to treat individually.  But at that level, network engineering is supposedly all about providing sufficient link capacity so that AQM of any kind is unnecessary, because there is no congestion.  Still, plain AQM is a valid and potentially useful mitigation against transient overloads.

There are exceptions.  Some ISPs have been known to deliberately restrict peering capacity in certain directions to deliberately cause congestion to specific types of traffic, supposedly as a financial lever.  These ISPs would not be interested in applying AQM to reduce the impact of this deliberately-induced congestion.

I do have to ask, though, what protection this “shared economic view" provides against a single bulk flow which simply ignores all congestion signals?

> If you have a way to create proper incentives about congestion (e.g. price and chargeback), #2 is probably a strong system; if that fails #1 is probably stronger.
> 
> Note that half solutions or solutions split between the models don't work. period.  Arguing about incomplete systems that are missing some of the parts is pointless because they don't work at some level (often layer 8 or 9).

Indeed.  I’m not aware of any “complete” systems in category 2 - and I don’t count “data caps” among them, unpopular though they are.

 - Jonathan Morton

---

From: Dave Täht <[email-masked]>
Date: Tue, 29 Nov 2016 19:40:31 -0800

The advantages of FQ are probably more widespread than people think.

* Switches tend to multiplex between ports, thus mixing up traffic
naturally.

* Most modern ethernet cards expose 8-64 queues - why? because this
provides clear paths for multiple cpus to forward them - it's not
network related at all! - 1gig cards typically have 8, 10gig, 64.

While 8 induces a severe birthday problem, 64 is generally "just fine"
and in either case, large amounts of mixed traffic respond reasonably to
even only a few queues.

I don't see the trend to ever more hardware queues subsiding, as it's
a function of the number of cores, which keeps going up and up.

* Google'd hosts - and those of a few cloudy providers - long ago
switched to sch_fq which also interleaves (and paces) flows at an
appropriate burst level on a 1ms interval. So source flows are getting
fq'd on an ever more regular basis.

So when the assertion is made that fq is impossible on core routers, I
have to point to the fact that much of the traffic traversing them have
already had a great deal of mixing already applied to it by devices
downstream, AND most core routers have more than one link they are
aggregating via multiplexing in the first place.

That's the FQ in the core and datacenter today. I view the side-effects
of the 1 cpu per hw queue as essentially accomplishing all the fq
needed, AND that applying a single aqm to all those flows is well nigh
impossible as these flows MUST be spread across cores to reach these rates.

...

I see no future in hardware designs that can scale past 25Gbit that do
not do some form of parallelism across flows. This again, is something
that already essentially happens within vlans and with multiple
forwarding lookup tables in those.

100Gbit devices are 4 25GB queues wired together, at minimum. Etc.

(Most of our problems nowadays btw, is on rx, not tx - it's really hard
to do all the work required on the rx side!)

In terms of fq_codel -

For strong reasons (birthday problem) we settled on 1024 queues as a
good general purpose number that works well in practice on real data on
real loads, for speeds between 0 and 10Gbits, for homes and small
business. Initially.

We are now seeing people deploy fq_codel successfully on mid-range
systems with 64 hardware queues at 10Gbit - 64K fq_codel'd queues.

Whether or not the aqm part is that effective with that many queues on
real traffic is beyond me. I've had no complaints - as the alternative -
no aqm, no fq - is far worse.

...

The early experiments with fq_pie (not the bsd version, the sch_fq
derived version) showed something like a 12% (don't quote me!)
improvement in the aqm performance by presenting a more equitable
distribution of test traffic (that is presently over-bursty, created by
bursty things like IWXX, tso/gso/gro offloads, and so on).

I was happy with that result, but cognizant that so much test traffic
looks *nothing like* real traffic. Real traffic does not consist of the
long sustained loads so beloved of experimenters and theorists.

The benchmarks I care about most are voip, videoconferencing, gaming,
dns, tls setup, page load time, and anything else that demands low
latency, of course.

...

Cake has pushed the limits of the possible, with the 8 way set
associative idea making for the perfect fq all the math is based on,
I rather would like more folk testing it - it's stable enough now, and
backported as far back as linux 3.14. I am still unsure of the various
mods to the aqm algo - the new per host fq within fq feature (a common
feature request), the denatting, and the optional shaper is a major
advance, and the classification ideas needing further evaluation.

It seems eminently possible to add l4s ideas to it, as soon as someone
gets around to a usable ect(1) patch for tcp ecn negotiation.

When last I tried it, it could push 17GBit through a 40Gbit card, where
pfifo fast did about 27 on a single core/hw queue benchmark. We were
bottlenecking on rx! to be unable to test harder than that.

---

From: Dave Täht <[email-masked]>
Date: Tue, 29 Nov 2016 20:09:16 -0800

>> If you have intervals between events which follow a 1/sqrt(N) sequence, where N is the number of preceding events, you get an event frequency which increases linearly with time.  

We do have a tendency to talk past each other regarding codel's rate.
the 1/sqrt(N) only applies during codel's initial seeking toward an
ideal drop rate. after that -

for 99.99999% of the time, after that, there is no definable *rate* in
codel - it varies based on the workload and the time spent above and
below the target. I really wish I'd produced an animation of this rather
than the first graph I did at stanford.

...

as for what that initial ramp has to do with reno or tcp's behaviors, it
seems to work, and we've tried other things that didn't work as well.
Give cake's variant a shot. You tell me.

> good for controlling load from N flows, where N is proportional to sqrt(p)?"

Why do you keep insisting that the rate is this rate? See above.

> The word "optimal" has a precise meaning. I think you mean simply "I am

I have yet to see one convincing benchmark or paper of a single queued
aqm that outperforms fq_codel.

We've done thousands over here. And shared the test data. And the tool.
What you got?

> lower p needs to be, the more it will overshoot within a round trip.

Sigh.

> than they do about CoDel itself.

Benchmarks, here. Not zeal.

> can we have some objectivity here.

I really, really feel like this is really going overboard.

It took me this long to reply to this mail because of this bluster.

We have, on this side, published all the code, published every
benchmark, published the vast majority of our test data, and at every
point, shown fq_codel to be the win it is. Cake is a few percentage
points better in some respects. If some technology arrives worth
benchmarking then by god we'll benchmark it.

I don't know how to fall on our sword more than this.

So far as I know, to date, you - and very few on these lists - have
bothered to even try the production quality code we've made readily
available in now dozens of router distributions, and cheap hardware, in
any scenario. But I suspect millions now have. With no complaints. In
fact, near universal praise.

The religion is on your side, bob, not ours. I will always benchmark any
code you give me with the tools I have. My sole mission in life is to
beat bufferbloat in my lifetime.

I have tired of the dialog on these lists because since the bufferbloat
effort started, several billion machines has shipped without decent
queue management,
and I would much rather solve the problems as best I can, for those that
are willing to try the solutions we have developed so far.

- and especially - as we learn from the deployment, what other real
problems exist, to feed forward into the process.

Even then, even with the make-wifi-fast fq_codel and airtime fairness
patches for the ath9k still landing, we will at best, improve the
performance of only a few million more devices, in the hands of early
adopters, next year, a small fraction of the sold total. There are
hundreds of chipsets left to fix.

It's a big Internet. There's room for everyone's ideas on it. I look
forward to seeing yours reach a deployable state.

---

From: Dave Täht <[email-masked]>
Date: Tue, 29 Nov 2016 22:15:37 -0800

>> Even worse, in Jan 2017, I am told that fq_CoDel will become hard coded into the Linux WiFi drivers, without even a framework to dynamically load any alternative(s). Of course, we can add such a f…

A) As soon as the need for a pluggable framework arises, the linux wifi
architecture is now simple enough to do so. As it is the results are so
spectacular compared the previous "std" behavior that it will take some
doing to prove a different point. Reductions in latency under load of
over an order of magnitude, bandwidth improvements of 5x, stuff like
that... ( https://lwn.net/Articles/705884/ )

Given that this effort took 3 years out of multiple people's lives, and
a year of slow steady, patching out the old wifi engine, with thousands
of hours of testing and refinement, and a summer lost to fixing bugs, on
next to no budget...

http://blog.cerowrt.org/post/crypto_fq_bug/

I hope that more people give it a go soon (most of it is already
available in nightly builds of lede-project ar71xx based platforms, and
nearing shipment elsewhere in things like the turris omnia). There is at
least one bug left to fix, always.

The ECN support works beautifully, in particular, and this is the first
time in my life, since working on wifi in 1998, that I'd feel
comfortable deploying a single AP to multiple station solution in a
commercial multi-customer environment - (after a bit more testing in the
yurtlab campground testbed) - before now, only p2p was sanely saleable.

B) Secondly, FQ technology has always existed in wifi - many WISP
vendors altered the MAC to behave via TDMA - or called something TDMA
that was really SFQ. Some like ubnt, have even gone so far as
implementing full duplex over separate channels in their high end
airstream products, on top of that. FQ has long been a part of meraki's
product line. I don't know how cisco does their magic, but...

C) Airtime fairness first appeared as optional features in cisco and at
least one other manufacture's gear at least 6 years back. the problem it
solves was first described in 2003.

So it is ironically the inclusion of an aqm for the first time that is
the most novel feature of this make-wifi-fast work (on top of fq, and
with airtime fairness), integrated tightly. Very little work on aqm
technology prior to now has been applied to highly contended, half
duplex, lossy layer 2 medium systems, and I expect there will be much
more work to be done here.

So... it is far from "just linux" you need to deal with - and far less -
the initial patchset is for the ath9k chipset only, with the ath10k
running a bit behind, and the mt76 still floating in limbo. We are still
in search of other chipsets worth optimizing, with the hope we'll find
something that is in an android as well as at least one other 802.11ac
chipset. From a queue management perspective there is mu-mimo next
(which requires gang scheduling of up to 16 stations), reducing txop
size under contention, and dealing with currently excessive layer 2
retries sanely under contention.

BUT: from an engineering perspective, there are two other things wrong
with wifi that are more severe than these latter problems - the first
one - excessive channel scans - was seriously detrimental to voip and
videoconferencing and the air around the user - and is on its way to
being improved - bug and pending fix described here:

http://blog.cerowrt.org/post/disabling_channel_scans/

The second huge problem is that rate control, now that there is such a
large search space in wireless-n and ac - is no longer as quickly
effective as it needs to be - taking 10-20sec or more to settle in on a
good rate, when 100ms would be ideal. Work on fixing that (which also
leverages the all the work above) is hopefully coming along again
(google for minstrel-blues for some of the more public details)

These last four problems are much more severe than coming up with a
different fq, aqm, or airtime fairness algorithm is, and thus, most
worth focusing on next. Hopefully - someone *else* focusing on next -
I'm tired of working on microsecond timescales and would like to find a
new gig that relied on no timescale shorter than a season.

...

Certainly I hope that by january an ath9k laptop and an ath9k AP will
behave better than any linux system has behaved since 802.11b has,
certainly be superior to current LTE in all respects - but it will
probably be another 3-6 months before the major distros pick it up, and
we have not identified another common laptop wifi chipset that can be
improved in these ways.

(the core work mostly applies to APs, and does help a lot no matter the
client chipset or OS. I also hope to find someone working on an enodeb
to start applying the same algorithms in this timeframe)

> Wifi is also a last-mile technology, and it is often the bottleneck in several types of practical deployment.  Large conferences are a particular example.  I’m rather looking forward to seeing the…

Hopefully the upcoming SCALE conference will be our first major
deployment test. If not, battlemesh, wlan slovinia, or sudoroom.

Or for all I know, open-mesh, eero, portal, meraki, and/or plume have
already picked up the code.

---

From: John Leslie <[email-masked]>
Date: Wed, 30 Nov 2016 16:56:08 -0500

(could we agree to reduce the Cc list on this thread?)

Jonathan Morton <[email-masked]> wrote:
> belonging to these flows are typically delivered *immediately* and
> *without loss*.

   This is a nice feature; and I certainly don't mean to badmouth fq_codel.

   But it's guessing... And it depends on those flows being able to know
"their fair share" milliseconds in advance: which can fail on short notce.

   Better would be to identify latency-sensitive flows by a specific signal,
and continue to deliver a "fair share" of packets "immediately" and marked
to show the congestion.

   Consider voice traffic. Voice can be intelligible at rather low data
rates; but sounds much better at higher data rates. For conferencing uses,
low delay is pretty critical.

   Sudden reductions in data rate are annoying, but unexpected silence
while you wait for delayed packets are much worse; and losing track of
how much delay there actually is can be fatal.

   Low-latency is "nice" for all traffic; but I claim it's critical for
voice. (Of course, it's "critical" for some other things, too...)

+++++++++++++++++

   Ideally, each packet would carry an indication of a latency target,
beyond which it's less useful and congestion needs to be indicated. I
have been very disappointed that we designed IPv6 without leaving space
for this.

++++++++++++++++

   (Actually, we _did_ originally design for this: TTL was indended to be
decremented every second, whether or not forwarded. Alas, this was grossly
inadequate resolution; so this usage was abandoned.)

--
John Leslie <[email-masked]>

---

From: Jonathan Morton <[email-masked]>
Date: Thu, 1 Dec 2016 03:53:34 +0200

> On 30 Nov, 2016, at 23:56, John Leslie <[email-masked]> wrote:
> 

Cake does that, too.  It looks at Diffserv markings and sorts them into categories, each having a separate set of queues.  It even takes care to avoid both obvious incentives for using inappropriate DSCPs, and hard bandwidth caps where there is spare bandwidth to use - properties that few other Diffserv implementations achieve simultaneously.

All of the presets (except for a legacy “precedence” mode) recognise at least CS1 (for background traffic, which in theory BitTorrent should be using), TOS4 (often used by SSH), CS6 (for NTP), and VA/EF (for voice traffic) as distinct from best-effort traffic, and apply appropriate parameters.

A similar system can be built using multiple fq_codel instances and a classifier, and such a construction has been available as part of OpenWRT for some time.

In practice, not very much traffic carries an appropriate Diffserv code.  We’ve found the “sparse = latency sensitive” heuristic to be a good one in practice, both in Cake and in fq_codel.

 - Jonathan Morton