Subject: Re: [aqm] ECT(1)

From: Bob Briscoe <[email-masked]>
Date: Tue, 04 Aug 2015 00:23:24 +0100

John, Richard,

I do not believe an IP (v4) option or a v6 extension would be necessary. 
If ECT(1) were used that would surely be sufficient alone.

I believe the main criteria for an identifier for this new service are:
1. preferably orthogonal to Diffserv classes.
2. preferably end-to-end in scope
3. preferably classic (RFC168) ECN and 'L4S' ECN would not permanently 
burn two codepoints, since it seems that 'L4S' could eventually subsume 
classic ECN (a fork would not be needed, because classic ECN doesn't 
seem to do anything that L4S cannot do).

*ECT(1) **
*Seems a good identifier, but it has the following problems:

a) L4S traffic would need to be distinguished from classic ECN both when 
unmarked (ECT0 vs ECT1) and when marked (CE vs CE???). Ie. congestion 
experienced (CE) would have to be shared between the classes. It would 
not be so problematic if all queues classified all CE packets as the 
lowest latency class (L4S); CE packets from classic flows would then be 
delivered early out of order, requiring some buffering, but probably no 
more buffering than is already needed for retransmissions, and at least 
they would never be late out of order. See also {Note 1}.

b) ECT(1) is the last available ECN codepoint (for both v4 & v6). Using 
ECT(1) for L4S and ECT(0) for Classic ECN would burn the last codepoint 
just for migration purposes (contravening my criterion #3). If we could 
predict that migration might one day finish, we could foresee a time 
when ECT(0) might become available again. But that's a long shot.

c) For the record, the following uses of ECT(1) have been proposed by 
the IETF and by researchers:
* receiver cheat detection (the ECN nonce [RFC3540] - experimental)
* ECN path testing (ECN for RTP [RFC6679] - standards track)
* various intermediate congestion level proposals (including PCN 
[RFC6660] - standards track)
* various fast-start proposals (in research, e.g. VCP)

d) PCN is defined for a controlled environment, so that's not a problem. 
THe wording or RTP-ECN does not mandate the use of ECT(1), but it is not 
always clear that it is optional either. So I am trying to find out 
whether any implementations have used ECT(1). Even if none of the IETF 
uses of ECT(1) are problematic in practice, we should think very 
carefully before burning ECT(1) for L4S, because there do appear to be 
new uses being proposed for it that address a new potentially important 
class of problems: getting up to speed fast.

*DSCP**
*It might be better to distinguish L4S ECN from Classic ECN by using 
only ECT(0) and CE, but also using a distinctive DS codepoint for L4S. 
L4S could start off local-network only (e.g. for a network operator's 
premium services), or a global DSCP could be burned so that hosts could 
set it without needing to be configured for the network they happen to 
be connected to at any one time. Then, assuming all Classic ECN might 
eventually migrate to L4S ECN, a DSCP would no longer be needed as well 
as ECT(0) to identify L4S. Then the ECN field alone could represent L4S 
end-to-end.

We all know that DSCP has the following problems:
a) Diffserv is not orthogonal to Diffserv (obviously), so multiple DSCPs 
might be needed for L4S in each DS class
b) DS is not end-to-end
c) few global DSCPs left, altho certainly there are more DS codepoints 
than ECN codepoints left.

*Summary**
*Combining ECT(0) and CE with a globally assigned DSCP solely during 
initial deployment of L4S seems the least worst choice.

Bob

{Note 1} I'm sure some SDN-enabled equipment could easily decide which 
queue to classify CE packets into by classifying transport layer flows 
or the SPI of IPsec packets based on earlier ECT(1) or ECT(0) packets. 
However, we shouldn't and can't always rely on visibility of transport 
flows or on availability of SDN-capable hardware.

On 28/07/15 15:50, John Leslie wrote:
> Scheffenegger, Richard <[email-masked]> wrote:
>> Regarding the identifier to differentiate legacy ECN (cc reaction on
>> ECN per RTT = cc reaction to loss in RTT) and the differentiated
>> response (proportional to the # of CEs per RTT) is an open question.
>     Wide open!
>
>> Perhaps we can start to gather the views of the group on this list
>> already; There have been other uses for ECT(1) been proposed over
>> time, and it has also been pointed out that repurposing a DiffServ
>> Codepoint may not be the easierst to do from a procedural view.
>     Nor (IMHO) is a repurposed DiffServ Codepoint likely to work well.
>
>     OTOH, I think the time is ripe to deprecate ECN Nonce; and reserve
> ECT(1) for (currently) Experimental uses.
>
>     We could pretty easily combine that with an IP Option to define the
> particular experiment, and even record data about forwarding node action.
> (Admittedly, this is more difficult in IPv6, so I suggest initially
> considering IPv4; and then adding an IPv6 kludge, probably hop-by-hop.)
>
>     IMHO, the only other uses for ECT(1) that have approached "traction"
> relate to differentiation of forwarder action for ECN marking.
>
>> Also, I would like to encourage members of the aqm wg to review the various
>> versions of http://tools.ietf.org/html/draft-kuehlewind-tcpm-accurate-ecn
>> (the technical content did change in each version substantially).
>> Disclaimer, I'm one of the authors of that draft.
>     I believe I've reviewed them all; but is it truly reasonable to expect
> all WG members to review more than the latest one?
>
>     Fundamentally, the latest version calls for three counters, to be
> masked to low-order bits reporting changes in the counters. There are
> three possibilities on _where_ these bits will be reported. (I'd rather
> not bike-shed the virtues of each yet...)
>
>     draft-briscoe-aqm-dualq-coupled does not appear to have been posted
> yet: I'll be only too happy to discuss it once it is...
>
>     Fundamentally, it proposes to identify a separate low-latency queue
> which will report via ECN a near-zero-latency condition (vs. a zero
> congestion in this second queue), while the default queue operates by
> a legacy AQM mecanism, probably signaled by packet drop.
>
>     This strikes me as Experimental: there are a number of issues, such
> as how this low-latency queue overflows and what happens when it does,
> which need to be resolved before we could even consider Standards Track.
> At first blush, this looks like a good match for Experimental use of
> ECT(1).
>
> --
> John Leslie <[email-masked]>
>
> _______________________________________________
> aqm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/aqm
>
>

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: John Leslie <[email-masked]>
Date: Tue, 4 Aug 2015 00:08:24 -0400

> If ECT(1) were used that would surely be sufficient alone.

   Alas, we're facing a political question: not just a technical one.

   I think folks are ready to deprecate ECN Nonce; but I'm not
optimistic that folks are ready to embrace "Low-Latency, Low-Loss and
Scalable" (L4S) service, as introduced in draft-briscoe-aqm-dualq-coupled
(Has this draft been posted?).

   (IMHO, this work promises to be very valuable for the _many_ uses
that are latency-sensitive; but adoption is going to be a major
challenge!)

   We may indeed eventually get to where Bob is thinking today; but
I don't see a clear path to IETF-wide consensus yet. Even getting an
Experimental RFC approved which re-purposes ECT(1) strikes me as a
very significant challenge. :^(

> 1. preferably orthogonal to Diffserv classes.

   IMHO, Diffserv classes are poison!

   There are a number of good folks pursing a Less-than-Best-Effort
diffserv class. I wish them luck! But I'd be amazed if they succeed.

   Diffserv classes are the private preserve of a _large_ number of
network-service-providers. Best-Effort is the only one with universal
agreement what it means.

   All the others are subject to non-documented shuffling at the
boundaries between providers (and "bleaching" to Best-Effort at many
points within providers' networks).

> 2. preferably end-to-end in scope

   I'm really not sure how we can make L4S useful if it lacks an
End-to-End meaning. The signal must enter at the sender and mostly
survive all the way to the receiver in order that the receiver
(by whatever magic) can tell the sender about any congestion.

>    burn two codepoints, since it seems that 'L4S' could eventually
>    subsume classic ECN (a fork would not be needed, because classic
>    ECN doesn't seem to do anything that L4S cannot do).
> 

   This is "nice to have", I suppose; but it seems too optimistic
to take seriously. Deployment of L4S will take at least five years;
and nobody's crystal-ball is good enough to see beyond that.

   Furthermore, I don't see how we can _ever_ entirely eradicate the
RFC 3168 behavior of "same as drop". Furthermore, L4S _can't_
eliminate packet drops; and IMHO a packet-drop in an L4S stream
must be treated _differently_ than a L4S congestion mark.

> a) L4S traffic would need to be distinguished from classic ECN both
>    when unmarked (ECT0 vs ECT1) and when marked (CE vs CE???).
>    Ie. congestion experienced (CE) would have to be shared between
>    the classes.

   Actually, there are _two_ ways ECT(1) could be used:
- ECT(1) could be set to request L4S forwarding rules marking CE
  to indicate L4S congestion; or
- L4S forwarders could change ECT(1) to ECT(0) (or vice-versa?),
  to mark L4S congestion.

>    It would not be so problematic if all queues classified all CE
>    packets as the lowest latency class (L4S); CE packets from classic
>    flows would then be delivered early out of order, requiring some
>    buffering, but probably no more buffering than is already needed
>    for retransmissions, and at least they would never be late out of
>    order. See also {Note 1}.

   I'm trying to follow this...

   What exactly does Bob mean by "all queues"? Mostly we think of
queues as part of the forwarding action. But some forwarders choose
their action upon packet entry to the queue; other at packet exit.
And, AFAIR, no forwarder takes an action based upon the packet being
CE-marked when it arrives.

> b) ECT(1) is the last available ECN codepoint (for both v4 & v6).
>    Using ECT(1) for L4S and ECT(0) for Classic ECN would burn the last
>    codepoint just for migration purposes (contravening my criterion
>    #3). If we could predict that migration might one day finish, we
>    could foresee a time when ECT(0) might become available again.
>    But that's a long shot.

   This is a political problem, more than a technical one.

   We've painted ourselves into a corner, where there aren't spare
bits -- and the "spare bits" in IPv6 turn out to be unusable. (We
seem to have done this quite deliberatly -- I don't understand why!)

   Nonetheless, we have a major need to mark incipient congestion, so
that we can avoid over-filling buffers at forwarding nodes. The fact
that we have only half-a-bit left to do this is the inevitable result
of our refusal to allocate enough bits in the first place (or if you
prefer, our insistence on using six bits for DSCP, defined in such a
way as to prevent end-to-end meaning of them).

   (Personally, I'd love to reclaim a few bits from DSCP; but to propose
this would label me a clueless kook, so I won't.)

   ECT(1) is there! It's allocated for ECN use. Refusing to define it
with an ECN meaning is simply irrational.

   Furthermore: there _is_ another bit! See RFC 3514. ;^)

>  * various fast-start proposals (in research, e.g. VCP)

   IMHO, only RFCs count as "proposals".

   RFC 3540 is ripe for deprecation, IMHO.

   RFC 6679 covers "ECN for RTP over UDP". Somehow I missed it coming
out in 2012 (though I must have been listening to the IESG telechat
where it was approved). Mea culpa!

   It's not an easy read (58 pages, heavy with RTP details)! At first
blush, I don't see what it's trying to do with ECT(1). It references
RFC 3168 for the meaning of ECT(1); it keeps separate counters for
ECT(0) and ECT(1); and it has a "random" mode (not RECOMMENDED) which
is supposed to randomize whether ECT(0) or ECT(1) is sent.

   The overall impression is that it tries to define feedback for all
possible ECN cases: thus supporting ECN Nonce use as well as all other
uses known at the time it was written.

   To deprecate ECN Nonce, we'd need to UPDATE RFC 6679 as well as
RFC 3168; but I don't see any new issues introduced by 6679 (and the
features of it are already appropriate for L4S.

>    The wording or RTP-ECN does not mandate the use of ECT(1), but it is
>    not always clear that it is optional either.

   Clearly, keeping separate counters for ECT(1) and ECT(0) is required;
but sending ECT(1) vs ECT(0) is not specified within RFC 6679.

>    So I am trying to find out whether any implementations have used
>    ECT(1).

   At first blush, it would appear that the only _current_ use of ECT(1)
would be for ECN Nonce. But of course, RFC 6679 says nothing to prevent
its use for L4S.

>    Even if none of the IETF uses of ECT(1) are problematic in practice,
>    we should think very carefully before burning ECT(1) for L4S,
>    because there do appear to be new uses being proposed for it that
>    address a new potentially important class of problems: getting up to
>    speed fast.

   Some citations, please...

   (BTW, I think L4S could be _very_ helpful for "speeding up" slow-start.)

> be connected to at any one time.

   I don's see L4S as useful in "local-network-only" mode.

   Granted, there _are_ many cases where the benefit of L4S would be
greatest at the first hop (DOCSIS box). But the expected "bleaching"
could be very confusing as to the meaning of CE marking that could be
generated farther along the path. There is no such thing as a condition
where _only_ the first hop can experience congestion.

> Then, assuming all Classic ECN might eventually migrate to L4S ECN,
> a DSCP would no longer be needed as well as ECT(0) to identify L4S.
> Then the ECN field alone could represent L4S end-to-end.

   This is overly optimistic.

>    might be needed for L4S in each DS class

   That seems fatal...

>    than ECN codepoints left.

   Network operators don't believe in "global DSCPs" They bleach anyway.

   (I would tend to support carving out part of the "Experimental"
subset of DHCPs as "must propagate if not understood" -- and possibly
in ten years there might be enough equipment out there that respected
that... but for now, it _all_ gets bleached.)

> initial deployment of L4S seems the least worst choice.

   We certainly could Experiment with that; but I'm very pessimistic.

   OTOH, Experimenting with ECT(1) seems likely to work. IMHO...

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 04 Aug 2015 12:39:09 +0100

John,

There was much discussion about this identifier issue in Prague. Many 
people are well-aware that the sticking point is availability of an 
identifier, and its associated politics.

> points within providers' networks).
DSCPs are only poison in standards. As you say, ISPs use DSCPs for many 
internal services already.

(BTW, the request for a global scope DSCP for less-than-BE is very 
different to this case, because bleaching LBE (remarking to the 
all-zeros DSCP at borders) always /increases/ its priority.)

> (by whatever magic) can tell the sender about any congestion.

ISPs could go ahead with using a local DSCP now for L4S for their own 
premium services. A large proportion of traffic these days is served 
from within the same ISP as the user is connected to (esp using CDNs), 
so this would be very "useful". Using a DSCP for alternate ECN semantics 
is already recommended in RFC4774, so the IETF would not really need to 
do anything to get L4S started.

The IETF might want to head-off possible interop problems by assigning a 
global-scope DSCP for the alternate ECN semantics, which we all know are 
in short supply. If we did use a global-scope DSCP it would be solely 
for a migration period, which is a double-edged sword:
* if the migration period were truly short-lived, the global codepoint 
would become available soon.
* if the migration period took longer, fears that burning a global DSCP 
for a just a brief migration period would have proved unfounded.

If local-only L4S usage became widespread, it /could/ be used between 
domains by simply ignoring the DSCP and only using ECN as the 
classifier. But that all depends what the take-up of classic ECN is in 
the meantime, and whether all the major classic ECN host implementations 
migrate to L4S. (In the TCP Prague Bar BoF, Andrew McGregor made the 
point that all the major OS developers who control what nearly all the 
Internet's traffic looks like were in the room.) Then the IETF could 
come along afterwards and standardise the new ECN semantics.

> and nobody's crystal-ball is good enough to see beyond that.
Deployment of something that enables new valuable apps and products can 
take 2yrs.

Nonetheless, your general point is true.
> RFC 3168 behavior of "same as drop".
<Flippant> According to measurement studies RFC3168 behaviour is 
currently entirely eradicated. At least at such a low level that the two 
CE packets that seemed to exhibit the behaviour are probably a symptom 
of bugs.
</Flippant>

<Seriously>I know what you mean: RFC3168 behaviour is latent in 
ECN-enabled servers waiting for a client request.

I have proposed that L4S behaviour is associated with e2e negotiation of 
new Accurate ECN semantics. Nonetheless, even if an L4S client attempts 
to negotiate AccECN with a server, if the server only supports classic 
ECN, the session should{Note 1} fall back to classic ECN. So we will 
need to distinguish classic ECN from L4S ECN... unless we all agree that 
AccECN must fall back to drop, even if the other end says it supports 
classic ECN.

{Note 1} AccECN is yet to be specified by the IETF, but this is the 
current thinking, which seems reasonable.
</Seriously>

> Furthermore, L4S _can't_
No-one is questioning that behaviour on drop needs to stay as in Reno or 
Cubic. The question is only over whether behaviour in response to ECN-CE 
should be distinct from drop behaviour.

> - L4S forwarders could change ECT(1) to ECT(0) (or vice-versa?),
The latter doesn't work, I'm afraid. Reason:
* If all buffers on a path (say X, Y, Z) classify L4S and Classic by 
ECT(1) and ECT(0) resp.,
* and if buffer X indicates L4S congestion by changing some L4S packets 
from ECT(1) to ECT(0)
* then at subsequent buffers on the path (Y or Z), the L4S packets that 
X remarked to ECT(0) will get classified into the Classic queue at Y and Z.

Result: a proportion L4S packets will get 'demoted' into low latency 
queues, introducing intermittent re-ordering delay, thus increasing the 
effective delay of the low latency L4S service to that of the classic 
queues.

> CE-marked when it arrives.
I didn't intend to say anything about whether actions are on entry or on 
exit to the queue - I don't think that's relevant here.
Again, I was thinking about the problem of one queue remarking some 
packets in both L4S and Classic ECN to CE, then how those CE packets 
would get classified at subsequent queues on the path. I'll try again 
with a picture...

,------------. ,------------. ,---------------. ,------------.
L4S source --> ECT(1)\    |      ECT(1)|-->| L4S      |\CE & ECT(1) |    
CE & ECT(1)|==>|   L4S      |
                       \   |  L3        | `------------' \ | L3       
      |   `------------'
                        ==>| Classifier|     Bottleneck    
==========>|Classifier    |
                       /   |            | ,------------. / | 
               | ,------------.
Classic src--> ECT(0)/    |      ECT(0)|-->| Classic   |/CE & ECT(0) | 
ECT(0)|-->|  Classic   |
                           `------------' 
`------------'`---------------' `------------'

---

From: "De Schepper, Koen (Koen)" <[email-masked]>
Date: Tue, 4 Aug 2015 11:46:39 +0000

Dave,

> When switched to fq_codel it showed 20ms delay for both queues. This
> struck me as overlarge. 

This is due to the low base RTT (7ms) and the number of flows. The FQ_Codel qsize for DCTCP flows are around double the size of Cubic flows. With a base RTT of 7ms, Codel was not holding on its target RTT (for Cubic) of 5ms for more than 2 flows. For 20 flows the Cubic queues became on average around 10ms (tail up to 30ms) and the DCTCP queues around 20ms (tail up to 60ms).

> A common problem is that perhaps you had
> ethernet offloads still on for GRO/GSO/TSO on the aqm hop?

Good that you mention this, because we didn't realize that the NICs with GRO on the router are gluing incoming packets together again. All our experiments were with GRO/GSO/TSO on.

Doing some quick tests, GRO/GSO/TSO on or off had no noticeable effect on the FQ_Codel Q-sizes. So the results were only due to the higher level of congestion (lower RTTs). So in fact, at a certain extent, Codel is also trading loss with delay, as we proposed for Curvy RED, only at a much higher loss level. 

GRO at the router has a noticeable effect for the very low queues when DCTCP runs alone. It glues the initial window packets together and makes them send at line speed (1Gbps) at the NIC. I think it is due to this burstyness that we get a slightly bigger average queuing delay. Also, when DCTCP runs alone, we have a simple packet level threshold (mark when q size > 5 packets). In that case packets with a size of 15K (are the biggest that we saw) will get through without marking. We plan to work on alternatives for the fallback AQM (at least byte based, or delay based). But for the Linux implementation we might have to partially mark (split?) these glued packets.

Koen.

> -----Original Message-----
> From: Dave Taht [mailto:[email-masked]]
> Sent: dinsdag 28 juli 2015 11:37
> To: De Schepper, Koen (Koen)
> Cc: Scheffenegger, Richard; [email-masked]
> Subject: Re: [aqm] Minutes of the AQM WG session
> 
> On Tue, Jul 28, 2015 at 10:57 AM, De Schepper, Koen (Koen)
> <[email-masked]> wrote:
> > Hi Dave,
> >
> > 2) Yes, we used only ECN to distinguish between classic and L4S TCP.
> As Richard already mentioned, we hope indeed that the final deployment
> could be that simple too. The benefits of using ECN combined with 1/p
> congestion controllers is so big that we hope nobody will want anything
> less ;-). Also, (see CurvyRED and PI^2 in my backup slides) AQMs that
> think twice to drop "at-the-output" are much simpler to implement and
> understand (due to linear relation between p and number of flows), and
> are immediately compatible with 1/p congestion controllers (by skipping
> the square-at-the-output). It might be useful to think about a curvy or
> squared CoDel/Cake (note the intentionally missing "fq_") as well...
> (to improve dctcp support as mentioned in your topic 3). As far as I
> understand the new option in the *FQ_*codel is disabling codel
> completely and applies also a sojourn time based step function for ECN
> (which makes sense for the fq_ version, where every flow has its own
> queue & AQM, and RR-scheduling is used for fairness).
> 
> I have not yet had time to push the ce_threshold idea into toke's
> testbed. You can infer from the commit message where it is being tried
> in production, and the commit message has the only public data so far
> on how it works:
> 
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
> id=80ba92fa1a92dea128283f69f55b02242e213650
> 
> and I believe it entered the linux mainline as of linux 4.1
> 
> As for thinking about much of this, curvyred, etc, more deeply I
> prefer to have patches and data to play with first.
> 
> > 4) After I presented in the ICCRG, Daniel Borkmann suggested trying
> the dctcp_clamp_alpha_on_loss option, but it didn't work. The problem
> is probably that the alpha, once set, is moving quickly to a lower
> value before it is applied (at the end of the RTT). So it's a matter of
> keeping the state in another variable and reset that one, once the
> alpha=1 is set just before it is applied to reduce the window. Daniel
> Borkmann, Florian Westfall and Glenn Judd are aware of the
> dctcp_clamp_alpha_on_loss issue.
> 
> Good, then I shall cease to worry about it. Still bothersome.
> 
> > 5) Actually Curvy RED is used to control the classic queue and couple
> it (non-curved) to the DCTCP traffic (for the throughput fairness). The
> size of the L4S queue is normally 0, only initial windows and other
> flow variations above the bottleneck links are shortly queued in the
> L4S queue. If no classic traffic is available, our experiments used 5
> packets (= 1.6ms using the DCTCP step function marking threshold).
> 
> The demo showed the dctcp queue having < 5ms delay, and the "classic"
> queue having 42 ms delay.
> 
> When switched to fq_codel it showed 20ms delay for both queues. This
> struck me as overlarge. A common problem is that perhaps you had
> ethernet offloads still on for GRO/GSO/TSO on the aqm hop? Codel's
> (and pie and qfq!) original deployment was designed and tested with
> offloads off, mostly, and we strenously encouraged experimenters to
> turn it off - so much so that we did not do enough testing with it on.
> Recently we were blindsided by the armada 385 chip (used by a whole
> bunch of new high speed home routers) doing very aggressive GRO - on
> everything - I had never seen it crack 24k before, it routinely
> cracked 64k and that led to some headaches.
> 
> Codel had the maxpacket concept in it which was designed to operate
> correctly at a single MTU, not 64K!
> 
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?
> id=a5d28090405038ca1f40c13f38d6d4285456efee
> 
> I do not know how to fix it in red or pie at the moment. pie's
> estimator is byte based, but there may be other issues. So it would be
> my hope that your demo would run better against fq_codel with offloads
> off and/or this patch applied. ?
> 
> Cake has "peeling", which turns the up to 64k superpackets back into
> appropriately-sized packets for the afforded workload. So has TBF, in
> a more limited fashion. HTB does not, presently. Probably we will add
> peeling features to htb at some point.
> 
> 
> >
> > 6) Would be good, as hashed FQ gives a surprisingly (for us at least)
> high number of collisions.
> 
> set associative hashing in cake *is* turning in pretty great and
> interesting results, so far. and computationally cheap. set
> associativity, of course, is a very common thing in conventional cpu
> cache design, it is surprising I have not seen it in queue design as
> well before now (anyone got any prior cites?)
> 
> I don't know what a "surprisingly" high number of collisions is, the
> math is straightforward. Admittedly most of the workloads were tons of
> short flows against a very few number of big flows, and only recently
> (with cake) did we start trying to tackle torrent-like flows again.
> 
> >
> > 7) Nothing decided yet (see topic 2).
> 
> l.
> 
> > Regards,
> > Koen.
> >
> >
> >> -----Original Message-----
> >> From: aqm [mailto:[email-masked]] On Behalf Of Dave Taht
> >> Sent: donderdag 23 juli 2015 13:24
> >> To: Scheffenegger, Richard
> >> Cc: [email-masked]
> >> Subject: Re: [aqm] Minutes of the AQM WG session
> >>
> >> 1) in hard delay targets, I am credited with what matt mathis said
> >> (not that I disagree).
> >>
> >> 2) In the dual queue thing I had noted that distinguishing between
> the
> >> two queues based solely on the presence of ecn capability and then
> >> using dctcp was non-sensical as plenty of other things like cubic
> >> would also end up with ecn enabled.
> >>
> >> 3) and for the record, fq_codel as it exists today works reasonably
> >> well when hammered with dctcp + ecn, cubic + ecn, and and any other
> >> stuff without markings with the defaults. It may not give the
> desired
> >> reduction in individual queue length - and cubic vs dctcp will do
> >> badly against each other on a hash collision - but it is, at least,
> >> "mostly safe" were some sysadmin finger foo things and push dctcp
> (or
> >> some other non traditional cc) out on the world internet against
> >> fq_codel'd ecn-enabled systems.
> >>
> >> 4) If you can't tell, it *really bothers me* when someone reports a
> >> bug in dctcp - at ietf - rather than through proper channels -
> >> particularly when it leads to evil behavior on loss, and yet I have
> no
> >> patches submitted nor means to reproduce it.  PLEASE GET A PATCH OUT
> >> to the netdev maintainers, it will be immediately put into the next
> >> release of linux AND backported to the linux stable releases.
> >>
> >> What I see is dctcp_clamp_alpha_on_loss not defaulting to on, which
> >> based on the comments, should default to on. Is there somewhere else
> >> this is busted?
> >>
> >> static void dctcp_state(struct sock *sk, u8 new_state)
> >>
> >> {
> >>
> >>         if (dctcp_clamp_alpha_on_loss && new_state == TCP_CA_Loss) {
> >>
> >>                 struct dctcp *ca = inet_csk_ca(sk);
> >>
> >>
> >>                 /* If this extension is enabled, we clamp
> dctcp_alpha
> >> to
> >>
> >>                  * max on packet loss; the motivation is that
> >> dctcp_alpha
> >>
> >>                  * is an indicator to the extend of congestion and
> >> packet
> >>
> >>                  * loss is an indicator of extreme congestion;
> setting
> >>
> >>                  * this in practice turned out to be beneficial, and
> >>
> >>                  * effectively assumes total congestion which
> reduces
> >> the
> >>
> >>                  * window by half.
> >>
> >>                  */
> >>
> >>
> >> 5) Missing from the preso was the actual dual queue length, just a
> >> comparison (cool demo tho!) - you get what queue length using curvy
> >> red?
> >>
> >> 6) cake, of course, gets to 100s of flows without hash collisions.
> >>
> >> 7) As I missed the tcp-prague discussion, is the proposal to reserve
> >> ect(1) - the former nonce bit - for dctcp? or some other diffserv or
> >> flowheader marking?
> >>
> >> On Thu, Jul 23, 2015 at 8:56 AM, Scheffenegger, Richard
> <[email-masked]>
> >> wrote:
> >> > Hi,
> >> >
> >> > Thanks to David Schinazi for serving as the notes taker;
> >> >
> >> > I've uploaded the minutes
> >> https://www.ietf.org/proceedings/93/minutes/minutes-93-aqm
> >> >
> >> > If you made a statement during the session, please review that the
> >> notes capture the essence of what you wanted to convey!
> >> >
> >> > Also, one name is missing (and I'm unable to replay the meetecho
> >> recording currently) in the minutes, if you know who made that
> comment
> >> please inform us chairs.
> >> >
> >> > Thanks,
> >> >   Richard (aqm chair)
> >> >
> >> Dave Täht
> >> worldwide bufferbloat report:
> >> http://www.dslreports.com/speedtest/results/bufferbloat
> >> And:
> >> What will it take to vastly improve wifi for everyone?
> >> https://plus.google.com/u/0/explore/makewififast
> >>
> Dave Täht
> worldwide bufferbloat report:
> http://www.dslreports.com/speedtest/results/bufferbloat
> And:
> What will it take to vastly improve wifi for everyone?
> https://plus.google.com/u/0/explore/makewififast

---

From: Bob Briscoe <[email-masked]>
Date: Tue, 04 Aug 2015 13:42:21 +0100

John,

[Ignore previous - clicked 'send' too early]

Again, I was thinking about the problem of one queue remarking some 
packets in both L4S and Classic ECN to CE, then how those CE packets 
would get classified at subsequent queues on the path.

With solely L3 classification, all CE packets would have to be 
classified as L4S, including those from Classic ECN flows marked as CE 
earlier on the path.

(I tried ASCII art, but email mangled it.)

As I said, mis-classifying is not a problem as long as it is arranged to 
be from the worse to the better queue.

> where it was approved). Mea culpa!

I missed it too. During the WG process in AVT I wrote a long review 
including concern about the ECT(1) parts. The authors took lots of my 
concerns into account, but I wasn't awake to notice that they had left 
in the ECT(1) parts when it went to WG last call and subsequently to IESG.

>     To deprecate ECN Nonce, we'd need to UPDATE RFC 6679

What about existing implementations of 6679?

> as well as

VCP: Xia, Y., Subramanian, L., Stoica, I. & Kalyanaraman, S., "One more 
bit is enough," Proc. ACM SIGCOMM'05, Computer Communication Review 
35(4):37--48 In: SIGCOMM '05: Proceedings of the 2005 conference on 
Applications, technologies, architectures, and protocols for computer 
communications Vol.35 No.4 pp.37-48 ACM Press (2005)
<http://doi.acm.org/10.1145/1080091.1080098>

Kunniyur, S.S., "AntiECN Marking: A Marking Scheme for High Bandwidth 
Delay Connections ," In: Proc. ICC'03 IEEE (May 2003)
<http://repository.upenn.edu/cgi/viewcontent.cgi?article=1053&context=ese_papers>

>     (BTW, I think L4S could be _very_ helpful for "speeding up" slow-start.)

Indeed. I have ideas myself too (unsurprisingly).

I think the only workable schemes ought not to rely on a new packet 
marking, but we might not be able to achieve this ideal, so I would be 
wary of burning the last ECN codepoint 'just' to distinguish between 
something we want to get to, and something we know will be legacy one day.

>     I don's see L4S as useful in "local-network-only" mode.

Please listen to the ISPs who have seen the DCttH demo. They have their 
own ideas of what is 'useful' to them.

> where _only_ the first hop can experience congestion.

If an ISP were using L4S for delivery from its local caches and servers, 
it would be doing the bleaching itself at its border. It would not be 
overly concerned if its bleaching prevented 'OTT' traffic  from using 
the L4S queues in the bottlenecks within its access network.

>     This is overly optimistic.

Again, in Prague people from the major OS developers were talking as if 
such optimism would not be so mad.

>>     might be needed for L4S in each DS class

Not necessarily. For instance, a separate L4S DSCP might only be needed 
to distinguish BE L4S from BE Classic.

Indeed, once L4S gives all traffic low latency, ISPs and enterprises 
don't necessarily need to distinguish between EF and AF, etc. Then an 
ISP or enterprise might make all its 'premium' (non-BE) traffic solely 
L4S without any need to distinguish a Classic subset.

> that... but for now, it _all_ gets bleached.)

Half true. Bilteral arrangements are starting to appear at some borders 
(DT is leading the way). Where there is a global DSCP, this helps make 
such bilateral arrangements simple to administer and deploy.

(BTW, there is no point writing RFCs that dictate what operational 
policies must be used.)

>     We certainly could Experiment with that; but I'm very pessimistic.

Unfortunately, I think the IETF has to make a choice.

I overheard Spencer Dawkins say: "The IETF is only good at doing the 
right thing when there is no other choice."

>     OTOH, Experimenting with ECT(1) seems likely to work. IMHO...

I already said that either could 'work'. I tend to agree that using DSCP 
/and/ ECN is likely to fall foul of the union of the sets of problems 
that both suffer from.

I'm currently in information gathering and promulgation mode, not 
decision mode - I might switch to preferring ECT(1) depending on what we 
find out.

My concern is primarily about burning the last ECN codepoint merely for 
a transition arrangement.

Bob

---

From: Jonathan Morton <[email-masked]>
Date: Tue, 4 Aug 2015 15:56:56 +0200

> On 4 Aug, 2015, at 14:42, Bob Briscoe <[email-masked]> wrote:
> 
> <Seriously>I know what you mean: RFC3168 behaviour is latent in ECN-enabled servers waiting for a client request.
> 
> I have proposed that L4S behaviour is associated with e2e negotiation of new Accurate ECN semantics. Nonetheless, even if an L4S client attempts to negotiate AccECN with a server, if the server only supports classic ECN, the session should{Note 1} fall back to classic ECN. So we will need to distinguish classic ECN from L4S ECN... unless we all agree that AccECN must fall back to drop, even if the other end says it supports classic ECN.
> 
> {Note 1} AccECN is yet to be specified by the IETF, but this is the current thinking, which seems reasonable.
> </Seriously>
> 
>> Furthermore, L4S _can't_
> No-one is questioning that behaviour on drop needs to stay as in Reno or Cubic. The question is only over whether behaviour in response to ECN-CE should be distinct from drop behaviour.

It sounds as though ELR and L4S are aiming at the same sort of problem space.  However, I think ELR has some advantages:

1) ELR doesn’t require changes to the response to CE.  In fact, it expects that the RFC3168 behaviour, or something similar to it, remains in place.

2) ELR doesn’t rely on a robust distinction between ELR and classic ECN traffic, and does not require that they be separately queued.  This follows naturally from point 1.

3) ELR uses all three ECN codepoints (omitting Not-ECT) for signalling.  CE is used for urgent reductions, and the distinction between ECT(0) and ECT(1) is used for finer-grained information; losing this information en route is not fatal.  This, I think, is a natural use of the remaining codepoint.

At some point, I’ll get around to writing it up more formally.

 - Jonathan Morton

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 06 Aug 2015 09:56:46 +0100

A few comments in-line, as someone interested in taking things forward:

>> very significant challenge. :^(
However, it's not the ECT(1) codepoint that would be problematic, but 
more the fact that there is only a single CE-mark. Hence, if one 
redefined ECT(1) to indicate a different use of ECN in the network, then 
it would likely need to result in CE-marking. Such packets would be 
indistinguishable from packets marked from ECT(0). All CE-marks would 
then need to be queued (and counted) the accounted for the same in the 
marking algorithms. As far as I understand, that isn't what the 
dual-queue method was wanting.

> DSCPs are only poison in standards. As you say, ISPs use DSCPs for 
> many internal services already.
>
There may be signs of hope here for increased use of network-wide 
markings (ever optimistic).
> so this would be very "useful". Using a DSCP for alternate ECN 
> semantics is already recommended in RFC4774, so the IETF would not 
> really need to do anything to get L4S started.
>
> The IETF might want to head-off possible interop problems by assigning 
> a global-scope DSCP for the alternate ECN semantics, which we all know 
> are in short supply. If we did use a global-scope DSCP it would be 
> solely for a migration period, which is a double-edged sword:
> * if the migration period took longer, fears that burning a global 
> DSCP for a just a brief migration period would have proved unfounded.
>
> the meantime, and whether all the major classic ECN host 
> implementations migrate to L4S. (In the TCP Prague Bar BoF, Andrew 
> McGregor made the point that all the major OS developers who control 
> what nearly all the Internet's traffic looks like were in the room.) 
> Then the IETF could come along afterwards and standardise the new ECN 
> semantics.
>
>
> Deployment of something that enables new valuable apps and products 
> can take 2yrs.
>
> currently entirely eradicated. At least at such a low level that the 
> two CE packets that seemed to exhibit the behaviour are probably a 
> symptom of bugs.
> </Flippant>
>
Even this remark, I can't resist a response (although I see this also is 
discussed again later):

That was not what our measurements looked for - the measurements we did 
with Brian, Mirja and Richard did not try to induce congestion. 
Specifically they were short test sequences to test ability to pass and 
negotiate the ECN codepoint usage. We hence don't *know* whether any 
network devices had ECN-marking support enabled, we just know that if 
they had, these devices didn't experience congestion at the time of the 
test. We also know they didn't experience (congestive) loss, for what 
little that's worth.

My point is: I *do* have ECN enabled in at least one of my home routers, 
presently it's not a bottleneck, and hence I see no CE-marks, and if it 
becomes congested I know it will use FQ-Codel to mark (whatever way that 
does) - I can't control this as an endpoint - and in general as a 
transport I can't find out what ECN-marking has deployed along my path, 
if there happens to be any.
> ECN-enabled servers waiting for a client request.
So: It's going to be hard to require (MUST) a new replacement marking in 
network devices. It may be possible to allow one, and this would be 
great if there were appropriate mechanisms at the endpoints to detect 
this and do something sensible.
> I have proposed that L4S behaviour is associated with e2e negotiation 
> of new Accurate ECN semantics. Nonetheless, even if an L4S client 
> attempts to negotiate AccECN with a server, if the server only 
> supports classic ECN, the session should{Note 1} fall back to classic 
> ECN. So we will need to distinguish classic ECN from L4S ECN... unless 
> we all agree that AccECN must fall back to drop, even if the other end 
> says it supports classic ECN.
>
> No-one is questioning that behaviour on drop needs to stay as in Reno 
> or Cubic. The question is only over whether behaviour in response to 
> ECN-CE should be distinct from drop behaviour.
>
I'd agree. Both ABE (as proposed in TCPM) and DCTP (also TCPM) would 
change this, as I believe should any new TCPM-defined methods based on 
AccECN.
> * and if buffer X indicates L4S congestion by changing some L4S 
> packets from ECT(1) to ECT(0)
> * then at subsequent buffers on the path (Y or Z), the L4S packets 
> that X remarked to ECT(0) will get classified into the Classic queue 
> at Y and Z.
>
> queues, introducing intermittent re-ordering delay, thus increasing 
> the effective delay of the low latency L4S service to that of the 
> classic queues.
>
Sadly, I think probably true for any general deployment - this I think 
was one of the motivations for marking this method with a different DSCP 
when used internally within a provider network when RFC4774 was discussed.
> I didn't intend to say anything about whether actions are on entry or 
> on exit to the queue - I don't think that's relevant here.
>
> As I said, mis-classifying is not a problem as long as it is arranged 
> to be from the worse to the better queue.
>
>
>>
> VCP: Xia, Y., Subramanian, L., Stoica, I. & Kalyanaraman, S., "One 
> more bit is enough," Proc. ACM SIGCOMM'05, Computer Communication 
> Review 35(4):37--48 In: SIGCOMM '05: Proceedings of the 2005 
> conference on Applications, technologies, architectures, and protocols 
> for computer communications Vol.35 No.4 pp.37-48 ACM Press (2005)
> Please listen to the ISPs who have seen the DCttH demo. They have 
> their own ideas of what is 'useful' to them.
>
>>
> If an ISP were using L4S for delivery from its local caches and 
> servers, it would be doing the bleaching itself at its border. It 
> would not be overly concerned if its bleaching prevented 'OTT' 
> traffic  from using the L4S queues in the bottlenecks within its 
> access network.
>
>>
> Again, in Prague people from the major OS developers were talking as 
> if such optimism would not be so mad.
>
>>
> Not necessarily. For instance, a separate L4S DSCP might only be 
> needed to distinguish BE L4S from BE Classic.
>
> Half true. Bilteral arrangements are starting to appear at some 
> borders (DT is leading the way). Where there is a global DSCP, this 
> helps make such bilateral arrangements simple to administer and deploy.
>
> I already said that either could 'work'. I tend to agree that using 
> DSCP /and/ ECN is likely to fall foul of the union of the sets of 
> problems that both suffer from.
>
> decision mode - I might switch to preferring ECT(1) depending on what 
> we find out.
>
> My concern is primarily about burning the last ECN codepoint merely 
> for a transition arrangement.
>
I'd certainly need more persausion to use ECT(1) for this, given all 
that is above. A new DSCP requires a lot of consensus if it is to be 
standards action, and less if local use. To me at least, I think this is 
worth exploring, if the arguments can be presented simply.
> Bob
Gorry

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 06 Aug 2015 15:23:30 +0100

Gorry,

On 06/08/15 09:56, Gorry Fairhurst wrote:
> redefined ECT(1) to indicate a different use of ECN in the network, 
> then it would likely need to result in CE-marking. Such packets would 
> be indistinguishable from packets marked from ECT(0). All CE-marks 
> would then need to be queued (and counted) the accounted for the same 
> in the marking algorithms. As far as I understand, that isn't what the 
As I explained later, I believe a dual-use CE marking is not a problem. 
As long as CE is always promoted (to L4S), not demoted (to Classic), it 
is OK.

I.e., if some CE packets had started life as ECT(0) (Classic), 
misclassifying them into the L4S queue would only forward them faster, 
not slower. They would arrive earlier out of order than the rest of the 
Classic flow, but earlier is OK, and it would be far less re-ordering 
than retransmission already causes today.

>> from within the same ISP as the user is connected to (esp using 
>> CDNs), so this would be very "useful". Using a DSCP for alternate ECN 
>> The IETF might want to head-off possible interop problems by 
>> assigning a global-scope DSCP for the alternate ECN semantics, which 
>> we all know are in short supply. If we did use a global-scope DSCP it 
>> would be solely for a migration period, which is a double-edged sword:
>> * if the migration period were truly short-lived, the global 
>> codepoint would become available soon.
>> classifier. But that all depends what the take-up of classic ECN is 
>> in the meantime, and whether all the major classic ECN host 
> Even this remark, I can't resist a response (although I see this also 
> is discussed again later):
>
> That was not what our measurements looked for - the measurements we 
> did with Brian, Mirja and Richard did not try to induce congestion. 
> Specifically they were short test sequences to test ability to pass 
> and negotiate the ECN codepoint usage. We hence don't *know* whether 
> any network devices had ECN-marking support enabled, we just know that 
> if they had, these devices didn't experience congestion at the time of 
> the test. We also know they didn't experience (congestive) loss, for 
> what little that's worth.
>
> My point is: I *do* have ECN enabled in at least one of my home 
> routers, presently it's not a bottleneck, and hence I see no CE-marks, 
> and if it becomes congested I know it will use FQ-Codel to mark 
> (whatever way that does) - I can't control this as an endpoint - and 
> in general as a transport I can't find out what ECN-marking has 
> deployed along my path, if there happens to be any.
I prefer to distinguish between what relatively small numbers of 
enlightened individual engineers might have deployed and what is 
deployed by large organisations on behalf of 'the masses'.

I am not saying we should screw-over those enlightened engineers. I'm 
just saying that we don't need to worry about backwards compatibility 
for individual early adopters, because
a) they are likely to keep up to date
b) if they don't, they are likely to understand that they need to update 
when they experience problems.

AFAICT, classic ECN is not in mainline Linux and not on-by-default in 
any other router OS
So everyone needs to think really, really carefully before they allow that.

> So: It's going to be hard to require (MUST) a new replacement marking 
> in network devices. It may be possible to allow one, and this would be 

To clarify, I don't think you mean that an RFC will need to say network 
devices MUST respond to this new replacement marking. Rather it will 
need to say network devices that respond to classic ECT MUST also check 
for this new replacement marking.

And actually, I think this could be a SHOULD. As Matt Mathis said, the 
consequences of not complying are no worse than the rate mismatch seen 
today between two Classic TCP flows with significantly different RTTs.

The additional problem here (at least for purists) is that RFCs are 
intended to be followed by developers. It is not realistic to expect 
equipment operators to follow RFCs. So, if an operator already has a 
router with ECN implemented (e.g. Cisco), we are expecting to ask them 
not to turn ECN on unless they update the code to distinguish Classic 
ECN from the new replacement identifier.

I call this as a problem for purists because it's only of concern for 
someone who imagines it is important to keep the RFC series completely 
logically consistent. In practice, we can discount the chance that an 
operator is going to pick up a router and randomly turn on Classic ECN 
without finding out the latest position, given none have turned on ECN 
since it was standardised 14 years ago.

>> ECN. So we will need to distinguish classic ECN from L4S ECN... 
>> unless we all agree that AccECN must fall back to drop, even if the 
>> other end says it supports classic ECN.
>>
> was one of the motivations for marking this method with a different 
> DSCP when used internally within a provider network when RFC4774 was 
> discussed.
>>>
>> classified as L4S, including those from Classic ECN flows marked as 
>> CE earlier on the path.
>>
>> concerns into account, but I wasn't awake to notice that they had 
>> left in the ECT(1) parts when it went to WG last call and 
>> subsequently to IESG.
>>
>>>
>> conference on Applications, technologies, architectures, and 
>> protocols for computer communications Vol.35 No.4 pp.37-48 ACM Press 
>> (2005)
>> marking, but we might not be able to achieve this ideal, so I would 
>> be wary of burning the last ECN codepoint 'just' to distinguish 
>> between something we want to get to, and something we know will be 
>> legacy one day.
>>
>>
>>>
>>>> *DSCP**
>> ISP or enterprise might make all its 'premium' (non-BE) traffic 
>> solely L4S without any need to distinguish a Classic subset.
>>
>>>
>>>> b) DS is not end-to-end
> standards action, and less if local use. To me at least, I think this 
> is worth exploring, if the arguments can be presented simply.

We would still require the RFC language discussed earlier: an 
ECN-enabled AQM SHOULD check for the new identifier (DSCP in this case).

Thanks

Bob

---

From: Michael Welzl <[email-masked]>
Date: Thu, 6 Aug 2015 17:18:05 +0200

Hi Bob, Gorry, all,

A few comments in line:

> On 6. aug. 2015, at 16.23, Bob Briscoe <[email-masked]> wrote:
> 
> Gorry,
> 
>>> There was much discussion about this identifier issue in Prague. Many people are well-aware that the sticking point is availability of an identifier, and its associated politics.
>>> 
>>> 
>> However, it's not the ECT(1) codepoint that would be problematic, but more the fact that there is only a single CE-mark. Hence, if one redefined ECT(1) to indicate a different use of ECN in the network, then it would likely need to result in CE-marking. Such packets would be indistinguishable from packets marked from ECT(0). All CE-marks would then need to be queued (and counted) the accounted for the same in the marking algorithms. As far as I understand, that isn't what the dual-queue method was wanting.
> As I explained later, I believe a dual-use CE marking is not a problem. As long as CE is always promoted (to L4S), not demoted (to Classic), it is OK.
> 
> I.e., if some CE packets had started life as ECT(0) (Classic), misclassifying them into the L4S queue would only forward them faster, not slower. They would arrive earlier out of order than the rest of the Classic flow, but earlier is OK, and it would be far less re-ordering than retransmission already causes today.
> 
>>> DSCPs are only poison in standards. As you say, ISPs use DSCPs for many internal services already.
>>> 
>> There may be signs of hope here for increased use of network-wide markings (ever optimistic).
>>> (BTW, the request for a global scope DSCP for less-than-BE is very different to this case, because bleaching LBE (remarking to the all-zeros DSCP at borders) always /increases/ its priority.)
>>> 
>>> ISPs could go ahead with using a local DSCP now for L4S for their own premium services. A large proportion of traffic these days is served from within the same ISP as the user is connected to (esp using CDNs), so this would be very "useful". Using a DSCP for alternate ECN semantics is already recommended in RFC4774, so the IETF would not really need to do anything to get L4S started.
>>> 
>>> The IETF might want to head-off possible interop problems by assigning a global-scope DSCP for the alternate ECN semantics, which we all know are in short supply. If we did use a global-scope DSCP it would be solely for a migration period, which is a double-edged sword:
>>> * if the migration period were truly short-lived, the global codepoint would become available soon.
>>> * if the migration period took longer, fears that burning a global DSCP for a just a brief migration period would have proved unfounded.
>>> 
>>> If local-only L4S usage became widespread, it /could/ be used between domains by simply ignoring the DSCP and only using ECN as the classifier. But that all depends what the take-up of classic ECN is in the meantime, and whether all the major classic ECN host implementations migrate to L4S. (In the TCP Prague Bar BoF, Andrew McGregor made the point that all the major OS developers who control what nearly all the Internet's traffic looks like were in the room.) Then the IETF could come along afterwards and standardise the new ECN semantics.
>>> 
>>> 
>>> Deployment of something that enables new valuable apps and products can take 2yrs.
>>> 
>>> <Flippant> According to measurement studies RFC3168 behaviour is currently entirely eradicated. At least at such a low level that the two CE packets that seemed to exhibit the behaviour are probably a symptom of bugs.
>>> </Flippant>
>>> 
>> Even this remark, I can't resist a response (although I see this also is discussed again later):
>> 
>> That was not what our measurements looked for - the measurements we did with Brian, Mirja and Richard did not try to induce congestion. Specifically they were short test sequences to test ability to pass and negotiate the ECN codepoint usage. We hence don't *know* whether any network devices had ECN-marking support enabled, we just know that if they had, these devices didn't experience congestion at the time of the test. We also know they didn't experience (congestive) loss, for what little that's worth.
>> 
>> My point is: I *do* have ECN enabled in at least one of my home routers, presently it's not a bottleneck, and hence I see no CE-marks, and if it becomes congested I know it will use FQ-Codel to mark (whatever way that does) - I can't control this as an endpoint - and in general as a transport I can't find out what ECN-marking has deployed along my path, if there happens to be any.
> I prefer to distinguish between what relatively small numbers of enlightened individual engineers might have deployed and what is deployed by large organisations on behalf of 'the masses'.
> 
> I am not saying we should screw-over those enlightened engineers. I'm just saying that we don't need to worry about backwards compatibility for individual early adopters, because
> b) if they don't, they are likely to understand that they need to update when they experience problems.
> 
> AFAICT, classic ECN is not in mainline Linux and not on-by-default in any other router OS

"really, really carefully" only because that would "burn" the codepoint rather than reserving it for a scheme that requires installing a dual Queue in routers AND precise feedback from receivers AND a sender behaviour that (so far) isn't compatible with any other queue doing CE-marking? That's asking a bit much I think.

>> So: It's going to be hard to require (MUST) a new replacement marking in network devices. It may be possible to allow one, and this would be great if there were appropriate mechanisms at the endpoints to detect this and do something sensible.
> 
> To clarify, I don't think you mean that an RFC will need to say network devices MUST respond to this new replacement marking. Rather it will need to say network devices that respond to classic ECT MUST also check for this new replacement marking.
> 
> And actually, I think this could be a SHOULD. As Matt Mathis said, the consequences of not complying are no worse than the rate mismatch seen today between two Classic TCP flows with significantly different RTTs.

I challenge that. Why would that be the case? A DCTCP-like flow will almost not react in response to a single CE-mark in a window of packets, whereas any other sender sees that as a congestion event which normally comes with a multiplicative congestion control reaction. I agree that we shouldn't be religious about "TCP-friendliness" but this is not quite the same. A single DCTCP flow can certainly completely starve a large number of "normal" ECN-capable senders.

> The additional problem here (at least for purists) is that RFCs are intended to be followed by developers. It is not realistic to expect equipment operators to follow RFCs. So, if an operator already has a router with ECN implemented (e.g. Cisco), we are expecting to ask them not to turn ECN on unless they update the code to distinguish Classic ECN from the new replacement identifier.
> 
> I call this as a problem for purists because it's only of concern for someone who imagines it is important to keep the RFC series completely logically consistent. In practice, we can discount the chance that an operator is going to pick up a router and randomly turn on Classic ECN without finding out the latest position, given none have turned on ECN since it was standardised 14 years ago.
> 
>> I'd agree. Both ABE (as proposed in TCPM) and DCTP (also TCPM) would change this, as I believe should any new TCPM-defined methods based on AccECN.
>>>>> *ECT(1) **
>>> * If all buffers on a path (say X, Y, Z) classify L4S and Classic by ECT(1) and ECT(0) resp.,
>>> * and if buffer X indicates L4S congestion by changing some L4S packets from ECT(1) to ECT(0)
>>> * then at subsequent buffers on the path (Y or Z), the L4S packets that X remarked to ECT(0) will get classified into the Classic queue at Y and Z.
>>> 
>>> Result: a proportion L4S packets will get 'demoted' into low latency queues, introducing intermittent re-ordering delay, thus increasing the effective delay of the low latency L4S service to that of the classic queues.
>>> 
>> Sadly, I think probably true for any general deployment - this I think was one of the motivations for marking this method with a different DSCP when used internally within a provider network when RFC4774 was discussed.
>>>> 
>>> I didn't intend to say anything about whether actions are on entry or on exit to the queue - I don't think that's relevant here.
>>> 
>>> Again, I was thinking about the problem of one queue remarking some packets in both L4S and Classic ECN to CE, then how those CE packets would get classified at subsequent queues on the path.
>>> 
>>> With solely L3 classification, all CE packets would have to be classified as L4S, including those from Classic ECN flows marked as CE earlier on the path.
>>> 
>>> As I said, mis-classifying is not a problem as long as it is arranged to be from the worse to the better queue.
>>> 
>>> 
>>>> 
>>> I missed it too. During the WG process in AVT I wrote a long review including concern about the ECT(1) parts. The authors took lots of my concerns into account, but I wasn't awake to notice that they had left in the ECT(1) parts when it went to WG last call and subsequently to IESG.
>>> 
>>>> 
>>> VCP: Xia, Y., Subramanian, L., Stoica, I. & Kalyanaraman, S., "One more bit is enough," Proc. ACM SIGCOMM'05, Computer Communication Review 35(4):37--48 In: SIGCOMM '05: Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications Vol.35 No.4 pp.37-48 ACM Press (2005)
>>> Kunniyur, S.S., "AntiECN Marking: A Marking Scheme for High Bandwidth Delay Connections ," In: Proc. ICC'03 IEEE (May 2003)
>>> I think the only workable schemes ought not to rely on a new packet marking, but we might not be able to achieve this ideal, so I would be wary of burning the last ECN codepoint 'just' to distinguish between something we want to get to, and something we know will be legacy one day.
>>> 
>>> 
>>>> 
>>>>> *DSCP**
>>> Please listen to the ISPs who have seen the DCttH demo. They have their own ideas of what is 'useful' to them.
>>> 
>>>> 
>>> If an ISP were using L4S for delivery from its local caches and servers, it would be doing the bleaching itself at its border. It would not be overly concerned if its bleaching prevented 'OTT' traffic  from using the L4S queues in the bottlenecks within its access network.
>>> 
>>>> 
>>> Again, in Prague people from the major OS developers were talking as if such optimism would not be so mad.
>>> 
>>>> 
>>> Not necessarily. For instance, a separate L4S DSCP might only be needed to distinguish BE L4S from BE Classic.
>>> 
>>> Indeed, once L4S gives all traffic low latency, ISPs and enterprises don't necessarily need to distinguish between EF and AF, etc. Then an ISP or enterprise might make all its 'premium' (non-BE) traffic solely L4S without any need to distinguish a Classic subset.
>>> 
>>>> 
>>>>> b) DS is not end-to-end
>>> Half true. Bilteral arrangements are starting to appear at some borders (DT is leading the way). Where there is a global DSCP, this helps make such bilateral arrangements simple to administer and deploy.
>>> 
>>> (BTW, there is no point writing RFCs that dictate what operational policies must be used.)
>>> 
>>>> 
>>>>> *Summary**
>>> I overheard Spencer Dawkins say: "The IETF is only good at doing the right thing when there is no other choice."
>>> 
>>>> 
>>> I already said that either could 'work'. I tend to agree that using DSCP /and/ ECN is likely to fall foul of the union of the sets of problems that both suffer from.
>>> 
>>> I'm currently in information gathering and promulgation mode, not decision mode - I might switch to preferring ECT(1) depending on what we find out.
>>> 
>>> My concern is primarily about burning the last ECN codepoint merely for a transition arrangement.
>>> 
>> I'd certainly need more persausion to use ECT(1) for this, given all that is above. A new DSCP requires a lot of consensus if it is to be standards action, and less if local use. To me at least, I think this is worth exploring, if the arguments can be presented simply.
> 
> We would still require the RFC language discussed earlier: an ECN-enabled AQM SHOULD check for the new identifier (DSCP in this case).

- not a MUST? Because if it doesn't, your sender MUST be able to cope with that I'd say.

Cheers,
Michael

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 6 Aug 2015 22:38:55 +0200 (CEST)

On Tue, 4 Aug 2015, Bob Briscoe wrote:

> *Combining ECT(0) and CE with a globally assigned DSCP solely during initial 
> deployment of L4S seems the least worst choice.

Having the same bits in the header mean different things in combination 
with DSCP seems like a really hard to get deployed Internet-wide.

ECN is just now gaining traction and seems like it might actually see real 
deployment. Repurposing those bits just now would most likely just cause 
confusion.

I started using ECN when it first appeared in the Linux kernel around 2001 
or whenever it was. I had to immediately turn it off because some 
firewalls dropped those packets. Now almost 15 years later after this 
sitting in the operating systems for at least 10 years, we're now getting 
to a point where we're ready to start turning it on widely because things 
do not break when it's turned on.

So whatever you come up with now that requires host stack changes, expect 
5-10 years at least until it can be deployed. This means you have to be 
really sure this is what you actually want before you start to push for 
deployment. Also, deployment impacts should be taken a lot into account 
when deciding what to do.

So how sure are you that L4S as it currently stands is the way to go? If 
you think you're going to invent something new in 2-3 years, then please 
wait until then. Experimentation is all fine and dandy, but until we can 
actually get DSCP codepoints working on Internet-wide scale, this approach 
isn't feasable for that use-case (which for me is close to "the only" 
use-case).

My proposal has been before that we should try to get 7 DSCP codepoints 
deployed by using 000xxx, and nudge providers to incrementally just not 
bleach them and treat them as BE in their core networks, so we can use 
them on the edge to influence AQM there.

So, if we're going to invent new meaning of ECN bits in combination with 
DSCP, then that needs to be coupled with work of getting some DSCP working 
Internet-wide in a fashion that someone actually believes will work out, 
as in actually getting significant Internet-wide deployment.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Andrew Mcgregor <[email-masked]>
Date: Fri, 7 Aug 2015 10:51:16 +1000

I believe getting a DSCP deployed for this is a non-starter; that space is
a complete mess, and if we tie this proposal to cleaning up that mess we'll
get nowhere.  The evil bit doesn't fly either, for a lot of reasons.

That leaves us with ECT(1).

So, how bad is that?  Well, not very.  In places which are ECN-enabled but
not dual-queue, DCTCP or another 1/p wrt ECN marking transport will still
respond to loss; sure, we'll drive the network into a lossy regime, but
those parts of the network are guaranteed to be there anyway due to the
presence of TCP.  Provided the loss response is still sane, that will
equilibrate out and we will end up with a reasonable outcome.  No, it will
not be fair, but in the real world TCP never is since the equilibrium takes
hours to establish while essentially no flows last that long; in real world
practice, the few flows that do last that long need special protection
because they are so fragile (BGP, I'm looking at you), or else nobody
really cares how long they take to complete.  TCP does not share capacity
in any reasonable manner, it is a circuit breaker for avoiding congestion
collapse, and nobody is proposing running completely oblivious traffic on
the internet.  DCTCP still has enough congestion avoidance.

The dual-queue solution is not the only way to deploy either; it's a nice
solution, if one actually wants to achieve capacity sharing by router
feedback, but there are other ways to do the capacity sharing (for example
http://conferences.sigcomm.org/sigcomm/2015/pdf/papers/p1.pdf).  In an
admission controlled environment, separating the queues is not necessary
for sane coexistence.  Further I'm sure there are other single-queue
solutions, involving AQM control systems, although I don't have a proposal
right now (nor do I think one is necessary).

I do think that assuming dual queue to be necessary for deployment is a red
herring.

On 7 August 2015 at 06:38, Mikael Abrahamsson <[email-masked]> wrote:

> or whenever it was. I had to immediately turn it off because some firewalls
> dropped those packets. Now almost 15 years later after this sitting in the
> operating systems for at least 10 years, we're now getting to a point where
> we're ready to start turning it on widely because things do not break when
> it's turned on.
>
> bleach them and treat them as BE in their core networks, so we can use them
> on the edge to influence AQM there.
>
> Internet-wide in a fashion that someone actually believes will work out, as
> in actually getting significant Internet-wide deployment.
>
> --

-- 
Andrew McGregor | SRE | [email-masked] | +61 4 1071 2221

---

From: Jonathan Morton <[email-masked]>
Date: Fri, 7 Aug 2015 10:55:29 +0300

A flow-isolating AQM such as fq_codel would be able to deal with different
flows responding differently to CE.

The problem is that core and semi-core network nodes don't have the
resources to do flow isolation at line rate, or (equivalently) they deal
with too many simultaneous flows to make a meaningful flow isolation scheme
practically implementable.  This, I assume, is where the dual queue
proposal comes in.

Fundamentally, dual queue needs to be able to identify the packets
belonging to each queue.  DSCP is a non-starter because it doesn't, in
practice, survive end to end.  I'm wary of using the ECN field for the same
purpose - it could work, but promotimg CE packets from classic ECN flows
could result in spurious retransmissions due to reordering.  Simply using
the presence of ECN capability is also a bad idea, because Apple's imminent
deployment of classic ECN to millions of end hosts will break that
assumption.

A more robust deployment option would be to use a new transport protocol
ID.  Then it wouldn't be TCP any more, and you can throw away some cruft
while you're at it.  I'll leave the problems with that idea to your fertile
imaginations.

As a gentle reminder, ELR is designed to solve the same problem while
avoiding these deployment problems.  It does "burn" ECT(1), but for a good
cause and - crucially - doesn't require the network to be aware of it
before endpoints can safely turn it on.

- Jonathan Morton

---

From: John Leslie <[email-masked]>
Date: Sat, 8 Aug 2015 08:13:47 -0400

> flows responding differently to CE.

   Flow-isolating AQMs, however, don't have a sufficient way to tell
whether a flow wants DCTCP-like signals.

   The fundamental question Bob Briscoe raises is how to make better
use of ECN-marking. He argues that DCTCP's "1/p" marking gives much
better information.

> resources to do flow isolation at line rate, or (equivalently) they
> deal with too many simultaneous flows to make a meaningful flow
> isolation scheme practically implementable.  This, I assume, is
> where the dual queue proposal comes in.

   I wouldn't assume that.

   I know that Bob is thinking about dualq for the first hop (and that
IMHO is why he likes DSCP as a distinguishing mark which will survive
the first few hops).

> belonging to each queue.

   Yes.

> DSCP is a non-starter because it doesn't, in practice, survive end
> to end.

   That worries me, too.

> I'm wary of using the ECN field for the same purpose - it could work,
> but promotimg CE packets from classic ECN flows could result in
> spurious retransmissions due to reordering.

   That is a serious issue, though hard to see for the casual reader.
Bob proposes that ECT(1) mark the choice to use "1/p" marking; but he
also proposes that any packet already marked CE go through the same
"1/p" queue. I don't agree that's good.

> Simply using the presence of ECN capability is also a bad idea,
> because Apple's imminent deployment of classic ECN to millions of
> end hosts will break that assumption.

   Yes. (Not that the issue doesn't _already_ exist...)

> while you're at it.  I'll leave the problems with that idea to your fertile

   ;^)

> avoiding these deployment problems.  It does "burn" ECT(1), but for a
> good cause and - crucially - doesn't require the network to be aware
> of it before endpoints can safely turn it on.

   Neither scheme, IMHO, can work unless we deprecate ECN Nonce. That
should be a no-brainer (IMHO) since ECN Nonce was experimental, with
a very clear statement of intent to submit as Proposed Standard after
experience showed it to be useful; and that hasn't happened in twelve
years.

   Once we deprecate ECN Nonce, we can address the question of what use
of ECT(1) is best. My current belief is that the "1/p" marking would be
the best use; but there are many issues to consider.