Subject: Re: [tsvwg] regarding L4S

From: Bob Briscoe <[email-masked]>
Date: Thu, 20 Dec 2018 23:34:46 +0000

Mikael,

Just found this email. Sry I failed to answer it at the time. Pls see 
inline...

On 21/11/2018 08:02, Mikael Abrahamsson wrote:
>
> Hi,
>
> I find the concept of L4S interesting. However, I see this as spending 
> the last bit combination in the IP header we still can realistically 
> do anything statelessly with when it comes to queue management, so I 
> want to make sure we get it right.
>
> I would like to bring this to a wider audience in the operator 
> community, but that would need some new material from the authors 
> where they create an "introduction to L4S for network people" which 
> focuses less on the congestion avoidance, but more on "what does the 
> router need to do" and aspects such as "is this implementable on 
> existing hardware with software update" (and this is specifically for 
> NPUs etc today that might only implement WRED). This document should 
> probably have a brief introduction to ECN as well. For me as a network 
> guy dabbling in transport and queue management, the whole L4S 
> combination of documents is quite hard to understand. I don't doubt 
> it's correct, but it uses a lot of terms that wasn't immediately 
> familiar to me (like the fact that ECT(1) was 01 in the TOS bit). If 
> the authors could go to NANOG, RIPE or similar place and present 
> (preferably where there is video of the presentation), that would be 
> brilliant. I volunteer to review any material beforehand and advice on 
> things that might help to be included.

[BB] Don't worry, I wouldn't expect folks who live and breathe network 
equipment to necessarily have subtle transport layer stuff at the 
forefront of their minds. I've taken on board the earlier comment that 
the L4S architecture draft doesn't actually explain how L4S works and I 
will fix that. For the other L4S spec's I think we have to keep them as 
spec's rather than tutorials.

Thanks for the offer to review a presentation. Here's a stock L4S 
presentation I've refined for network operators over a number of 
occasions (I sometimes also add operator-specific variations). Altho you 
don't get the voice track, I try to write it on the slides, in order to 
help non-native-English speakers. Pls tell me if that's not actually the 
case:
http://bobbriscoe.net/presents/1706L4S-for-operators/1707L4S-BobBriscoe.pdf
It's available via the presentations section of the L4S landing page: 
https://riteproject.eu/dctth/#presentations
There's an explanation of the ECN codepoints on slide 16.

I don't have a video of it being presented. But we do have a couple of 
videos of presentations at the IETF on the L4S landing page (4-minute 
edited highlights <https://riteproject.eu/dctth/#1507bnb> and the 30-min 
presentation to the IETF AQM WG <https://riteproject.eu/dctth/#1507aqmwg>).

NANOG or RIPE wouldn't be my priority for L4S, cos the primary problem 
is in access networks, which tends to be a very different community 
within operators. I believe the target is more organizations like 
broadband forum, GSMA and NCTA.

> I would like to see more test results for instance testing done with 
> "flent" (https://flent.org/) so we can compare L4S with/without 
> FQ_CODEL/FIFO as the second queue, different TCP algorithms, UDP real 
> time flows etc. In the early days of BBF we saw starvasion issues that 
> weren't obvious before this kind of testing was done. This is just one 
> example. 
[BB] I'll send you a copy of a paper we are preparing for a journal in 
the Spring with comparisons against fq-CoDel and PIE. Anyone else on the 
list who wants a copy, pls drop me an email.

Also, here's the early paper with all the comparisons against fq-CoDel 
and PIE (and RED for some cases):
http://www.bobbriscoe.net/projects/latency/dctth_preprint.pdf
It's also available via the L4S landing page 
https://riteproject.eu/dctth/#papers (the 'Data Centre to the Home' paper)

There's a huge amount of evaluation data compressed into this paper, so 
it takes quite some work to really appreciate it. The copy I will send 
you summarizes similar experiments using just summary statistical 
metrics, whereas this earlier paper gives a better visualization of all 
the underlying data (compressing hundreds of CDFs, time-series and 
scatter plots into a few pages).

You will see we show all the main metrics from each experiment: qDelay, 
throughput fairness, completion time, utilization, drop. Many proponents 
of qDelay solutions tend not to watch the effect on all the other 
parameters.

> We need deployment suggestions, and I'd really like to see the authors 
> implement L4S on for instance OpenWrt (the sqm framework provides way 
> to combine qdisc:s that is easily deployable and testable). This in 
> combination with flent would mean it's easy to do testing for people 
> on their own.
[BB] It's available as a qdisc for Linux via the source code section of 
the landing page:
https://riteproject.eu/dctth/#code
In a few days, there's going to be a major update, so you might want to 
keep watching.

Bear in mind you need DCTCP for the end systems as well.

>
> I'd like to see comprehensible testing at lots of different access 
> speeds, all the way from 1 megabit/s all the way up to 1000 megabit/s, 
> with long and short RTT paths combined. Just to figure out where L4S 
> is usable and not. Remember, we still have lots of people connected to 
> ADSL lines with < 1 megabit/s upstream speed.
[BB] I believe there is little point evaluating a sub-ms queuing delay 
solution below a bottleneck rate of about 4Mb/s, because serialization 
delay dominates queuing delay below that (a 1500B packet is 3ms @4Mb/s).

So the evaluations in the paper I will send you show a matrix of tests 
across 4,12,40,120,200Mb/s and 5,10,20,50,100ms RTT.
Also, a matrix of pairs of all these RTTs. And mixes of long and short 
flows at all these link rates and RTTs. Also, it gives overload tests 
with unresponsive flows.

>
> There is text regarding putting CE marked packets into the L4S queue 
> and causing reordering within the flow. I have doubts this text saying 
> this reordering isn't really a problem. I'd like to see testing of 
> this done in more real life scenarios (can be done with OpenWrt + 
> flent for instance), with different TCP implementations and more. 
[BB] I admit we haven't measured this, mainly 'cos I'm fairly certain of 
my intuition that it's a corner of a corner-case so this hasn't been a 
priority over other more pressing work.

We (or anyone else - hi! all you researchers out there looking for 
research problems) would need to artificially create a scenario in which 
multiple bottlenecks will occur on one path then prove that queuing 
Classic CE and L4S CE into the L queue won't trigger spurious reordering 
detection for Classic ECN traffic.

Then to test our assertion that multiple bottlenecks don't often happen 
on a path, you really need passive measurements not a testbed. The only 
paper I know of that has tried to measure this is:
Deng, L. & Kuzmanovic, A., "Pong: Diagnosing Spatio-temporal Internet 
Congestion Properties," SIGMETRICS Perform. Eval. Rev. 35(1):381--382 
ACM (June 2007)
However, it only describes the technique, not any results.

> Do we have UDP implementations that use ECN that might not like this? 
> I don't know. I'd like to get more eyes on it.
>
[BB] It depends on the loss detection algorithm, so the transport 
protocol itself is unlikely to affect the outcome. My assertion that 
problems will be rare assumed 3DupACK. If RACK were used, problems would 
be even rarer.

HTH

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 21 Dec 2018 07:22:51 +0100 (CET)

On Thu, 20 Dec 2018, Bob Briscoe wrote:

> presentation I've refined for network operators over a number of occasions (I 
> sometimes also add operator-specific variations). Altho you don't get the 
> voice track, I try to write it on the slides, in order to help 
> non-native-English speakers. Pls tell me if that's not actually the case:

Thanks, I looked through 1707L4S-BobBriscoe.pdf which is definitely 
helpful, but it makes no mention of the relaxed requirement for packet 
ordering in the network for L4S traffic? I think this is one of the major 
benefits to L4S that I believe will help. Btw, do you have any numbers on 
this? I know mobile networks typically have 10% BER and that's seen as a 
good level to aim for as it brings a good tradeoff between bit/s 
throughput and not get too many L1/L2 retransmits (ARQ). Now that we have 
protocols such as QUIC that have internal muxing with different flows in 
same 5 tuple, I'd imagine this might help significantly?

> broadband forum, GSMA and NCTA.

Right, makes sense. I'd still however like to bring this to the attention 
of the NANOG crowd so they know what's going on.

> Bear in mind you need DCTCP for the end systems as well.

Yes, but installing OpenWrt on a device makes it really easy to emulate a 
low-speed link with your dual-Q AQM to do own testing.

> across 4,12,40,120,200Mb/s and 5,10,20,50,100ms RTT. Also, a matrix of 
> pairs of all these RTTs. And mixes of long and short flows at all these 
> link rates and RTTs. Also, it gives overload tests with unresponsive 
> flows.

Ok, I'll read the document.

> problems) would need to artificially create a scenario in which multiple 
> bottlenecks will occur on one path then prove that queuing Classic CE and L4S 
> CE into the L queue won't trigger spurious reordering detection for Classic 
> ECN traffic.

What I'm afraid of is to have one end CE mark 1% of packets which then 
goes into one of these dual-q with a fairly deep individual FQ buffer fill 
and then the CE packets sneaks by 10-20ms before the other packets. With 
SACK this is probably not really a problem, but what about flows that do 
not use SACK? What about non-TCP protocols with weak packet reordering 
handling? Hm, actually, do they even get by default ECN enabled on systems 
with ECN enabled?

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 21 Dec 2018 13:45:52 +0000

Mikael,

On 21/12/2018 06:22, Mikael Abrahamsson wrote:
>> occasions (I sometimes also add operator-specific variations). Altho 
>> you don't get the voice track, I try to write it on the slides, in 
>> order to help non-native-English speakers. Pls tell me if that's not 
>> actually the case:
> ordering in the network for L4S traffic? I think this is one of the 
> major benefits to L4S that I believe will help. Btw, do you have any 
> numbers on this? I know mobile networks typically have 10% BER and 
> that's seen as a good level to aim for as it brings a good tradeoff 
> between bit/s throughput and not get too many L1/L2 retransmits (ARQ). 
> Now that we have protocols such as QUIC that have internal muxing with 
> different flows in same 5 tuple, I'd imagine this might help 
> significantly?

[BB] You're correct that the slides haven't been updated since we 
realized we should add the RACK requirement. I've added it to my ToDo list.

I had an MSc student simulating this to get some numbers, but he tried 
to extend his time while also running a full-time job and ended up 
having to curtail the experiments. I'll see what I can recover from it 
in terms of headline numbers.

>
>> NANOG or RIPE wouldn't be my priority for L4S, cos the primary 
>> problem is in access networks, which tends to be a very different 
>> community within operators. I believe the target is more 
>> organizations like broadband forum, GSMA and NCTA.
>
> Right, makes sense. I'd still however like to bring this to the 
> attention of the NANOG crowd so they know what's going on.
[BB] Are you saying you will, or do you mean we the L4S developers ought 
to? Happy either way.

> Yes, but installing OpenWrt on a device makes it really easy to 
> emulate a low-speed link with your dual-Q AQM to do own testing.
>
>> So the evaluations in the paper I will send you show a matrix of 
>> tests across 4,12,40,120,200Mb/s and 5,10,20,50,100ms RTT. Also, a 
>> matrix of pairs of all these RTTs. And mixes of long and short flows 
>> at all these link rates and RTTs. Also, it gives overload tests with 
>> unresponsive flows.
>
>> problems) would need to artificially create a scenario in which 
>> Classic CE and L4S CE into the L queue won't trigger spurious 
>> reordering detection for Classic ECN traffic.
>
> goes into one of these dual-q with a fairly deep individual FQ buffer 
> fill and then the CE packets sneaks by 10-20ms before the other 
> packets. With SACK this is probably not really a problem, but what 
> about flows that do not use SACK? What about non-TCP protocols with 
> weak packet reordering handling? Hm, actually, do they even get by 
> default ECN enabled on systems with ECN enabled?
>
[BB] Yes, I understand the problem, and you've described it (nearly) 
correctly, except I think it's no worse with or without SACK.

  * First it is quite unusual for a flow to be bottlenecked at two
    points on the same path (sorry I can't quantify that statement).
  * If there is a multiple bottleneck, and the first queue supports
    classic ECN marking (which is currently also quite unusual but might
    not be in future), then a CE packet might get 10-20ms ahead of the
    rest of a flow, as you say.
  * Then as later non-CE packets arrive the sequence might be 100, 101,
    130, 102, 103, 104, ... (packet numbering rather than byte numbering
    for simplicity)
  * Even without SACK, packet#130 will increment the dup ACK counter
    once, but subsequent packets (#102, #103, etc) will cancel that as
    they continue to push forward the cumulative ACK boundary

So there's no problem unless:

  * within a Classic flow the first router marks at least 3 CEs in a row.
  * If the Classic ECN source was not using RACK, that would trigger 3
    DupACK detection and cause a spurious retransmission
  * If the Classic ECN source was using RACK, a whole reordering window
    would need to be CE-marked in a row, which is typically many more
    than 3.

So in summary, you would hit a problem if and only if there were:

 1. multiple bottlenecks on one path (unusual and transient if it
    happens at all)
 2. the first queue supports classic ECN marking and the second supports
    the L4S DualQ (presumably this will always be quite unusual)
 3. the classic ECN source does not support RACK (already getting quite
    unusual even tho RACK hasn't been out long)
 4. at least 3 packets in sequence within a flow get CE-marked by the
    first classic ECN router (unusual)

So 3 or 4 occurrences that are all quite unusual have to coincide, and 
even if they do the likely outcome will only be an odd spurious 
retransmission.

Or am I missing something?

Bob

---

From: Mikael Abrahamsson <[email-masked]>
Date: Fri, 21 Dec 2018 15:02:00 +0100 (CET)

On Fri, 21 Dec 2018, Bob Briscoe wrote:

> [BB] Are you saying you will, or do you mean we the L4S developers ought to? 
> Happy either way.

I'll send an email to the nanog list and notify you when I have done that. 
I'd like to see something on the re-ordering tolerance I can refer to, as 
I'd really like to have this discussion in a wider context. The "known 
truth" in the networking business is that re-ordering is a no-no. If we're 
going to relax that for some traffic I'd like to have more eyes on it.

> So 3 or 4 occurrences that are all quite unusual have to coincide, and even 
> if they do the likely outcome will only be an odd spurious retransmission.
>

You know this more than I do. If some more people put eyes on this and 
agree that it's not a problem then I'm fine with it. I'd just like to have 
something to refer to, because this discussion is going to come up.

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 21 Dec 2018 16:26:19 +0000

Mikael,

On 21/12/2018 14:02, Mikael Abrahamsson wrote:
>> [BB] Are you saying you will, or do you mean we the L4S developers 
>> ought to? Happy either way.
>
> I'll send an email to the nanog list and notify you when I have done 
> that. I'd like to see something on the re-ordering tolerance I can 
> refer to, as I'd really like to have this discussion in a wider 
> context. The "known truth" in the networking business is that 
> re-ordering is a no-no. If we're going to relax that for some traffic 
> I'd like to have more eyes on it.
>
[snip]
>> So 3 or 4 occurrences that are all quite unusual have to coincide, 
>> and even if they do the likely outcome will only be an odd spurious 
>> retransmission.
>>
> agree that it's not a problem then I'm fine with it. I'd just like to 
> have something to refer to, because this discussion is going to come up.
>
This was deeply discussed on the list with a number of TCP experts (in 
tsvwg and tcpm) when the decision to use ECT(1) was first taken.

There is text justifying this decision in the appendix that gives the 
rationale for using the ECT(1) and CE codepoints: 
https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-B.1 
under the bullet headed: " Risk of reordering classic CE packets:"

I have altered the first para of 5.1 in the body to point to this 
appendix as follows:

5.1. Prerequisite Classification and Re-Marking Behaviour 
<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#section-5.1>
CURRENT:

    A network node that implements the L4S service MUST classify arriving
    ECT(1) packets for L4S treatment and it SHOULD classify arriving CE
    packets for L4S treatment as well.Section 5.3 
<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#section-5.3> 
describes a possible exception to this latter rule for some CE packets.

PROPOSED:

    A network node that implements the L4S service MUST classify arriving
    ECT(1) packets for L4S treatment and it SHOULD classify arriving CE
    packets for L4S treatment as well.CE packets might have originated as ECT(1) or ECT(0), but the above rule 
to classify them as if they originated as ECT(1) is the safe choice (see 
Appendix B.1 for rationale). Section 5.3 describes an optional mechanism 
for distinguishing CE packets that originated as ECT(0), but it is 
highly unlikely to be needed.

The appendix itself was written before we thought of the RACK 
requirement. So I've changed this bullet as follows:

CURRENT:

    Risk of reordering classic CE packets:  Having to classify all CE
       packets as L4S risks some classic CE packets arriving early, which
       is a form of reordering.  Reordering can cause the TCP sender to
       retransmit spuriously.  However,one or two packets delivered early does not cause any spurious 
retransmissions because the subsequent packets continue to move the 
cumulative acknowledgement boundary forwards. Anyway, the risk of 
reordering would be low, because: i) it is quite unusual to experience 
more than one bottleneck queue on a path; ii) even then, reordering 
would only occur if there was simultaneous mixing of classic and L4S 
traffic, which would be more unlikely in an access link, which is where 
most bottlenecks are located; iii) even then, spurious retransmissions 
would only occur if a contiguous sequence of three or more classic CE 
packets from one bottleneck arrived at the next, which should in itself 
happen very rarely with a good AQM. The risk would be completely 
eliminated in AQMs that were transport-aware (but they should not need 
to be);

PROPOSED:

    Risk of reordering classic CE packets:  Having to classify all CE
       packets as L4S risks some classic CE packets being wrongly
       classified as L4S and arriving early, which is a form of
       reordering.  Reordering can cause the TCP sender to retransmit
       spuriously.  However,the risk of spurious retransmissions would be extremely low, because: 1. 
it is quite unusual to experience more than one bottleneck queue on a 
path. 2. It would be even more unusual for the first bottleneck to 
support classic ECN marking and for the second to support L4S ECN 
marking 3. even then, reordering would only occur if there was 
simultaneous mixing of classic and L4S traffic, which would be more 
unlikely in an access link, which is where most bottlenecks are located; 
4. even then, spurious retransmissions would only occur if a contiguous 
sequence of three or more packets in one classic ECN flow were all 
CE-marked at the first bottleneck; 5. even then, a spurious 
retransmission would only occur if the source did not support RACK 
[I-D.ietf-tcpm-rack], which is already widely supported. Otherwise a 
whole reordering window within one classic ECN flow would have to be 
marked CE at the first bottleneck to cause a spurious retransmission. It 
is extremely unlikely that a set of 5 eventualities that are each 
unusual in themselves would all happen simultaneously. But, even if they 
did, it would only cause spurious retransmission of a packet.

OK?

Bob