Subject: [tcpm] Cross-area alignment on "L4S and RACK"

From: "Scharf, Michael" <[email-masked]>
Date: Tue, 6 Nov 2018 19:20:11 +0000

A comment on the TCPM presentation "L4S and RACK":

As far as I understand, the DETNET WG in RTG area has quite some uses cases for ultra-low latency transport - in particular also with bounded jitter. And some of these applications (e.g. using UDP) apparently may not be able to tolerate *any* out-of-order packet delivery.

So perhaps some cross-area alignment on ulta-low-latency application requirements would be useful?

Michael
(who recently had to review draft-ietf-detnet-architecture)

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Wed, 7 Nov 2018 05:14:12 +0700

Michael,

On Wed, Nov 7, 2018 at 2:20 AM Scharf, Michael <
[email-masked]> wrote:

> As far as I understand, the DETNET WG in RTG area has quite some uses
> cases for ultra-low latency transport - in particular also with bounded
> jitter. And some of these applications (e.g. using UDP) apparently may not
> be able to tolerate *any* out-of-order packet delivery.
>

I'm not a responsible AD for DETNET, but we approved
https://datatracker.ietf.org/doc/draft-ietf-detnet-use-cases/ within the
last couple of weeks.

My understanding is that at least some of the use cases have that
intolerance for out-of-order delivery (among other characteristics). That
was certainly the case when we chartered DETNET.

I'm sure TSV will do the right thing ;-)

Spencer

---

From: "Black, David" <[email-masked]>
Date: Wed, 7 Nov 2018 02:01:06 +0000

> As far as I understand, the DETNET WG in RTG area has quite some uses cases for ultra-low latency transport - in particular also with bounded jitter.
> And some of these applications (e.g. using UDP) apparently may not be able to tolerate *any* out-of-order packet delivery.

I’m the Transport Technical Advisor to the DETNET WG – the envisioned DETNET data path structure is similar to Intserv or policy-based routing- a full 5-tuple match (2 x addr, protocol 2 x port) on all inbound traffic picks off the flows that requires deterministic forwarding and diverts them into a DETNET data path that maintains per-flow state before the Diffserv logic looks at the DSCP or L4S logic  looks at ECT(1) to figure out what to do with the (non-deterministic) rest of the traffic which is handled as flow aggregates (no per-flow state in forwarding path).

This could be made clearer in their drafts …

Thanks, --David

From: tsvwg [mailto:[email-masked]] On Behalf Of Spencer Dawkins at IETF
Sent: Tuesday, November 6, 2018 5:14 PM
To: [email-masked]
Cc: tcpm IETF list; [email-masked]
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

[EXTERNAL EMAIL]
Please report any suspicious attachments, links, or requests for sensitive information.
Michael,

On Wed, Nov 7, 2018 at 2:20 AM Scharf, Michael <[email-masked]<mailto:[email-masked]>> wrote:
A comment on the TCPM presentation "L4S and RACK":

Michael

I'm not a responsible AD for DETNET, but we approved https://datatracker.ietf.org/doc/draft-ietf-detnet-use-cases/ within the last couple of weeks.

My understanding is that at least some of the use cases have that intolerance for out-of-order delivery (among other characteristics). That was certainly the case when we chartered DETNET.

Spencer

---

From: Bob Briscoe <[email-masked]>
Date: Thu, 8 Nov 2018 11:55:31 +0000

Michael,

This is merely a symptom of a difference in opinion on where the 
resequencing function should be located.

  * L4S takes the end-to-end approach saying that it is sufficient to do
    resequencing in one place (the receiving host) if it is needed. Then
    any resequencing delay only affects that flow.
  * The DETNET approach is saying the resequencing must be done
    hop-by-hop. This means there appears to be no resequencing needed on
    the receiver (except if there's re-ordering on the final hop).

  * In order to guarantee no resequencing in the network (DETNET
    approach), only the the worst case latency can be guaranteed, and
    every packet has to have that same worst-case latency.
  * When you leave resequencing to the receiver (end-to-end approach),
    most of the packets arrive earlier than they would with DETNET, but
    out of order ones might not. Then the application chooses whether it
    wants to wait.

The 3 DupACK rule in TCP led the Internet not to follow the end-to-end 
principle on re-sequencing. Now we're removing the 3 DupACK rule, we can 
take advantage of the e2e principle.

Of course, an application might choose to use TCP and L4S, rather than 
UDP and L4S (e.g QUIC). Then there could still be HoL blocking in the 
receiving TCP stack. But at least there's no HoL blocking in the 
network, and the app can choose between stream (TCP) or datagram (UDP).

Bob

-- 
________________________________________________________________
Bob Briscoe                               http://bobbriscoe.net/

---

From: "Scharf, Michael" <[email-masked]>
Date: Thu, 8 Nov 2018 16:35:01 +0000

Inline [ms]

From: Bob Briscoe [mailto:[email-masked]]
Sent: Thursday, November 8, 2018 12:56 PM
To: Scharf, Michael; [email-masked]
Cc: [email-masked]
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

This is merely a symptom of a difference in opinion on where the resequencing function should be located.

  *   L4S takes the end-to-end approach saying that it is sufficient to do resequencing in one place (the receiving host) if it is needed. Then any resequencing delay only affects that flow.
  *   The DETNET approach is saying the resequencing must be done hop-by-hop. This means there appears to be no resequencing needed on the receiver (except if there's re-ordering on the final hop).

  *   In order to guarantee no resequencing in the network (DETNET approach), only the the worst case latency can be guaranteed, and every packet has to have that same worst-case latency.
  *   When you leave resequencing to the receiver (end-to-end approach), most of the packets arrive earlier than they would with DETNET, but out of order ones might not. Then the application chooses whether it wants to wait.
[ms] My point is different: There seem to be other working groups in the IETF that do talk about “low latency” as well but apparently they also do need in-order delivery of packets, too. So the assumption that e.g. a link layer technology can simply disable in-order delivery for all “low latency” applications seems not correct. The reality may be a bit more complex.
The 3 DupACK rule in TCP led the Internet not to follow the end-to-end principle on re-sequencing. Now we're removing the 3 DupACK rule, we can take advantage of the e2e principle.
[ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC 5681 is standards track. So the bar for “removing” that rule for TCP is not low…
[ms] Also, the end-to-end principle comes with tradeoffs. For instance, relying on the end-to-end principle for recovery of bit errors is not efficient. For in-order delivery there are tradeoffs as well, see e.g. Section 4 in draft-ietf-tcpm-rack-04. There may be no free lunch.
Of course, an application might choose to use TCP and L4S, rather than UDP and L4S (e.g QUIC). Then there could still be HoL blocking in the receiving TCP stack. But at least there's no HoL blocking in the network, and the app can choose between stream (TCP) or datagram (UDP).
[ms] If LRO/GRO in the receiving TCP stack gets messed up, I fail to see the benefit (see draft-ietf-tcpm-rack-04).
Michael

Bob

Michael

--

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 9 Nov 2018 05:42:06 +0000

Michael,

That is the point I thought you were making, and my point /is/ about that.

If an application needs all the packets (i.e. a reliable protocol), it's 
not useful to reduce the latency of some packets but not all. That's 
where the in-order requirement comes from.

However, if a stream-based transport is used (i.e. TCP) the 
/application/ still gets in-order delivery from TCP. It just doesn't 
have to require links to provide in order delivery as well.

Note that there can be multiple flows over one link. So 'in-order 
delivery' for a link does not mean in order of TCP sequence number. It 
means 'delivered in the order that packets arrived at the link' (the 
link ingress adds its own sequence numbers, and the link egress buffers 
them until it can send out in the same order, or until a time-out or max 
no. of link retransmissions).

So, if the link loses a packet from flow A (the link isn't aware of 
microflows, I'm just saying the packet happens to be from flow A), 
link-layer in-order delivery will hold back all packets sent after that 
packet (from all flows, not just flow A) until the link repairs the gap 
or gives up.

So hop-by-hop in-order delivery gives every flow the same delay as the 
worst delayed flow.

Of course, someone building a network for DETNET would avoid link 
technologies like WiFi or LTE that frequently lose and retransmit 
packets. But wherever there is variable delay in a link, guaranteed 
in-order delivery per-hop means every flow is guaranteed to always get 
the worst delay from every link, which would have been experienced by 
only one flow without per-hop in-order delivery.

And many of the applications of DETNET involve dumb industrial machinery 
that just expects everything to arrive in order and to a clocked 
schedule. But it's always as effective to deploy a resequencing buffer 
as the penultimate hop, screwed onto the receiving machine if necessary, 
rather than require per-hop resequencing in the network.

________________
So, why do Internet links often ensure in-order delivery even though TCP 
puts everything in order for the application?

Short answer: the 3 DupACK rule (which is to do with loss-recovery, a 
different issue from the discussion above).

Long answer: As flow rates have scaled up, the typical time between 3 
packet arrivals has become so small that TCP's 3-DupACK rule makes links 
have to provide delivery with only a very small re-ordering degree - so 
small that it's easier for a link to just deliver everything in the 
order it received it; not because TCP doesn't put packets back in order, 
but just so as not to trigger TCP into generating spurious retransmissions.

Bob

On 08/11/2018 16:35, Scharf, Michael wrote:
>
> Inline [ms]
>
> *From:*Bob Briscoe [mailto:[email-masked]]
> *Sent:* Thursday, November 8, 2018 12:56 PM
> *To:* Scharf, Michael; [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [tsvwg] Cross-area alignment on "L4S and RACK"
>
> Michael,
>
>   * L4S takes the end-to-end approach saying that it is sufficient to
>     do resequencing in one place (the receiving host) if it is needed.
>     Then any resequencing delay only affects that flow.
>     hop-by-hop. This means there appears to be no resequencing needed
>     on the receiver (except if there's re-ordering on the final hop).
>
>     most of the packets arrive earlier than they would with DETNET,
>     but out of order ones might not. Then the application chooses
>     whether it wants to wait.
>
> [ms] My point is different: There seem to be other working groups in 
> the IETF that do talk about “low latency” as well but apparently they 
> also do need in-order delivery of packets, too. So the assumption that 
> e.g. a link layer technology can simply disable in-order delivery for 
> all “low latency” applications seems not correct. The reality may be a 
> bit more complex.
>
> principle on re-sequencing. Now we're removing the 3 DupACK rule, we 
> can take advantage of the e2e principle.
>
> [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC 5681 is 
> standards track. So the bar for “removing” that rule for TCP is not low…
>
> [ms] Also, the end-to-end principle comes with tradeoffs. For 
> instance, relying on the end-to-end principle for recovery of bit 
> errors is not efficient. For in-order delivery there are tradeoffs as 
> well, see e.g. Section 4 in draft-ietf-tcpm-rack-04. There may be no 
> free lunch.
>
> [ms] If LRO/GRO in the receiving TCP stack gets messed up, I fail to 
> see the benefit (see draft-ietf-tcpm-rack-04).
>
> Michael
>
>
> Bob
>
> Bob Briscoehttp://bobbriscoe.net/

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 9 Nov 2018 06:21:23 +0000

I don’t see the fundamental difference between a link technology that guarantees in-order delivery within a flow (say by flow hashing, see ECMP) vs. some link scheme that guarantees in-order delivery, say, only for some fraction of the traffic characterized by some other means (say, other header bits).

Clearly, reordering is a problem that needs to be discussed in the IETF as a whole. And we could discuss whether a revision of RFC 3366 is needed.

But I am not convinced that this topic has to be tied into an experiment such as L4S.

Michael

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, November 9, 2018 6:42 AM
To: Scharf, Michael; [email-masked]
Cc: [email-masked]
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

If an application needs all the packets (i.e. a reliable protocol), it's not useful to reduce the latency of some packets but not all. That's where the in-order requirement comes from.

However, if a stream-based transport is used (i.e. TCP) the /application/ still gets in-order delivery from TCP. It just doesn't have to require links to provide in order delivery as well.

Note that there can be multiple flows over one link. So 'in-order delivery' for a link does not mean in order of TCP sequence number. It means 'delivered in the order that packets arrived at the link' (the link ingress adds its own sequence numbers, and the link egress buffers them until it can send out in the same order, or until a time-out or max no. of link retransmissions).

So, if the link loses a packet from flow A (the link isn't aware of microflows, I'm just saying the packet happens to be from flow A), link-layer in-order delivery will hold back all packets sent after that packet (from all flows, not just flow A) until the link repairs the gap or gives up.

So hop-by-hop in-order delivery gives every flow the same delay as the worst delayed flow.

Of course, someone building a network for DETNET would avoid link technologies like WiFi or LTE that frequently lose and retransmit packets. But wherever there is variable delay in a link, guaranteed in-order delivery per-hop means every flow is guaranteed to always get the worst delay from every link, which would have been experienced by only one flow without per-hop in-order delivery.

And many of the applications of DETNET involve dumb industrial machinery that just expects everything to arrive in order and to a clocked schedule. But it's always as effective to deploy a resequencing buffer as the penultimate hop, screwed onto the receiving machine if necessary, rather than require per-hop resequencing in the network.

________________
So, why do Internet links often ensure in-order delivery even though TCP puts everything in order for the application?

Short answer: the 3 DupACK rule (which is to do with loss-recovery, a different issue from the discussion above).

Long answer: As flow rates have scaled up, the typical time between 3 packet arrivals has become so small that TCP's 3-DupACK rule makes links have to provide delivery with only a very small re-ordering degree - so small that it's easier for a link to just deliver everything in the order it received it; not because TCP doesn't put packets back in order, but just so as not to trigger TCP into generating spurious retransmissions.

Bob

From: Bob Briscoe [mailto:[email-masked]]
Sent: Thursday, November 8, 2018 12:56 PM
To: Scharf, Michael; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

Bob

Michael

--

--

---

From: Fred Baker <[email-masked]>
Date: Fri, 9 Nov 2018 13:57:09 +0700

> On Nov 9, 2018, at 1:21 PM, Scharf, Michael <[email-masked]> wrote:
> 

A comment on ECMP via hashing; it doesn't guarantee in-order delivery. It does support-if-not-quite-guarantee in-order delivery as long as nothing changes. However, if relevant routing changes, sequence can be affected until routing stabilizes.

A lawyer told me once that "guarantee" is a word freighted with money...

--------------------------------------------------------------------------------
Victorious warriors win first and then go to war,
Defeated warriors go to war first and then seek to win.
     Sun Tzu

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 9 Nov 2018 10:08:29 +0000

Michael,

On 09/11/2018 06:21, Scharf, Michael wrote:
>
> I don’t see the fundamental difference between a link technology that 
> guarantees in-order delivery within a flow (say by flow hashing, see 
> ECMP) vs. some link scheme that guarantees in-order delivery, say, 
> only for some fraction of the traffic characterized by some other 
> means (say, other header bits).
>
[The "other means" and "other bits" wording is pretty ambiguous here. I 
reckon I can guess what you mean but then I don't know why you're saying 
this, 'cos I don't think I've disagreed with you on this. So perhaps you 
could be more specific and we might be able to see why you've brought 
this up.]

Also I wasn't really talking about all "link technology that guarantees 
in-order delivery within a flow". {Note 1} I was talking about link 
technology where all flows are blocked while waiting to get the 
aggregate back in order that it was in when it arrived. I specifically said:
> link ingress adds its own sequence numbers, and the link egress 
> buffers them until it can send out in the same order, or until a 
> time-out or max no. of link retransmissions).
For example:

  * A packet resequencing buffer after allowing for link-layer
    retransmissions is common for radio link technologies such as LTE
    (PDCP layer) and 802.11 WiFi.
  * A packet resequencing buffer after merging multiple bonded link
    channels is common for other access link technologies such as
    downstream DOCSIS bonded channels.

> Clearly, reordering is a problem that needs to be discussed in the 
> IETF as a whole. And we could discuss whether a revision of RFC 3366 
> is needed.
>
No. There is no proposal to alter general guidance for all links like 
RFC 3366. No-one is saying this.

There is not even a proposal to require or even recommend that 
L4S-specific links do anything.

The specific proposal is to require L4S /sources/ to use a RACK-like 
scheme, as a condition for setting the ECT(1) codepoint. See the last 
bullet in S.4.3 of draft-ietf-tsvwg-ecn-l4s-id-05 
<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#section-4.3>.

This says nothing about what L4S links MUST, SHOULD or MAY do. It just 
gives an L4S link an opportunity not to have to do resequencing.

> But I am not convinced that this topic has to be tied into an 
It is an opportunity that currently only applies to L4S.

L4S is merely an example of a case where we can mandate that there will 
be no non-RACK traffic in a link, which creates an opportunity for those 
who are implementing the L4S DualQ Coupled AQM to ignore the RFC3366 
resequencing advice for the L4S queue (but we are not changing the 
advice, because L4S is only an experiment). The prize is to remove the 
head-of-line blocking problem for their users.

This is not L4S-specific, altho there are no other examples at the 
moment where this opportunity is possible (so in that sense it is 
/currently/ L4S-specific).

The message has to be handled really carefully, because the 
opportunities for misunderstanding are enormous (as evidenced by this 
thread, and we haven't even started talking to non-transport people yet).

So Gorry and David B want me to draft a short I-D that they will 
word-smith and author as tsvwg chairs to make the situation clear.

I have said I don't want this discussion to be officially raised outside 
the transport area until we are sure that a variant of RACK is even 
feasible without any packet counting at all (i.e. without the initial 
3-DupACK rule in current RACK). So I'll prioritize working that out for 
the next week or so, rather than drafting anything.

Bob

{Note 1}: I hadn't considered that this would affect technologies like 
ECMP that maintain order without a resequencing buffer and without 
introducing HoL blocking. But actually, the consequence there would be 
that L4S traffic could be sprayed by ECMP without per-flow hashing.

> *Sent:* Friday, November 9, 2018 6:42 AM
> If an application needs all the packets (i.e. a reliable protocol), 
> it's not useful to reduce the latency of some packets but not all. 
> That's where the in-order requirement comes from.
>
> link ingress adds its own sequence numbers, and the link egress 
> buffers them until it can send out in the same order, or until a 
> time-out or max no. of link retransmissions).
>
> link-layer in-order delivery will hold back all packets sent after 
> that packet (from all flows, not just flow A) until the link repairs 
> the gap or gives up.
>
> And many of the applications of DETNET involve dumb industrial 
> machinery that just expects everything to arrive in order and to a 
> clocked schedule. But it's always as effective to deploy a 
> resequencing buffer as the penultimate hop, screwed onto the receiving 
> machine if necessary, rather than require per-hop resequencing in the 
> network.
>
> ________________
> So, why do Internet links often ensure in-order delivery even though 
> TCP puts everything in order for the application?
>
> packet arrivals has become so small that TCP's 3-DupACK rule makes 
> links have to provide delivery with only a very small re-ordering 
> degree - so small that it's easier for a link to just deliver 
> everything in the order it received it; not because TCP doesn't put 
> packets back in order, but just so as not to trigger TCP into 
> generating spurious retransmissions.
>
>
>
> Bob
>
>     *To:* Scharf, Michael; [email-masked] <mailto:[email-masked]>
>     *Cc:* [email-masked] <mailto:[email-masked]>
>       * L4S takes the end-to-end approach saying that it is sufficient
>         to do resequencing in one place (the receiving host) if it is
>         needed. Then any resequencing delay only affects that flow.
>         hop-by-hop. This means there appears to be no resequencing
>         needed on the receiver (except if there's re-ordering on the
>         final hop).
>
>         approach), only the the worst case latency can be guaranteed,
>         and every packet has to have that same worst-case latency.
>       * When you leave resequencing to the receiver (end-to-end
>         approach), most of the packets arrive earlier than they would
>         with DETNET, but out of order ones might not. Then the
>         application chooses whether it wants to wait.
>
>     [ms] My point is different: There seem to be other working groups
>     in the IETF that do talk about “low latency” as well but
>     apparently they also do need in-order delivery of packets, too. So
>     the assumption that e.g. a link layer technology can simply
>     disable in-order delivery for all “low latency” applications seems
>     not correct. The reality may be a bit more complex.
>
>     The 3 DupACK rule in TCP led the Internet not to follow the
>     end-to-end principle on re-sequencing. Now we're removing the 3
>     DupACK rule, we can take advantage of the e2e principle.
>
>     [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC 5681
>     is standards track. So the bar for “removing” that rule for TCP is
>     not low…
>
>     errors is not efficient. For in-order delivery there are tradeoffs
>     as well, see e.g. Section 4 in draft-ietf-tcpm-rack-04. There may
>     be no free lunch.
>
>     Of course, an application might choose to use TCP and L4S, rather
>     than UDP and L4S (e.g QUIC). Then there could still be HoL
>     blocking in the receiving TCP stack. But at least there's no HoL
>     blocking in the network, and the app can choose between stream
>     (TCP) or datagram (UDP).
>
>     [ms] If LRO/GRO in the receiving TCP stack gets messed up, I fail
>     to see the benefit (see draft-ietf-tcpm-rack-04).
>
>     Michael
>
>
>     Bob
>

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 9 Nov 2018 17:44:04 +0000

I’ll not comment on link technology implementation, scheduler and queue design assumptions in Appendix A.1.7 of draft-ietf-tsvwg-ecn-l4s-id-05. People e.g. in RTG area may be more familiar with the implications of internal designs e.g. inside a router. IMHO the best way to get routing experts into the loop before further discussing what is currently listed in Appendix A.1.7.

Thus, I limit my response to the actual wording in the main part of draft-ietf-tsvwg-ecn-l4s-id-05:

    o  A scalable congestion control MUST detect loss by counting in
      units of time, which is scalable, and MUST NOT count in units of
      packets (as in the 3 DupACK rule of traditional TCP), which is not
      scalable (see Appendix A.1.7<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-A.1.7> for rationale).

As individual contributor, I disagree with this requirement. I would probably be OK with a wording that a TCP sender must be able to tolerate reordering beyond 3 Duplicate ACKs. I believe it does not matter how a sender realizes this internally.

And if that requirement is not just removed, I believe the wording “traditional TCP” has to be replaced by “standard TCP”.

BTW, also, in the rest of the document there has to be a clear distinction between experiments and standards. History tells us that some experiments can evolve into standards track specifications, while other experiments may fail. And there is no way of knowing that in advance.

Michael

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, November 9, 2018 11:08 AM
To: Scharf, Michael; [email-masked]
Cc: [email-masked]
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,
On 09/11/2018 06:21, Scharf, Michael wrote:
I don’t see the fundamental difference between a link technology that guarantees in-order delivery within a flow (say by flow hashing, see ECMP) vs. some link scheme that guarantees in-order delivery, say, only for some fraction of the traffic characterized by some other means (say, other header bits).
[The "other means" and "other bits" wording is pretty ambiguous here. I reckon I can guess what you mean but then I don't know why you're saying this, 'cos I don't think I've disagreed with you on this. So perhaps you could be more specific and we might be able to see why you've brought this up.]

Also I wasn't really talking about all "link technology that guarantees in-order delivery within a flow". {Note 1} I was talking about link technology where all flows are blocked while waiting to get the aggregate back in order that it was in when it arrived. I specifically said:

  *   A packet resequencing buffer after allowing for link-layer retransmissions is common for radio link technologies such as LTE (PDCP layer) and 802.11 WiFi.
  *   A packet resequencing buffer after merging multiple bonded link channels is common for other access link technologies such as downstream DOCSIS bonded channels.

Clearly, reordering is a problem that needs to be discussed in the IETF as a whole. And we could discuss whether a revision of RFC 3366 is needed.
No. There is no proposal to alter general guidance for all links like RFC 3366. No-one is saying this.

There is not even a proposal to require or even recommend that L4S-specific links do anything.

The specific proposal is to require L4S sources to use a RACK-like scheme, as a condition for setting the ECT(1) codepoint. See the last bullet in S.4.3 of draft-ietf-tsvwg-ecn-l4s-id-05<https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#section-4.3>.

This says nothing about what L4S links MUST, SHOULD or MAY do. It just gives an L4S link an opportunity not to have to do resequencing.

L4S is merely an example of a case where we can mandate that there will be no non-RACK traffic in a link, which creates an opportunity for those who are implementing the L4S DualQ Coupled AQM to ignore the RFC3366 resequencing advice for the L4S queue (but we are not changing the advice, because L4S is only an experiment). The prize is to remove the head-of-line blocking problem for their users.

This is not L4S-specific, altho there are no other examples at the moment where this opportunity is possible (so in that sense it is currently L4S-specific).

The message has to be handled really carefully, because the opportunities for misunderstanding are enormous (as evidenced by this thread, and we haven't even started talking to non-transport people yet).

So Gorry and David B want me to draft a short I-D that they will word-smith and author as tsvwg chairs to make the situation clear.

I have said I don't want this discussion to be officially raised outside the transport area until we are sure that a variant of RACK is even feasible without any packet counting at all (i.e. without the initial 3-DupACK rule in current RACK). So I'll prioritize working that out for the next week or so, rather than drafting anything.

Bob

{Note 1}: I hadn't considered that this would affect technologies like ECMP that maintain order without a resequencing buffer and without introducing HoL blocking. But actually, the consequence there would be that L4S traffic could be sprayed by ECMP without per-flow hashing.

Michael

Michael,

________________

Bob

Michael,

Bob

Michael

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 9 Nov 2018 18:03:45 +0000

Michael,

Please address the scaling rationale in the appendix for why it does 
matter how a sender realizes this internally.

The list of access links types I mentioned are rarely connected directly 
to routers.

Bob

On 09/11/2018 17:44, Scharf, Michael wrote:
>
> I’ll not comment on link technology implementation, scheduler and 
> queue design assumptions in Appendix A.1.7 of 
> draft-ietf-tsvwg-ecn-l4s-id-05. People e.g. in RTG area may be more 
> familiar with the implications of internal designs e.g. inside a 
> router. IMHO the best way to get routing experts into the loop before 
> further discussing what is currently listed in Appendix A.1.7.
>
>       scalable (see Appendix A.1.7 
> <https://tools.ietf.org/html/draft-ietf-tsvwg-ecn-l4s-id-05#appendix-A.1.7>for 
> rationale).
>
> As individual contributor, I disagree with this requirement. I would 
> probably be OK with a wording that a TCP sender must be able to 
> tolerate reordering beyond 3 Duplicate ACKs. I believe it does not 
> matter how a sender realizes this internally.
>
> And if that requirement is not just removed, I believe the wording 
> “traditional TCP” has to be replaced by “standard TCP”.
>
> BTW, also, in the rest of the document there has to be a clear 
> distinction between experiments and standards. History tells us that 
> some experiments can evolve into standards track specifications, while 
> other experiments may fail. And there is no way of knowing that in 
> advance.
>
> Michael
>
> *From:*Bob Briscoe [mailto:[email-masked]]
> *Sent:* Friday, November 9, 2018 11:08 AM
>     I don’t see the fundamental difference between a link technology
>     that guarantees in-order delivery within a flow (say by flow
>     hashing, see ECMP) vs. some link scheme that guarantees in-order
>     delivery, say, only for some fraction of the traffic characterized
>     by some other means (say, other header bits).
>
> [The "other means" and "other bits" wording is pretty ambiguous here. 
> I reckon I can guess what you mean but then I don't know why you're 
> saying this, 'cos I don't think I've disagreed with you on this. So 
> perhaps you could be more specific and we might be able to see why 
> you've brought this up.]
>
> Also I wasn't really talking about all "link technology that 
> guarantees in-order delivery within a flow". {Note 1} I was talking 
> about link technology where all flows are blocked while waiting to get 
> the aggregate back in order that it was in when it arrived. I 
> specifically said:
>
> L4S is merely an example of a case where we can mandate that there 
> will be no non-RACK traffic in a link, which creates an opportunity 
> for those who are implementing the L4S DualQ Coupled AQM to ignore the 
> RFC3366 resequencing advice for the L4S queue (but we are not changing 
> the advice, because L4S is only an experiment). The prize is to remove 
> the head-of-line blocking problem for their users.
>
> I have said I don't want this discussion to be officially raised 
> outside the transport area until we are sure that a variant of RACK is 
> even feasible without any packet counting at all (i.e. without the 
> initial 3-DupACK rule in current RACK). So I'll prioritize working 
> that out for the next week or so, rather than drafting anything.
>
>
>
> Bob
>

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 9 Nov 2018 18:19:04 +0000

Michael,

On 09/11/2018 18:03, Bob Briscoe wrote:
> The list of access links types I mentioned are rarely connected 
> directly to routers.
Correction to my own statement: Only LTE links are rarely connected to 
routers , WiFi are, and DOCSIS are (the CMTS).

The radio link (ARQ) issue is not to do with routing.
One can think of bonded links as a form of L2 multipath, and therefore 
broadly related to routing.
But it's access link expertise that's needed here, not routing 
expertise, I would say.

Bob

>> some experiments can evolve into standards track specifications, 
>> while other experiments may fail. And there is no way of knowing that 
>> in advance.
>>
>> Michael
>>
>>     delivery, say, only for some fraction of the traffic
>>     characterized by some other means (say, other header bits).
>>
>> about link technology where all flows are blocked while waiting to 
>> get the aggregate back in order that it was in when it arrived. I 
>> specifically said:
>>
>> delivery' for a link does not mean in order of TCP sequence number. 
>> It means 'delivered in the order that packets arrived at the link' 
>> (the link ingress adds its own sequence numbers, and the link egress 
>> This says nothing about what L4S links MUST, SHOULD or MAY do. It 
>> just gives an L4S link an opportunity not to have to do resequencing.
>>
>>
>> for those who are implementing the L4S DualQ Coupled AQM to ignore 
>> the RFC3366 resequencing advice for the L4S queue (but we are not 
>> changing the advice, because L4S is only an experiment). The prize is 
>> to remove the head-of-line blocking problem for their users.
>>
>> outside the transport area until we are sure that a variant of RACK 
>> is even feasible without any packet counting at all (i.e. without the 
>> {Note 1}: I hadn't considered that this would affect technologies 
>> like ECMP that maintain order without a resequencing buffer and 
>> without introducing HoL blocking. But actually, the consequence there 
>> would be that L4S traffic could be sprayed by ECMP without per-flow 
>> hashing.
>>
>>
>> Michael
>>
>> delivery' for a link does not mean in order of TCP sequence number. 
>> It means 'delivered in the order that packets arrived at the link' 
>> (the link ingress adds its own sequence numbers, and the link egress 
>> in-order delivery per-hop means every flow is guaranteed to always 
>> get the worst delay from every link, which would have been 
>> experienced by only one flow without per-hop in-order delivery.
>>
>> resequencing buffer as the penultimate hop, screwed onto the 
>> receiving machine if necessary, rather than require per-hop 
>> resequencing in the network.
>>
>> ________________
>>       * L4S takes the end-to-end approach saying that it is
>>         sufficient to do resequencing in one place (the receiving
>>         host) if it is needed. Then any resequencing delay only
>>         affects that flow.
>>     apparently they also do need in-order delivery of packets, too.
>>     So the assumption that e.g. a link layer technology can simply
>>     disable in-order delivery for all “low latency” applications
>>     seems not correct. The reality may be a bit more complex.
>>
>>     is standards track. So the bar for “removing” that rule for TCP
>>     is not low…
>>
>>     errors is not efficient. For in-order delivery there are
>>     tradeoffs as well, see e.g. Section 4 in draft-ietf-tcpm-rack-04.
>>     There may be no free lunch.
>>

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 9 Nov 2018 18:28:27 +0000

I may miss something, but I read the outcome of the appendix as a requirement “must be more robust to reordering”. As far as I understand, that can be implemented in different ways.

And I may be wrong, but those link technologies could be well implemented in a router and then one needs to analyze how that whole architecture fits into the scheduler, queue, policer, and line card hardware design of a router. But that is something routing people know better than me. My point is just to get the experts into the loop early.

Michael

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, November 9, 2018 7:04 PM
To: Scharf, Michael; [email-masked]
Cc: [email-masked]
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

Please address the scaling rationale in the appendix for why it does matter how a sender realizes this internally.

The list of access links types I mentioned are rarely connected directly to routers.

Bob

Michael

Michael,

Bob

Michael

Michael,

________________

Bob

Michael,

Bob

Michael

--

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Fri, 9 Nov 2018 18:52:23 +0000

Michael,

The idea is that (as now in the RACK draft since discussion on the tcpm 
list earlier this year under a subject something like vicious circle or 
virtuous circle) there will be a limit on the initial RACK reordering 
window expressed in fractional RTT units.

Link vendors know the min e2e RTT of the paths that their links are 
usually deployed over. Therefore they can estimate the max reordering 
they allow their links to introduce.

I don't expect this limit to be set in stone while both RACK and L4S are 
experimental. But as both these experiments progress, I think the 
industry will get a better idea of where the limit should lie.

It was the realization that, if TCP adapted its reordering window to the 
measured reordering degree without bound, the L2 world would continually 
increase the reordering degree of their links (the vicious circle).

If there is any aspect of this limit expressed in units of packets, the 
limit will decrease in time as flow rates increase, and therefore not 
serve to give the L2 world any clear guidance - no de facto limit agreed 
between the L4 world and the L2 world.

If we keep the limit in units of RTT, hopefully then we have a virtuous 
circle, not a vicious circle.

Bob

On 09/11/2018 18:28, Scharf, Michael wrote:
>
> I may miss something, but I read the outcome of the appendix as a 
> requirement “must be more robust to reordering”. As far as I 
> understand, that can be implemented in different ways.
>
> And I may be wrong, but those link technologies could be well 
> implemented in a router and then one needs to analyze how that whole 
> architecture fits into the scheduler, queue, policer, and line card 
> hardware design of a router. But that is something routing people know 
> better than me. My point is just to get the experts into the loop early.
>
> Michael
>
> *Sent:* Friday, November 9, 2018 7:04 PM
>     draft-ietf-tsvwg-ecn-l4s-id-05. People e.g. in RTG area may be
>     more familiar with the implications of internal designs e.g.
>     inside a router. IMHO the best way to get routing experts into the
>     loop before further discussing what is currently listed in
>     Appendix A.1.7.
>
>     As individual contributor, I disagree with this requirement. I
>     would probably be OK with a wording that a TCP sender must be able
>     to tolerate reordering beyond 3 Duplicate ACKs. I believe it does
>     not matter how a sender realizes this internally.
>
>     distinction between experiments and standards. History tells us
>     that some experiments can evolve into standards track
>     specifications, while other experiments may fail. And there is no
>     way of knowing that in advance.
>
>     Michael
>
>         I don’t see the fundamental difference between a link
>         technology that guarantees in-order delivery within a flow
>         (say by flow hashing, see ECMP) vs. some link scheme that
>         guarantees in-order delivery, say, only for some fraction of
>         the traffic characterized by some other means (say, other
>         header bits).
>
>     [The "other means" and "other bits" wording is pretty ambiguous
>     here. I reckon I can guess what you mean but then I don't know why
>     you're saying this, 'cos I don't think I've disagreed with you on
>     this. So perhaps you could be more specific and we might be able
>     to see why you've brought this up.]
>
>     guarantees in-order delivery within a flow". {Note 1} I was
>     talking about link technology where all flows are blocked while
>     waiting to get the aggregate back in order that it was in when it
>     arrived. I specifically said:
>
>
>     delivery' for a link does not mean in order of TCP sequence
>     number. It means 'delivered in the order that packets arrived at
>     the link' (the link ingress adds its own sequence numbers, and the
>     link egress buffers them until it can send out in the same order,
>     or until a time-out or max no. of link retransmissions).
>
>     For example:
>
>         retransmissions is common for radio link technologies such as
>         LTE (PDCP layer) and 802.11 WiFi.
>       * A packet resequencing buffer after merging multiple bonded
>         link channels is common for other access link technologies
>         such as downstream DOCSIS bonded channels.
>
>
>
>
>     IETF as a whole. And we could discuss whether a revision of RFC
>     3366 is needed.
>
>     The specific proposal is to require L4S /sources/ to use a
>     RACK-like scheme, as a condition for setting the ECT(1) codepoint.
>     See the last bullet in S.4.3 of draft-ietf-tsvwg-ecn-l4s-id-05
>     will be no non-RACK traffic in a link, which creates an
>     opportunity for those who are implementing the L4S DualQ Coupled
>     AQM to ignore the RFC3366 resequencing advice for the L4S queue
>     (but we are not changing the advice, because L4S is only an
>     experiment). The prize is to remove the head-of-line blocking
>     problem for their users.
>
>     opportunities for misunderstanding are enormous (as evidenced by
>     this thread, and we haven't even started talking to non-transport
>     people yet).
>
>     outside the transport area until we are sure that a variant of
>     RACK is even feasible without any packet counting at all (i.e.
>     without the initial 3-DupACK rule in current RACK). So I'll
>     prioritize working that out for the next week or so, rather than
>     drafting anything.
>
>
>
>     Bob
>
>     without introducing HoL blocking. But actually, the consequence
>     there would be that L4S traffic could be sprayed by ECMP without
>     per-flow hashing.
>
>
>
>     Michael
>
>     If an application needs all the packets (i.e. a reliable
>     protocol), it's not useful to reduce the latency of some packets
>     but not all. That's where the in-order requirement comes from.
>
>     /application/ still gets in-order delivery from TCP. It just
>     doesn't have to require links to provide in order delivery as well.
>
>     delivery' for a link does not mean in order of TCP sequence
>     number. It means 'delivered in the order that packets arrived at
>     the link' (the link ingress adds its own sequence numbers, and the
>     link egress buffers them until it can send out in the same order,
>     or until a time-out or max no. of link retransmissions).
>
>     So, if the link loses a packet from flow A (the link isn't aware
>     of microflows, I'm just saying the packet happens to be from flow
>     A), link-layer in-order delivery will hold back all packets sent
>     after that packet (from all flows, not just flow A) until the link
>     repairs the gap or gives up.
>
>     packets. But wherever there is variable delay in a link,
>     guaranteed in-order delivery per-hop means every flow is
>     guaranteed to always get the worst delay from every link, which
>     would have been experienced by only one flow without per-hop
>     in-order delivery.
>
>     So, why do Internet links often ensure in-order delivery even
>     though TCP puts everything in order for the application?
>
>     Short answer: the 3 DupACK rule (which is to do with
>     loss-recovery, a different issue from the discussion above).
>
>     Long answer: As flow rates have scaled up, the typical time
>     between 3 packet arrivals has become so small that TCP's 3-DupACK
>     rule makes links have to provide delivery with only a very small
>     re-ordering degree - so small that it's easier for a link to just
>     deliver everything in the order it received it; not because TCP
>     doesn't put packets back in order, but just so as not to trigger
>     TCP into generating spurious retransmissions.
>
>
>
>     Bob
>
>         This is merely a symptom of a difference in opinion on where
>         the resequencing function should be located.
>
>           * The DETNET approach is saying the resequencing must be
>             done hop-by-hop. This means there appears to be no
>             resequencing needed on the receiver (except if there's
>             re-ordering on the final hop).
>
>           * In order to guarantee no resequencing in the network
>             (DETNET approach), only the the worst case latency can be
>             guaranteed, and every packet has to have that same
>             worst-case latency.
>             approach), most of the packets arrive earlier than they
>             would with DETNET, but out of order ones might not. Then
>             the application chooses whether it wants to wait.
>
>         [ms] My point is different: There seem to be other working
>         groups in the IETF that do talk about “low latency” as well
>         but apparently they also do need in-order delivery of packets,
>         too. So the assumption that e.g. a link layer technology can
>         simply disable in-order delivery for all “low latency”
>         applications seems not correct. The reality may be a bit more
>         complex.
>
>         end-to-end principle on re-sequencing. Now we're removing the
>         3 DupACK rule, we can take advantage of the e2e principle.
>
>         [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and RFC
>         5681 is standards track. So the bar for “removing” that rule
>         for TCP is not low…
>
>         instance, relying on the end-to-end principle for recovery of
>         bit errors is not efficient. For in-order delivery there are
>         tradeoffs as well, see e.g. Section 4 in
>         draft-ietf-tcpm-rack-04. There may be no free lunch.
>
>         Of course, an application might choose to use TCP and L4S,
>         rather than UDP and L4S (e.g QUIC). Then there could still be
>         HoL blocking in the receiving TCP stack. But at least there's
>         no HoL blocking in the network, and the app can choose between
>         stream (TCP) or datagram (UDP).
>
>         [ms] If LRO/GRO in the receiving TCP stack gets messed up, I
>         fail to see the benefit (see draft-ietf-tcpm-rack-04).
>
>         Michael
>
>
>         Bob
>

---

From: Praveen Balasubramanian <[email-masked]>
Date: Fri, 9 Nov 2018 19:10:16 +0000

>> Therefore they can estimate the max reordering they allow their links to introduce
IMO L2 shouldn’t introduce more reordering just because L4 congestion control is more resilient to it. There is other performance impact due to reordering: LRO / GRO / RSC opportunity will be reduced, more memory and CPU cost to do reassembly at L4 (this is an active area for remote attacks) etc. IMO L4 guidance should remain that link vendors must avoid reordering as much as possible and not build solutions that cause frequent or consistent reordering.

From: tcpm <[email-masked]> On Behalf Of Bob Briscoe
Sent: Friday, November 9, 2018 10:52 AM
To: Scharf, Michael <[email-masked]>; [email-masked]
Cc: [email-masked]
Subject: Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

The idea is that (as now in the RACK draft since discussion on the tcpm list earlier this year under a subject something like vicious circle or virtuous circle) there will be a limit on the initial RACK reordering window expressed in fractional RTT units.

Link vendors know the min e2e RTT of the paths that their links are usually deployed over. Therefore they can estimate the max reordering they allow their links to introduce.

I don't expect this limit to be set in stone while both RACK and L4S are experimental. But as both these experiments progress, I think the industry will get a better idea of where the limit should lie.

It was the realization that, if TCP adapted its reordering window to the measured reordering degree without bound, the L2 world would continually increase the reordering degree of their links (the vicious circle).

If there is any aspect of this limit expressed in units of packets, the limit will decrease in time as flow rates increase, and therefore not serve to give the L2 world any clear guidance - no de facto limit agreed between the L4 world and the L2 world.

If we keep the limit in units of RTT, hopefully then we have a virtuous circle, not a vicious circle.

Bob

Michael

Michael,

Bob

    o  A scalable congestion control MUST detect loss by counting in
      units of time, which is scalable, and MUST NOT count in units of
      packets (as in the 3 DupACK rule of traditional TCP), which is not
      scalable (see Appendix A.1.7<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tsvwg-ecn-l4s-id-05%23appendix-A.1.7&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895496498&sdata=%2Bez9OCZi1m5wx7TGHkvfG9QsJfJ0VxhsXynqWW2jXG4%3D&reserved=0> for rationale).

Michael

Michael,

The specific proposal is to require L4S sources to use a RACK-like scheme, as a condition for setting the ECT(1) codepoint. See the last bullet in S.4.3 of draft-ietf-tsvwg-ecn-l4s-id-05<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tsvwg-ecn-l4s-id-05%23section-4.3&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895506511&sdata=m%2Fn6HNQnJYbTPIo0MR8cTXKfcj0DX579HAyD9LMe4aQ%3D&reserved=0>.

Bob

Michael

Michael,

________________

Bob

Michael,

Bob

Michael

--

Bob Briscoe                               http://bobbriscoe.net/<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895506511&sdata=%2B4XolnKyPnY5Cu8ktuWXi2SJ5qboN7Br9%2B6DMrCHnmc%3D&reserved=0>

--

Bob Briscoe                               http://bobbriscoe.net/<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895516515&sdata=xZPvdqkPWt0B1RaGlo0u5Pvryjazmqjm98pV00zOMWw%3D&reserved=0>

--

Bob Briscoe                               http://bobbriscoe.net/<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895526524&sdata=SpOJ%2FduwSERDmslDytwNLNIJ5L4OxvpTzncaAx0oFrU%3D&reserved=0>

--

Bob Briscoe                               http://bobbriscoe.net/<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895526524&sdata=SpOJ%2FduwSERDmslDytwNLNIJ5L4OxvpTzncaAx0oFrU%3D&reserved=0>

--

Bob Briscoe                               http://bobbriscoe.net/<https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895536537&sdata=lyBhcXU74SU56s7qr1OUSfwaIL68LcbF7WJxtmz%2BUb0%3D&reserved=0>

---

From: "Scharf, Michael" <[email-masked]>
Date: Fri, 9 Nov 2018 19:28:48 +0000

First, I don’t know if the currently discussed parameters for the RACK reordering window are indeed future proof. Maybe TCP stacks could use different parameters in one or two years from now? So what is the real-world benefit for a link designer *now* in relying on an parameters that can change easily? And I would actually assume that the exact parameter values in a (hypothetical) future standards track specification of RACK would be a SHOULD. Different TCP stacks may pick different values. How useful is this then to a link designer?

Second, I don’t understand why one cannot try to provide some other guidance, say, in terms of number of triggered DUPACKs or the like. Let’s assume we said a stack must be able to deal with a reorder degree of 10? For which link technologies would this not be good enough?

Third, some real-world data would facilitate this discussion quite a bit, IMHO. I have a hard time to follow this proposal without examples for different RTTs, different number of parallel connections, different mixtures of traffic, etc.

Michael

From: Bob Briscoe [mailto:[email-masked]]
Sent: Friday, November 9, 2018 7:52 PM
To: Scharf, Michael; [email-masked]
Cc: [email-masked]
Subject: Re: [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

Bob

Michael

Michael,

Bob

Michael

Michael,

Bob

Michael

Michael,

________________

Bob

Michael,

Bob

Michael

--

--

--

--

--

---

From: Bob Briscoe <[email-masked]>
Date: Sat, 10 Nov 2018 01:34:39 +0000

Praveen, Michael,

The idea is not to relax ordering requirement on links. It's to make 
sure it doesn't keep getting tighter for ever.

As flow-rates get faster, the requirements on links get tighter if the 
reordering window is expressed in packets.
For instance (as I showed in my presentation), with RTT = 24ms (say), if 
the window is 12 segments, a reordering window of 3 DupACKs equates to 
6ms. But as flow rates increase, say 8 times faster (window = 96) 
reduces the time over which links have to keep packets in order to 
750μs. Another 8x faster, and links have to keep order within 94μs. And 
so on.

This cannot go on indefinitely, because the constraints on link-layer 
retransmission are constant in time (length of link / speed of light + 
processing time). If the 3 DupACK rule is not removed, link timings will 
be continually squeezed, so radio links will eventually have to give up 
ARQ, which will be awful for performance.

It will take decades before /all/ legacy (non-RACK) TCPs have stopped 
using any particular link. So links in general will still have to keep 
tightening their re-sequencing time because of legacy 3 Dup-ACK flows. 
But eventually the 3-DupACK flows that remain will tend to be those on 
unmaintained machines that will not be taking advantage of higher 
speeds. Then, assuming the RACK experiment remains successful over these 
decades, links will be able to stop tightening their reordering degree.

The idea is that, for the L4S experiment, we can remove the 3 DupACK 
rule entirely from the start, and solely express reordering in units of 
time. Then, L4S transports will evolve, presumably using similar 
RACK-like reordering logic to non-L4S transports. And we shall see where 
the minimum reordering window ends up (in time units), and hopefully 
converge on a value we can standardize - both in TCP and in other 
transports.

No L4S link would relax its ordering requirement if it led to worse TCP 
performance. So layer 4 will be in control of this process. Nonetheless, 
it is important to stop measuring reordering in packets. But hopefully, 
from the start of the L4S experiment, logical links solely bearing L4S 
traffic will not need to continually tighten their ordering requirement.

If an OS supports RACK, which yours does, then buffering on the receiver 
will remain constant in time, which will mean the memory requirement 
will grow as flow rates scale up.

You (Praveen) supported and implemented RACK before I'd even thought 
about all this. I'm also now a convert. I'm recognizing and articulating 
its wider advantages.

Bob

On 09/11/2018 19:10, Praveen Balasubramanian wrote:
> IMO L2 shouldn’t /introduce/ more reordering just because L4 
> /congestion control/ is more resilient to it. There is other 
> performance impact due to reordering: LRO / GRO / RSC opportunity will 
> be reduced, more memory and CPU cost to do reassembly at L4 (this is 
> an active area for remote attacks) etc. IMO L4 guidance should remain 
> that link vendors must avoid reordering as much as possible and not 
> build solutions that cause frequent or consistent reordering.
>
> *From:*tcpm <[email-masked]> *On Behalf Of *Bob Briscoe
> *Sent:* Friday, November 9, 2018 10:52 AM
> *To:* Scharf, Michael <[email-masked]>; [email-masked]
> *Cc:* [email-masked]
> *Subject:* Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"
>
> Michael,
>
> The idea is that (as now in the RACK draft since discussion on the 
> tcpm list earlier this year under a subject something like vicious 
> circle or virtuous circle) there will be a limit on the initial RACK 
> reordering window expressed in fractional RTT units.
>
> I don't expect this limit to be set in stone while both RACK and L4S 
> are experimental. But as both these experiments progress, I think the 
> It was the realization that, if TCP adapted its reordering window to 
> the measured reordering degree without bound, the L2 world would 
> continually increase the reordering degree of their links (the vicious 
> circle).
>
> If there is any aspect of this limit expressed in units of packets, 
> the limit will decrease in time as flow rates increase, and therefore 
> not serve to give the L2 world any clear guidance - no de facto limit 
> agreed between the L4 world and the L2 world.
>
>     implemented in a router and then one needs to analyze how that
>     whole architecture fits into the scheduler, queue, policer, and
>     line card hardware design of a router. But that is something
>     routing people know better than me. My point is just to get the
>     experts into the loop early.
>
>     Michael
>
>     Please address the scaling rationale in the appendix for why it
>     does matter how a sender realizes this internally.
>
>         I’ll not comment on link technology implementation, scheduler
>         and queue design assumptions in Appendix A.1.7 of
>         inside a router. IMHO the best way to get routing experts into
>         the loop before further discussing what is currently listed in
>         Appendix A.1.7.
>
>         <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tsvwg-ecn-l4s-id-05%23appendix-A.1.7&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895496498&sdata=%2Bez9OCZi1m5wx7TGHkvfG9QsJfJ0VxhsXynqWW2jXG4%3D&reserved=0>for
>         rationale).
>
>         would probably be OK with a wording that a TCP sender must be
>         able to tolerate reordering beyond 3 Duplicate ACKs. I believe
>         it does not matter how a sender realizes this internally.
>
>         And if that requirement is not just removed, I believe the
>         wording “traditional TCP” has to be replaced by “standard TCP”.
>
>         distinction between experiments and standards. History tells
>         us that some experiments can evolve into standards track
>         specifications, while other experiments may fail. And there is
>         no way of knowing that in advance.
>
>         Michael
>
>             guarantees in-order delivery, say, only for some fraction
>             of the traffic characterized by some other means (say,
>             other header bits).
>
>         [The "other means" and "other bits" wording is pretty
>         ambiguous here. I reckon I can guess what you mean but then I
>         don't know why you're saying this, 'cos I don't think I've
>         disagreed with you on this. So perhaps you could be more
>         specific and we might be able to see why you've brought this up.]
>
>         talking about link technology where all flows are blocked
>         while waiting to get the aggregate back in order that it was
>         in when it arrived. I specifically said:
>
>
>
>         Note that there can be multiple flows over one link. So
>         'in-order delivery' for a link does not mean in order of TCP
>         sequence number. It means 'delivered in the order that packets
>         arrived at the link' (the link ingress adds its own sequence
>         numbers, and the link egress buffers them until it can send
>         out in the same order, or until a time-out or max no. of link
>         retransmissions).
>
>         For example:
>
>             retransmissions is common for radio link technologies such
>             as LTE (PDCP layer) and 802.11 WiFi.
>         Clearly, reordering is a problem that needs to be discussed in
>         the IETF as a whole. And we could discuss whether a revision
>         of RFC 3366 is needed.
>
>         No. There is no proposal to alter general guidance for all
>         links like RFC 3366. No-one is saying this.
>
>         RACK-like scheme, as a condition for setting the ECT(1)
>         codepoint. See the last bullet in S.4.3 of
>         draft-ietf-tsvwg-ecn-l4s-id-05
>         <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-ietf-tsvwg-ecn-l4s-id-05%23section-4.3&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895506511&sdata=m%2Fn6HNQnJYbTPIo0MR8cTXKfcj0DX579HAyD9LMe4aQ%3D&reserved=0>.
>
>
>         This says nothing about what L4S links MUST, SHOULD or MAY do.
>         It just gives an L4S link an opportunity not to have to do
>         resequencing.
>
>
>
>
>         L4S is merely an example of a case where we can mandate that
>         there will be no non-RACK traffic in a link, which creates an
>         opportunity for those who are implementing the L4S DualQ
>         Coupled AQM to ignore the RFC3366 resequencing advice for the
>         L4S queue (but we are not changing the advice, because L4S is
>         only an experiment). The prize is to remove the head-of-line
>         blocking problem for their users.
>
>         This is not L4S-specific, altho there are no other examples at
>         the moment where this opportunity is possible (so in that
>         sense it is /currently/ L4S-specific).
>
>
>         opportunities for misunderstanding are enormous (as evidenced
>         by this thread, and we haven't even started talking to
>         non-transport people yet).
>
>         So Gorry and David B want me to draft a short I-D that they
>         will word-smith and author as tsvwg chairs to make the
>         situation clear.
>
>         I have said I don't want this discussion to be officially
>         raised outside the transport area until we are sure that a
>         variant of RACK is even feasible without any packet counting
>         at all (i.e. without the initial 3-DupACK rule in current
>         RACK). So I'll prioritize working that out for the next week
>         or so, rather than drafting anything.
>
>
>
>         Bob
>
>         {Note 1}: I hadn't considered that this would affect
>         technologies like ECMP that maintain order without a
>         resequencing buffer and without introducing HoL blocking. But
>         actually, the consequence there would be that L4S traffic
>         could be sprayed by ECMP without per-flow hashing.
>
>
>
>
>         Michael
>
>         protocol), it's not useful to reduce the latency of some
>         packets but not all. That's where the in-order requirement
>         comes from.
>
>         Note that there can be multiple flows over one link. So
>         'in-order delivery' for a link does not mean in order of TCP
>         sequence number. It means 'delivered in the order that packets
>         arrived at the link' (the link ingress adds its own sequence
>         numbers, and the link egress buffers them until it can send
>         out in the same order, or until a time-out or max no. of link
>         retransmissions).
>
>         So, if the link loses a packet from flow A (the link isn't
>         aware of microflows, I'm just saying the packet happens to be
>         from flow A), link-layer in-order delivery will hold back all
>         packets sent after that packet (from all flows, not just flow
>         A) until the link repairs the gap or gives up.
>
>         Of course, someone building a network for DETNET would avoid
>         link technologies like WiFi or LTE that frequently lose and
>         retransmit packets. But wherever there is variable delay in a
>         link, guaranteed in-order delivery per-hop means every flow is
>         guaranteed to always get the worst delay from every link,
>         which would have been experienced by only one flow without
>         per-hop in-order delivery.
>
>         machinery that just expects everything to arrive in order and
>         to a clocked schedule. But it's always as effective to deploy
>         a resequencing buffer as the penultimate hop, screwed onto the
>         between 3 packet arrivals has become so small that TCP's
>         3-DupACK rule makes links have to provide delivery with only a
>         very small re-ordering degree - so small that it's easier for
>         a link to just deliver everything in the order it received it;
>         not because TCP doesn't put packets back in order, but just so
>         as not to trigger TCP into generating spurious retransmissions.
>
>
>
>         Bob
>
>             This is merely a symptom of a difference in opinion on
>             where the resequencing function should be located.
>
>                 sufficient to do resequencing in one place (the
>                 receiving host) if it is needed. Then any resequencing
>                 delay only affects that flow.
>                 (DETNET approach), only the the worst case latency can
>                 be guaranteed, and every packet has to have that same
>                 worst-case latency.
>               * When you leave resequencing to the receiver
>                 (end-to-end approach), most of the packets arrive
>                 earlier than they would with DETNET, but out of order
>                 ones might not. Then the application chooses whether
>                 it wants to wait.
>
>             groups in the IETF that do talk about “low latency” as
>             well but apparently they also do need in-order delivery of
>             packets, too. So the assumption that e.g. a link layer
>             technology can simply disable in-order delivery for all
>             “low latency” applications seems not correct. The reality
>             may be a bit more complex.
>
>             The 3 DupACK rule in TCP led the Internet not to follow
>             the end-to-end principle on re-sequencing. Now we're
>             removing the 3 DupACK rule, we can take advantage of the
>             e2e principle.
>
>             [ms] For TCP, three DupACKS are a SHOULD in RFC 5681 and
>             RFC 5681 is standards track. So the bar for “removing”
>             that rule for TCP is not low…
>
>             [ms] Also, the end-to-end principle comes with tradeoffs.
>             For instance, relying on the end-to-end principle for
>             recovery of bit errors is not efficient. For in-order
>             delivery there are tradeoffs as well, see e.g. Section 4
>             in draft-ietf-tcpm-rack-04. There may be no free lunch.
>
>             rather than UDP and L4S (e.g QUIC). Then there could still
>             be HoL blocking in the receiving TCP stack. But at least
>             there's no HoL blocking in the network, and the app can
>             choose between stream (TCP) or datagram (UDP).
>
>             [ms] If LRO/GRO in the receiving TCP stack gets messed up,
>             I fail to see the benefit (see draft-ietf-tcpm-rack-04).
>
>             Michael
>
>
>             Bob
>
>             Bob Briscoehttp://bobbriscoe.net/  <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895506511&sdata=%2B4XolnKyPnY5Cu8ktuWXi2SJ5qboN7Br9%2B6DMrCHnmc%3D&reserved=0>
>
>
>
>
>
>
>         -- 
>
>         Bob Briscoehttp://bobbriscoe.net/  <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895516515&sdata=xZPvdqkPWt0B1RaGlo0u5Pvryjazmqjm98pV00zOMWw%3D&reserved=0>
>
>
>
>
>
>         -- 
>
>         Bob Briscoehttp://bobbriscoe.net/  <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895526524&sdata=SpOJ%2FduwSERDmslDytwNLNIJ5L4OxvpTzncaAx0oFrU%3D&reserved=0>
>
>
>
>
>     -- 
>
>     Bob Briscoehttp://bobbriscoe.net/  <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895526524&sdata=SpOJ%2FduwSERDmslDytwNLNIJ5L4OxvpTzncaAx0oFrU%3D&reserved=0>
>
>
>
> -- 
> Bob Briscoehttp://bobbriscoe.net/  <https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fbobbriscoe.net%2F&data=02%7C01%7Cpravb%40microsoft.com%7C547e503e218549483f3008d6467495b9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636773863895536537&sdata=lyBhcXU74SU56s7qr1OUSfwaIL68LcbF7WJxtmz%2BUb0%3D&reserved=0>

---

From: "Scharf, Michael" <[email-masked]>
Date: Sat, 10 Nov 2018 08:13:46 +0000

Hi Bob,

I believe more empirical evidence is needed that this is a problem that need to be addressed.

For instance, for what link technologies does this problem exist in reality?

Your example with 94us seems to imply a data rate of 384 Mbit/s for a single TCP stream. That is probably not what the vast majority of TCP connections use. So is there an engineering need?

And I don’t get why keeping reordering within 94us is so much of a challenge for a link layer technology. I would really like to learn technical reasons why (mostly) in-order delivery on links cannot be implemented now or in future.

If we indeed run into a real-world technical problem, it wouldn’t help to solve that problem for L4S – it must be solved for all Internet traffic then and the overall solution would (probably) be different to the one used in an experiment.

To me, one of the fundamental requirements on a packet network is that the network invests some reasonable effort to keep packets in order on a given path – albeit a packet network is not always perfect and that has always been OK. I am not convinced so far that it makes sense to give up that design guideline for link technologies – not even for L4S.

Michael

From: Bob Briscoe [mailto:[email-masked]]
Sent: Saturday, November 10, 2018 2:35 AM
To: Praveen Balasubramanian; Scharf, Michael; [email-masked]
Cc: [email-masked]
Subject: Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"

Praveen, Michael,

The idea is not to relax ordering requirement on links. It's to make sure it doesn't keep getting tighter for ever.

As flow-rates get faster, the requirements on links get tighter if the reordering window is expressed in packets.
For instance (as I showed in my presentation), with RTT = 24ms (say), if the window is 12 segments, a reordering window of 3 DupACKs equates to 6ms. But as flow rates increase, say 8 times faster (window = 96) reduces the time over which links have to keep packets in order to 750μs. Another 8x faster, and links have to keep order within 94μs. And so on.

This cannot go on indefinitely, because the constraints on link-layer retransmission are constant in time (length of link / speed of light + processing time). If the 3 DupACK rule is not removed, link timings will be continually squeezed, so radio links will eventually have to give up ARQ, which will be awful for performance.

It will take decades before /all/ legacy (non-RACK) TCPs have stopped using any particular link. So links in general will still have to keep tightening their re-sequencing time because of legacy 3 Dup-ACK flows. But eventually the 3-DupACK flows that remain will tend to be those on unmaintained machines that will not be taking advantage of higher speeds. Then, assuming the RACK experiment remains successful over these decades, links will be able to stop tightening their reordering degree.

The idea is that, for the L4S experiment, we can remove the 3 DupACK rule entirely from the start, and solely express reordering in units of time. Then, L4S transports will evolve, presumably using similar RACK-like reordering logic to non-L4S transports. And we shall see where the minimum reordering window ends up (in time units), and hopefully converge on a value we can standardize - both in TCP and in other transports.

No L4S link would relax its ordering requirement if it led to worse TCP performance. So layer 4 will be in control of this process. Nonetheless, it is important to stop measuring reordering in packets. But hopefully, from the start of the L4S experiment, logical links solely bearing L4S traffic will not need to continually tighten their ordering requirement.

If an OS supports RACK, which yours does, then buffering on the receiver will remain constant in time, which will mean the memory requirement will grow as flow rates scale up.

You (Praveen) supported and implemented RACK before I'd even thought about all this. I'm also now a convert. I'm recognizing and articulating its wider advantages.

Bob

From: tcpm <[email-masked]><mailto:[email-masked]> On Behalf Of Bob Briscoe
Sent: Friday, November 9, 2018 10:52 AM
To: Scharf, Michael <[email-masked]><mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [tcpm] [tsvwg] Cross-area alignment on "L4S and RACK"

Michael,

Bob

Michael

Michael,

Bob

Michael

Michael,

Bob

Michael

Michael,

________________

Bob

Michael,

Bob

Michael

--

--

--

--

--

--

---

From: "Scheffenegger, Richard" <[email-masked]>
Date: Thu, 22 Nov 2018 11:59:36 +0100

Hi,

Just to chime in here, for a (unfortunately common) real world example 
that I tend to run into more frequently recently:

Some implementations of (L2 or L3) ECMP appear to perform regular / 
load-triggered rebalancing events. I have observed these in particular 
with certain VSwitch implementations, as well as some modern, 
proprietary L2 "trill-like" switch topologies.

While there are adaptive experimental mechanisms in TCP to deal with 
relatively consistent reordering (measured in packets), the timescales 
involved in my examples above are such, that these adaptive variants 
really can not cope properly.

And the argument, to measure the reordering extent in units of time 
rather than packets seems sensible to me, as it bounds the maximum delay 
incurred before a retransmission happens.

In the above mentioned examples, doing rebalance events only when the 
relative buffer utilization allows for a bounded (in time) reorder 
extent should be relatively easy to work out. Compared to the status quo 
(where the potential worst case reordering extent is not considered), 
this seems to be an improvement.

Best regards,
   Richard

Am 09.11.2018 um 20:28 schrieb Scharf, Michael:
> First, I don’t know if the currently discussed parameters for the RACK 
> reordering window are indeed future proof. Maybe TCP stacks could use 
> different parameters in one or two years from now? So what is the 
> real-world benefit for a link designer **now** in relying on an 
> parameters that can change easily? And I would actually assume that the 
> exact parameter values in a (hypothetical) future standards track 
> specification of RACK would be a SHOULD. Different TCP stacks may pick 
> different values. How useful is this then to a link designer?
> 
> Second, I don’t understand why one cannot try to provide some other 
> guidance, say, in terms of number of triggered DUPACKs or the like. 
> Let’s assume we said a stack must be able to deal with a reorder degree 
> of 10? For which link technologies would this not be good enough?
> 
> Third, some real-world data would facilitate this discussion quite a 
> bit, IMHO. I have a hard time to follow this proposal without examples 
> for different RTTs, different number of parallel connections, different 
> mixtures of traffic, etc.
> 
> Michael
> 
> *Sent:* Friday, November 9, 2018 7:52 PM
>     hardware design of a router. But that is something routing people
>     know better than me. My point is just to get the experts into the
>     loop early.
> 
>     Michael
> 
>         here. I reckon I can guess what you mean but then I don't know
>         why you're saying this, 'cos I don't think I've disagreed with
>         you on this. So perhaps you could be more specific and we might
>         be able to see why you've brought this up.]
> 
>         waiting to get the aggregate back in order that it was in when
>         it arrived. I specifically said:
> 
> 
> 
>         numbers, and the link egress buffers them until it can send out
>         in the same order, or until a time-out or max no. of link
>         retransmissions).
> 
>         For example:
> 
>         the IETF as a whole. And we could discuss whether a revision of
>         RFC 3366 is needed.
> 
>         the moment where this opportunity is possible (so in that sense
>         it is /currently/ L4S-specific).
> 
> 
>         this thread, and we haven't even started talking to
>         actually, the consequence there would be that L4S traffic could
>         be sprayed by ECMP without per-flow hashing.
> 
> 
> 
> 
>         Michael
> 
>         numbers, and the link egress buffers them until it can send out
>         in the same order, or until a time-out or max no. of link
>         retransmissions).
> 
>         of microflows, I'm just saying the packet happens to be from
>         flow A), link-layer in-order delivery will hold back all packets
>         sent after that packet (from all flows, not just flow A) until
>         the link repairs the gap or gives up.
> 
>         machinery that just expects everything to arrive in order and to
>         a clocked schedule. But it's always as effective to deploy a
>         very small re-ordering degree - so small that it's easier for a
>         link to just deliver everything in the order it received it; not
>         because TCP doesn't put packets back in order, but just so as
>         not to trigger TCP into generating spurious retransmissions.
> 
> 
> 
>         Bob
> 
>             but apparently they also do need in-order delivery of
>             technology can simply disable in-order delivery for all “low
>             latency” applications seems not correct. The reality may be
>             a bit more complex.
> 
>             end-to-end principle on re-sequencing. Now we're removing
>             the 3 DupACK rule, we can take advantage of the e2e principle.
> 
>             delivery there are tradeoffs as well, see e.g. Section 4 in