Subject: Re: [LOOPS] [nwcrg] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

From: Carsten Bormann <[email-masked]>
Date: Sun, 5 Jan 2020 22:23:33 +0100

Hi Stu,

I think there are a number of people on the list who have a similar background with TCP acceleration, where the prominence of the SCPS-TP protocol varies.  (For me, SCPS-TP was mainly an interesting specification to look at and use as a quarry and source of ideas for our own, more optimized protocols.)  Traditional PEPs were as good as they were because:

(1) they had more intricate (and timely) knowledge of the critical path segment than the end-to-end protocol could have,
(2) techniques such as TCP splitting allowed mitigating some (but not all) of the latency induced performance issues end-to-end protocols exhibited over high-latency links.

With increasing use of encryption inside the transport layer, (2) is going away (and will probably only come back when encrypted transports such as QUIC start talking to halfway-houses in the network, without giving up the other benefits of encryption).  This reduces the benefit from the introduction of SCPS-TP segments into a sequence of split TCP segments.
(1) will stay relevant for those industries where that knowledge can be provided (satellite links, vehicular links, maybe more general wireless links).

What LOOPS is trying to do is providing a protocol that can help even in the absence of both (1) and (2).  One useful aspect here may be the aggregation of traffic, which is quite effective against tail loss events (and generally can provide a more reasonable basis for measurements), but also may make it less interesting to interact specifically with each end-to-end transport flow.  Fixing path segments outside “the critical” segment (e.g., a WiFi segment before going wide-area) can also help against mutual exacerbation of their problems.

So LOOPS is not trying to replace PEPs that benefit from (2) (and still do from (1), today).  It could still be run profitably over critical path segments where a forklift upgrade of PEP infrastructure is not feasible right away.  It is just one more tool in the toolkit that can be used to assemble a useful end-to-end path for some networking requirements, in particular where there already is a good reason to run a tunnel.  Probably later than sooner, it might also help with running such tunnels over multiple paths, but we are not proposing that the LOOPS activity should be actively looking into that just yet (even though some of us have been doing this in their day jobs for a decade or so now).

The fact that recovering packets can lead to reordering makes life harder for a LOOPS-like protocol.  As we saw in Singapore, we have quite different views on how much LOOPS could benefit from a general trend towards more reordering-robust end-to-end protocols.  LOOPS also could itself be made robust against reordering within its path segment — this needn’t involve (partial) resequencing at the LOOPS egress, where we again have different views on how useful that could be.

So yes, thanks for reminding us that SCPS-TP is good background material for optimizing path segments, but you are also right that we have to be aware of the limitations to its applicability in the LOOPS space.

I’m certainly interested in looking at more robust end-to-end protocols as well (including multipoint — reopening the discussion of NORM with a view to 2020’s networks and applications is very interesting!), and there already is a lot of attention to how QUIC could evolve to be more synergistic with efforts spent inside the network.  Ultimately, the applications will need to be part of the picture, evolving the transport abstractions (just as DTN did in a nicely radical way).  So LOOPS is just a start.  Much of what could be done beyond LOOPS is research - maybe it is also worth thinking about coordinating some of that, by posing the right questions.  E.g., for reliable multicast, we had the idea of using router assist, which from today’s perspective also would benefit from COINRG-like computing in the network.

Please do keep us informed about interesting developments in the greater design space here. 
As a proposed IETF WG, LOOPS itself will need to focus on a more narrow set of problems and opportunities, and this is the time to discuss whether we got that narrowing right in the current charter proposal [1].

Grüße, Carsten

[1]: https://github.com/loops-wg/charter/tree/post-105

---

From: "Stuart William Card" <[email-masked]>
Date: Sun, 5 Jan 2020 20:46:22 -0500

Carsten --

Thanks for your thoughtful response. I'll try to focus my inputs. :-)

> ...
> (1) ... more intricate (and timely) knowledge of the critical path
segment...
> (2) techniques such as TCP splitting allowed mitigating some...
> of the latency induced performance issues end-to-end protocols exhibited...
> With increasing use of encryption... (2) is going away...
> (1) will stay relevant... where that knowledge can be provided...
> What LOOPS is trying to do is providing a protocol that can help even in
> the absence of both (1) and (2)...
> recovering packets can lead to reordering...
> applications will need to be part of the picture...
> LOOPS itself will need to focus on a more narrow
> set of problems and opportunities, and this is the time to discuss whether
> we got that narrowing right in the current charter proposal...

Agreed. Further, not only transport but also network layer encryption
kills simple approaches to (2), leaving AFAIK only multipath or trading
increased mean latency to reduce latency variance.

Many of us have suffered from suboptimal and sometimes unanticipated
interactions between link and transport layer behaviors (retransmission in
both, flow control in the data link, congestion control in the transport).
Application layer adaptive behaviors (e.g. video streaming that switches
dynamically during a session among different source coding rates based
upon end to end performance as it varies) further complicate this. LOOPS
will be yet another interacting control loop. In the worst case, a
non-congestion loss of a link layer frame could lead to:
- data link retransmission
- LOOPS retransmission
- transport retransmission
- transport backoff
- application rate downshift
All at once! Presumably we can't prevent this altogether but want to
minimize its occurrence.

The charter looks good. I suggest broadening its scope of consideration of
cross-layer interactions: not just w/transport congestion control, but
also w/retransmission, misordering & resequencing at all layers.

-- 
Stuart W. Card, VP & Chief Scientist
Critical Technologies Inc. (CTI)
<[email-masked]>

---

From: "Zhouxingwang (Joe)" <[email-masked]>
Date: Mon, 6 Jan 2020 08:47:17 +0000

Hi,

It seems Linux TCP is much more out-of-order tolerant than we think.

1) TCP can dynamically adjust flow reordering level, not stick to 3. (https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt)
tcp_reordering - INTEGER Initial reordering level of packets in a TCP stream.
	TCP stack can then dynamically adjust flow reordering level
	between this initial value and tcp_max_reordering
	Default: 3
tcp_max_reordering - INTEGER Maximal reordering level of packets in a TCP stream. 300 is a fairly conservative value, but you might increase it
	if paths are using per packet load balancing (like bonding rr mode) Default: 300

2) RACK has been enabled by default in Linux for loss detection since kernel4.4. (https://kernelnewbies.org/Linux_4.4) 
tcp_recovery - INTEGER
	This value is a bitmap to enable various experimental loss recovery
	features.

	RACK: 0x1 enables the RACK loss detection for fast detection of lost
	      retransmissions and tail drops. It also subsumes and disables
	      RFC6675 recovery for SACK connections.
	RACK: 0x2 makes RACK's reordering window static (min_rtt/4).
	RACK: 0x4 disables RACK's DUPACK threshold heuristic

	Default: 0x1

The retransmission conflict between TCP and other layer will not be a big issue.

Thanks
Joe

-----Original Message-----
From: nwcrg [mailto:[email-masked]] On Behalf Of Stuart William Card
Sent: Monday, January 06, 2020 9:46 AM
To: Carsten Bormann <[email-masked]>
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [nwcrg] [LOOPS] IETF106: LOOPS side meeting on Tuesday 08:30, Orchard Room

Carsten --

segment...
> What LOOPS is trying to do is providing a protocol that can help even 
> in the absence of both (1) and (2)...
> LOOPS itself will need to focus on a more narrow set of problems and 
> opportunities, and this is the time to discuss whether we got that 
> narrowing right in the current charter proposal...

Agreed. Further, not only transport but also network layer encryption kills simple approaches to (2), leaving AFAIK only multipath or trading increased mean latency to reduce latency variance.

Many of us have suffered from suboptimal and sometimes unanticipated interactions between link and transport layer behaviors (retransmission in both, flow control in the data link, congestion control in the transport).
Application layer adaptive behaviors (e.g. video streaming that switches dynamically during a session among different source coding rates based upon end to end performance as it varies) further complicate this. LOOPS will be yet another interacting control loop. In the worst case, a non-congestion loss of a link layer frame could lead to:
- data link retransmission
- LOOPS retransmission
- transport retransmission
- transport backoff
- application rate downshift
All at once! Presumably we can't prevent this altogether but want to minimize its occurrence.

The charter looks good. I suggest broadening its scope of consideration of cross-layer interactions: not just w/transport congestion control, but also w/retransmission, misordering & resequencing at all layers.

_______________________________________________
nwcrg mailing list
[email-masked]
https://www.irtf.org/mailman/listinfo/nwcrg

---

From: Michael Welzl <[email-masked]>
Date: Mon, 6 Jan 2020 09:54:41 +0100

[dropping NWCRG and DTN to reduce noise a bit]

Hi,

Yes, Linux TCP has indeed been quite robust to re-ordering for many years, see here for a quite detailed study:
https://www.duo.uio.no/handle/10852/9034 <https://www.duo.uio.no/handle/10852/9034>

Cheers,
Michael

> On Jan 6, 2020, at 9:47 AM, Zhouxingwang (Joe) <[email-masked]> wrote:
> 
> Hi,
> 
> LOOPS mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/loops