Subject: [LOOPS] draft-li-tsvwg-loops-problem-opportunities-05 - why do traffic engineering at the overlay as well?

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 7 Jul 2020 16:28:38 +0100

I have a few quetsions about then problem draft:

Section 3.4 provides a use-case, which I would like to understand better:

“By continuously measuring the delay of path segments
    and use them as metrics for path selection, when the number of
    overlay nodes is sufficiently large, there is a high chance that a
    better path could be found".

- This seems in part to rely upon measuring and the choosing which 
“disjoint segments” to forward over. A previous discussion asked how 
this would operate when the network itself implemented methods such as 
BGP and ECMP to do some form of traffic engineerinfg, and whether this 
introduced additional level of automatic re-routing would in some way 
conflict?

- Is this decision within the network a good idea for stable oeprations? 
Is this a decision better made at the edge?

Gorry

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 7 Jul 2020 17:59:18 +0200

Hi Gorry,

>    better path could be found".

Multipath operation is out of scope for the LOOPS activity that we are presenting at the IETF108 BOF.  Multipath is a valid use case for in-network recovery, but I would expect we would want to combine the considerable knowledge gained with MPTCP and various other multipathing approaches (e.g., MPDCCP) before even defining the outline of a standardizable solution.

(Running LOOPS on a segment that is internally multipathed, or within a single segment contributing to a multipath arrangement, should be considerably simpler.)

> - This seems in part to rely upon measuring and the choosing which “disjoint segments” to forward over. A previous discussion asked how this would operate when the network itself implemented methods such as BGP and ECMP to do some form of traffic engineerinfg, and whether this introduced additional level of automatic re-routing would in some way conflict?

For LOOPS at IETF108, we are trying to avoid control loops fighting against each other.  For people who enjoy those fights, we can always ask to add it later.

> - Is this decision within the network a good idea for stable oeprations? Is this a decision better made at the edge?

I’m not sure we know the answer to that.  A vantage point within the network may have better data, both with respect to the speed of reaction (shorter RTTs for a partial path) and with respect to the aggregated traffic volume providing better measurements.  But you said “at the edge”, which is in the network…
In any case, getting this right, over a wide range of operating points, would be quite an accomplishment.

Grüße, Carsten

---

From: Gorry Fairhurst <[email-masked]>
Date: Tue, 7 Jul 2020 17:46:29 +0100

Hi Carsten,

OK, so if multipath operation is out of scope for the proposed LOOPS activity, what in scenario 3 of the problem statement remains in scope, relative to figure 2.

While this scenario  is in focus, can you explain section 3.1, this seems a discussion of TLP. Is this an “issue” fixed by RACK or have I missed something?

I'm curious why you split RTP, bulk and other transfers apart in this case - to be sure, is this just to explain different merits of the same advocated approach, or is it that you see differentiation within the network based on flow type?

Gorry

---

From: Liyizhou <[email-masked]>
Date: Wed, 8 Jul 2020 02:49:51 +0000

Hi Gorry and Carsten,

Please see inlines with [yz]

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Gorry Fairhurst
Sent: Wednesday, July 8, 2020 12:46 AM
To: Carsten Bormann <[email-masked]>
Cc: loops <[email-masked]>
Subject: Re: [LOOPS] draft-li-tsvwg-loops-problem-opportunities-05 - why do traffic engineering at the overlay as well?

Hi Carsten,

[yz] Current text of multi-pathing basically provided why multi-pathing existed in the scenario. Applying in-network recovery to segments of one of the paths is no much difference from the segments of a single path in current text. Consideration of the operations on the fork point of multi-paths would expose additional requirements but it is out of scope for now. So a separate section 3.4 seems not very necessary. Will edit it for next revision.  

[yz] Yes, it talks about negative impact of tail loss. TLP in RACK tries to fix it. In my understanding, In-network recovery (LOOPS) can help in the cases of: 1) sender has not used the time based loss detection in TCP, or not a TCP sender. 2) Segment detects the loss and then recover it faster, fire of TLP timer at the sender is reduced when RACK is in use.  

[yz] This tries to show that in-network recovery brings benefits to different types of transported data. Current doc structure seems a bit confusing. Could be interpreted as those merits only applicable to one scenario. Indeed they are common to all scenarios. Will re-structure the section alignment. Thank you.   

Gorry

On 07/07/2020 16:59, Carsten Bormann wrote:
> (Running LOOPS on a segment that is internally multipathed, or within 
> a single segment contributing to a multipath arrangement, should be 
> considerably simpler.)
>
> I’m not sure we know the answer to that.  A vantage point within the 
> network may have better data, both with respect to the speed of reaction (shorter RTTs for a partial path) and with respect to the aggregated traffic volume providing better measurements.  But you said “at the edge”, which is in the network… In any case, getting this right, over a wide range of operating points, would be quite an accomplishment.
>
> Grüße, Carsten

--
LOOPS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/loops

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 9 Jul 2020 10:46:47 +0100

> [yz] Current text of multi-pathing basically provided why multi-pathing existed in the scenario. Applying in-network recovery to segments of one of the paths is no much difference from the segments…
[gf] That would be really good to see soon, so I and others understand 
the planned scope.
> [yz] Yes, it talks about negative impact of tail loss. TLP in RACK tries to fix it. In my understanding, In-network recovery (LOOPS) can help in the cases of: 1) sender has not used the time based…
[gf] Why? I think I am missing something:  QUIC uses RACK-like recovery. 
Many TCP stacks are moving to do this. Why is this still a cricial thing 
to address within the network?
> [yz] This tries to show that in-network recovery brings benefits to different types of transported data. Current doc structure seems a bit confusing. Could be interpreted as those merits only appli…

[gf] It would be good to clarify: So I think that I understand then the 
proposed methods are neutral to the type of traffic: That is, they will 
not expose details of the flow to the network to select different 
treatments?

Gorry

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 9 Jul 2020 10:57:52 +0100

Hi,

I have questions about congetsion control will be performed. The problem 
statement says:

“LOOPS (Local Optimizations on Path Segments) is a network-assisted
    performance enhancement over path segment and it aims to provide
    local in-network recovery to achieve better data delivery by making
    packet loss recovery faster and by avoiding the senders over-reducing
    their sending rate.  “

[gf] ... which seems interesting.

Section 3.3 outlines precisely the problem that
“In long-haul links, when the loss is caused by non-persistent burst
    which is extremely short and pretty random, the sender's reaction of
    reducing sending rate is not able to respond in time to the
    instantaneous path situation or to mitigate such bursts.  On the
    contrary, reducing window size at the sender unnecessarily or too
    aggressively harms the throughput for application's long lasting
    traffic like bulk data transfer.”

[gf] ... Which sounds like it asks questions where I did not find an 
answer: How does the network segment differentiate between different 
pathologies that result in loss? I can imagine some random physical 
layer loss that isn't congestion. I can imagine some random loss that is 
a result of corruption in a shared media... which may or may not be 
link-layer congestion? How does the network segment know?

So, one reason for asking is I saw these two statements, on informing 
the end-to-end transport protocol about congestion loss events that 
distinguish losses:

"Existing tools such as ECN will be used."

“ A more recent draft,
    [I-D.ietf-tsvwg-tunnel-congestion-feedback], proposes to activate ECN
    for the tunnel regardless of whether the end-to-end protocol signals
    the use of an ECN-capable transport (ECT), which requires more
    complicated action at the tunnel egress.”

[gf] How does this help, please explain?

... The draft mentioned hasn't been changed since Jan 2017, and has not 
yet captured much experience or comment, so I am quite uneasy about 
predicting what will finally happen with thart draft. That aside, I'd 
like to understand how ECN helps in this case, since the 
congestion-control reaction seems a key part of any LOOPS method.

Best wishes,

Gorry

---

From: Michael Welzl <[email-masked]>
Date: Thu, 9 Jul 2020 12:08:45 +0200

Hi,

I’d like to answer one point below:

> [gf] Why? I think I am missing something:  QUIC uses RACK-like recovery. Many TCP stacks are moving to do this. Why is this still a cricial thing to address within the network?

In QUIC, the PTO is based on an RTO type calculation. In RACK it used to be 2*RTT, now I see an updated algorithm that makes 2*RTT a lower bound (or 1 sec in the absence of RTT samples).
Either way, this is multiple e2e RTTs - that’s potentially much, much longer than a LOOPS retransmission.

Cheers,
Michael

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 9 Jul 2020 11:32:19 +0100

On 09/07/2020 11:08, Michael Welzl wrote:
>>> While this scenario  is in focus, can you explain section 3.1, this 
>>> seems a discussion of TLP. Is this an “issue” fixed by RACK or have 
>>> I missed something?
>>>
>>> [yz] Yes, it talks about negative impact of tail loss. TLP in RACK 
>>> tries to fix it. In my understanding, In-network recovery (LOOPS) 
>>> can help in the cases of: 1) sender has not used the time based loss 
>>> detection in TCP, or not a TCP sender. 2) Segment detects the loss 
>>> and then recover it faster, fire of TLP timer at the sender is 
>>> reduced when RACK is in use.
>> [gf] Why? I think I am missing something:  QUIC uses RACK-like 
>> recovery. Many TCP stacks are moving to do this. Why is this still a 
>> cricial thing to address within the network?
>
> In QUIC, the PTO is based on an RTO type calculation. In RACK it used 
> to be 2*RTT, now I see an updated algorithm that makes 2*RTT a lower 
> bound (or 1 sec in the absence of RTT samples).
Depending on how TLP works, I see this could be a gain. (In QUIC, PTO = 
smoothed_rtt + max(4*rttvar, kGranularity) + max_ack_delay.)

- For a bulk app. Isn't the gain only when the last segment(s) are lost, 
and such a transfer is already many RTTs, and probably includes some 
other losses with additional delay. So it saves one PTO-time in a large 
number of RTTs?

- For short transactions (or bursts of activity within a flow), there 
can be a much bigger potential proportional benefit.

It would be interesting to see how much of a gain this really is?

Gorry

---

From: Liyizhou <[email-masked]>
Date: Thu, 9 Jul 2020 11:45:18 +0000

Hi Gorry,

Based on the discussions in the mailing list, we should have removed allusions to being able to detect non-congestion losses from the draft. This github issue was closed about same time as the draft uploading. Hence the draft did not absorb all the conclusions. Sorry about that.

Current scope focuses on drop-to-mark approach without trying to detect non-congestion loss.

Will update the draft by this week to reflect the most recent consensus (together with those mentioned in your previous email). 

Rgds,
Yizhou

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Gorry Fairhurst
Sent: Thursday, July 9, 2020 5:58 PM
To: loops <[email-masked]>
Subject: Re: [LOOPS] draft-li-tsvwg-loops-problem-opportunities-05 - How do endpoints understand congestion?

Hi,

I have questions about congetsion control will be performed. The problem statement says:

“ A more recent draft,

Best wishes,

Gorry

---

From: Carsten Bormann <[email-masked]>
Date: Thu, 9 Jul 2020 14:26:03 +0200

On 2020-07-09, at 12:32, Gorry Fairhurst <[email-masked]> wrote:
> 
> - For short transactions (or bursts of activity within a flow), there can be a much bigger potential proportional benefit. 

Indeed.  I think the recent discussions have arrived at the conclusion that the minimum viable protocol (MVP) we are targeting now, will be beneficial for ants and for transactional flows (flows that stop and restart), less so for throughput-bound flows.  Essentially we get the benefits 2.1 to 2.4 listed in RFC 8087.

Grüße, Carsten

---

From: Gorry Fairhurst <[email-masked]>
Date: Thu, 9 Jul 2020 13:35:51 +0100

Thanks - I look forward to reading a clean set of drafts.

Gorrt

> On 9 Jul 2020, at 12:45, Liyizhou <[email-masked]> wrote:
> 
> Hi Gorry,
>

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 17 Jul 2020 08:57:09 +0100

See below:

> Indeed.  I think the recent discussions have arrived at the conclusion that the minimum viable protocol (MVP) we are targeting now, will be beneficial for ants and for transactional flows (flows th…

OK, so I think this means the plan would be to target chatty flows, but 
only for endpoints that have  enabled ECN?

Am I correct in assuming that this is classic ECN, using ECT(0)?

As I think I understand, a path increases the workload in response to 
loss by adding FEC that recovers the loss but hides the increase in rate 
from the endpoints, instead setting ECN-CE marks to notify of 
congestion. If the endpoint reacts, then the load stabilizes. Do I need 
to worry whether the endpoint's reaction to ECN-CE marks is something 
that the network would also need to monitor, to confirm that CE-marking 
is actually causing a congestion reaction?

Gorry

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Jul 2020 10:26:49 +0200

> 
> OK, so I think this means the plan would be to target chatty flows, but only for endpoints that have  enabled ECN?

Yes.  (I don’t know if you group short flows under chatty.)

> Am I correct in assuming that this is classic ECN, using ECT(0)?

If there is another form of ECN, with defined rules how to convert losses into marks, these could be added.  But I don’t think there is, at this point in time.

> As I think I understand, a path increases the workload in response to loss by adding FEC that recovers the loss but hides the increase in rate from the endpoints, instead setting ECN-CE marks to notify of congestion. If the endpoint reacts, then the load stabilizes. Do I need to worry whether the endpoint's reaction to ECN-CE marks is something that the network would also need to monitor, to confirm that CE-marking is actually causing a congestion reaction?

Do we do that without LOOPS?

(We could add flow-specific circuit breakers, but that creates a lot of processing.)

Grüße, Carsten

---

From: Gorry Fairhurst <[email-masked]>
Date: Fri, 17 Jul 2020 09:39:53 +0100

See below.

> Yes.  (I don’t know if you group short flows under chatty.)
OK. I see short flows too.
> If there is another form of ECN, with defined rules how to convert losses into marks, these could be added.  But I don’t think there is, at this point in time.

:-).

I expect that targeting ECT(0) is at least the best first step. TSVWG is 
publishing L4S, and that might (soon) be on the roadmap for the way ECN 
is used.

Still this relies on endpoints to enable ECT(0) processing that might or 
might not be a hurdle to deploying something else. Do people see LOOPS 
brings benefit  if the endpoints do not support ECN?

> Do we do that without LOOPS?
I think I am concerned enough to worry about this, unless I can see a 
lot more experience from ietf-tsvwg-tunnel-congestion-feedback.
> (We could add flow-specific circuit breakers, but that creates a lot of processing.)

Indeed and a circuit-breaker wouldn't be easy to realise (or perhaps 
appropiate) for short/chatty flows, and that makes me wonder really how 
much need there is to experimentally investigate what is needed.

Gorry

---

From: Carsten Bormann <[email-masked]>
Date: Fri, 17 Jul 2020 10:56:58 +0200

> 
> Still this relies on endpoints to enable ECT(0) processing that might or might not be a hurdle to deploying something else. Do people see LOOPS brings benefit  if the endpoints do not support ECN?

Not the MVP (minimum viable protocol) we are targeting now — doing a one-to-one drop-to-drop signaling doesn’t give you a lot of recovery :-)

Since the issue really is what traffic can be directed to the LOOPS tunnel, it is somewhat trivial to provide benefit once we have better segment-to-path congestion signaling.

> I think I am concerned enough to worry about this, unless I can see a lot more experience from ietf-tsvwg-tunnel-congestion-feedback.

How can we have any useful ECT traffic at all if we need to suspect that doesn’t react to CE marks?  I’m not sure I understand what the existence of the LOOPS tunnel adds to that (assuming there is some sane overhead limit, such as 20 % — I’m sure 10000 % wouldn’t be so great).

> Indeed and a circuit-breaker wouldn't be easy to realise (or perhaps appropiate) for short/chatty flows, and that makes me wonder really how much need there is to experimentally investigate what is needed.

A single unruly flow won’t influence a high-mux aggregate.
If there are many unruly flows, the aggregate circuit breaker will trip.
Is there anything interesting in between?

Grüße, Carsten

---

From: Spencer Dawkins at IETF <[email-masked]>
Date: Fri, 17 Jul 2020 07:03:36 -0500

Spencer, with no hat (not even a bandana, I'm indoors in Texas),

On Fri, Jul 17, 2020 at 3:57 AM Carsten Bormann <[email-masked]> wrote:

> Not the MVP (minimum viable protocol) we are targeting now — doing a
> one-to-one drop-to-drop signaling doesn’t give you a lot of recovery :-)
>
> Since the issue really is what traffic can be directed to the LOOPS
> tunnel, it is somewhat trivial to provide benefit once we have better
> segment-to-path congestion signaling.
>
> from the endpoints, instead setting ECN-CE marks to notify of congestion.
> If the endpoint reacts, then the load stabilizes. Do I need to worry
> whether the endpoint's reaction to ECN-CE marks is something that the
> network would also need to monitor, to confirm that CE-marking is actually
> causing a congestion reaction?
> How can we have any useful ECT traffic at all if we need to suspect that
> doesn’t react to CE marks?  I’m not sure I understand what the existence of
> the LOOPS tunnel adds to that (assuming there is some sane overhead limit,
> such as 20 % — I’m sure 10000 % wouldn’t be so great).
>

Let me first say how much I appreciate these exchanges happening this week
on the mailing list, and not during Session 1 of Friday during IETF 108
week :D.

I'm thinking about "first, do no harm".

   - If a path has a large number of short flows that respond to CE, that's
   awesome.
   - If a path has a large number of short flows that don't respond to CE
   markings, but do respond to drops, the senders should slow down as much as
   you can slow down an exchange with less than a dozen packers, when they see
   drops. Is LOOPS making that situation better, or worse?
   - If a path has a large number of short flows that don't respond to CE
   markings or drops, I THINK an aggregate circuit breaker would cover that
   situation. Is LOOPS making that situation better, or worse?

What am I missing?

Best,

Spencer, who was available to help co-chair LOOPS because he's not as smart
as you guys :p

---

From: Carsten Bormann <[email-masked]>
Date: Tue, 21 Jul 2020 11:03:22 +0200

Hi Spencer,

Interesting questions.

> On 2020-07-17, at 14:03, Spencer Dawkins at IETF <[email-masked]> wrote:
> 
> 	• If a path has a large number of short flows that respond to CE, that's awesome.

Indeed.  Let’s talk about short + transactional (“chatty”) flows, though.

> 	• If a path has a large number of short flows that don't respond to CE markings, but do respond to drops, the senders should slow down as much as you can slow down an exchange with less than a dozen packers, when they see drops. Is LOOPS making that situation better, or worse?

Until countermeasures such as circuit breakers hit, worse.  Since LOOPS is converting drops to marks, it is converting observed congestion events into ignored congestion events for these flows.

So where do you get those flows that don’t respond to CE markings?  And how did you avoid them breaking other CE-marking places in the Internet already?  I think the argument really boils down to “we can’t have LOOPS adding CE-marking places because CE-marking is bad”, and I’m not sure we have consensus on the latter.

I know about the case where a middle box somwhere on the path outside the LOOPS segment removes ECE markings from TCP headers; this is about the worst case I can think about (but it is already bad without LOOPS).

For short flows, it may be worth to consider what “respond to drops” means.  For a short flow, the drop can be (1) in the establishment phase (where LOOPS cannot help TCP flows), essentially acting as a flow-delayer or flow-preventer (in the presence of racing).  LOOPS is not changing that picture.  Or it can be (2) in the IW10 phase, and there LOOPS is likely to react to a drop by adding a packet or two to the ones that already are flowing.  Doesn’t help with any congestion in the LOOPS-controlled path segment, but doesn’t really hurt that much.

> 	• If a path has a large number of short flows that don't respond to CE markings or drops, I THINK an aggregate circuit breaker would cover that situation. Is LOOPS making that situation better, or worse? 

LOOPS is introducing a circuit breaker where there might not have been one before, so it is making the situation better.

The question of course is how fast does it trip (i.e., what is the damage until it does), and what exactly does it circuit-break.  I would have designed it to just stop expending packets on recovery (i.e., stop LOOPS), but not to stop traffic at all.  That would mean there would be no benefit, but also no lasting damage.

I think in the end we will need to limit any damage from misbehaving endpoints (whether doing that on their own or due to some bad interventions from middle boxes) by limiting the amount of repair that is being done by LOOPS.

Grüße, Carsten