Subject: [LOOPS] Measuring forward latency

From: Carsten Bormann <[email-masked]>
Date: Tue, 18 Jun 2019 00:42:15 +0200

The proposal in

https://tools.ietf.org/html/draft-welzl-loops-gen-info

measures forward latency by having the egress node send back some acknowledgements that carry a timestamp of arrival (expressed in egress node local clock time).  No timing information is conveyed in the forward direction.

This assumes that the local clock of the egress node is useful for relaying timing information to the ingress node.  In many cases in today’s networks, nodes are synchronized using NTP (or maybe even PTP), so the clocks should not be diverging wildly.  Also, the assumption is that today’s nodes can keep their clock drift in check well enough that the latency measurements are not useless.

However, there is some systematic error being introduced if the nodes are not perfectly in sync.  This error should be an essentially constant addend (positive or negative) to the measured latency (maybe even leading to mostly negative “latency” measurements, which may seem illogical but may not invalidate the congestion detection math).

If the latency measurement is mostly used to detect upticks in latency that might be indicative of congestion, all this should not be a big problem.

Alternative approaches might:

— need to send more information (essentially running a per-tunnel time synchronization protocol, which is wasteful)
— complicate the processing.

Note that the ingress node has under control how many ACKs with a time stamp it receives, by controlling the rate of setting the “ACK desired” flag; an ingress node that does not care about latency might never set that flag (and rely on cumulative “bitmap” style acknowledgements, called “block 2” in section 4.3).

So are we happy with the latency measurement approach described in the proposal?

Note that an ingress node can independently estimate RTT by noting when packets have been sent (in ingress node clock time) and noting the arrival time of an acknowledgement for that packet (if the PSN is different per transmission there is no ambiguity).  The overhead for storing the transmission point in time nearly vanishes behind the storage of the packet itself (which is needed at least in retransmission mode).  This independent RTT measurement is necessary because the path for the reverse information may be rather different from the forward path and also because delayed acknowledgement schemes add to the RTT.

Does the proposed design look right?

Grüße, Carsten

---

From: Giuseppe Fioccola <[email-masked]>
Date: Tue, 18 Jun 2019 12:39:39 +0000

Dear Carsten, All,
I think it would be very useful to do a classification of the possible measurement methods for LOOPS.
And, there are Pros and Cons to be considered for each methodologies.
In addition to the methodologies already on the table (TCP-like mechanisms, one-way forward latency mechanism and so on...) I would suggest to consider also the so called spin bit approach as described in the following drafts: https://tools.ietf.org/html/draft-trammell-ippm-spin-00 and https://tools.ietf.org/html/draft-cfb-ippm-spinbit-new-measurements-00.
The main difference is that the spin bit is a resource conserving mechanism since there is no timestamp processing and no timestamp on the wire, the RTT is given by the duration of the spin period. But it can only do measurements once for every RTT.
What do you think?

Best Regards,

Giuseppe

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Tuesday, June 18, 2019 12:42 AM
To: [email-masked]
Subject: [LOOPS] Measuring forward latency

The proposal in

— need to send more information (essentially running a per-tunnel time synchronization protocol, which is wasteful) — complicate the processing.

Grüße, Carsten

--
LOOPS mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/loops

---

From: Michael Welzl <[email-masked]>
Date: Tue, 18 Jun 2019 14:52:41 +0200

Hi,

I thought the spin bit is meant for a router somewhere along the path to understand the end-to-end RTT of a flow.
We, on the other hand, need a LOOPS ingress node to understand the RTT (or, better, OWD) to the LOOPS egress node. So this is (LOOPS-)end-to-(LOOPS)-end.

Cheers,
Michael

> On 18 Jun 2019, at 14:39, Giuseppe Fioccola <[email-masked]> wrote:
> 
> Dear Carsten, All,

---

From: "Roni Even (A)" <[email-masked]>
Date: Tue, 18 Jun 2019 13:14:59 +0000

Hi,
The spin measurement depends on the number of measurement points. Using one point you can calculate the end to end RTT but by using two measurement points you can calculate the time between them
Roni

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Michael Welzl
Sent: Tuesday, June 18, 2019 3:53 PM
To: Giuseppe Fioccola
Cc: Cociglio Mauro; Brian Trammell (IETF); Carsten Bormann; [email-masked]
Subject: Re: [LOOPS] Measuring forward latency

Hi,

Cheers,
Michael

---

From: Giuseppe Fioccola <[email-masked]>
Date: Tue, 18 Jun 2019 13:32:29 +0000

Yes, it started as the solution for an on-path observer to measure the end-to-end RTT of a flow. But you can use it at every point with the same benefits in terms of simplicity and efficiency, even end to end between a LOOPS ingress node and a LOOPS egress node.

Regards,

Giuseppe

-----Original Message-----
From: Michael Welzl [mailto:[email-masked]] 
Sent: Tuesday, June 18, 2019 2:53 PM
To: Giuseppe Fioccola <[email-masked]>
Cc: Carsten Bormann <[email-masked]>; [email-masked]; Cociglio Mauro <[email-masked]>; Brian Trammell (IETF) <[email-masked]>
Subject: Re: [LOOPS] Measuring forward latency

Hi,

Cheers,
Michael

---

From: Liyizhou <[email-masked]>
Date: Wed, 19 Jun 2019 09:40:51 +0000

Hi Carsten,

There are different metrics to measure for LOOPS enabled segment, i.e. between ingress and egress.
- local RTT or one-way latency
- loss rate over certain time period
- throughput (by calculation)

The fundamental purpose to measure different metrics and monitor their change is to help determine what kind of loss signal (CE or not CE marking, recover or not recover the loss, etc) should be relayed to end host sender when a packet loss is discovered.  
So measurement should be frequent enough. 

TCP's delayed ACK mechanism looks also a good fit here if we set the delayed ACK number to be some larger value.

Another thing that is worth noting is which node will be consuming the measurement information. 
In local retransmission mode, ingress is the one consuming it as it determines whether to CE marking or stop retransmitting. In FEC mode, egress is more likely consuming the information as it recovers the packet and determine the loss signal independently.

Rgds,
Yizhou

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Tuesday, June 18, 2019 6:42 AM
To: [email-masked]
Subject: [LOOPS] Measuring forward latency

The proposal in

Grüße, Carsten

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 26 Jun 2019 08:12:27 +0200

> We, on the other hand, need a LOOPS ingress node to understand the RTT (or, better, OWD) to the LOOPS egress node. So this is (LOOPS-)end-to-(LOOPS)-end.

We need a number of measurements, and an implementation could also make use of additional measurements.  If a measurement for the end-to-end RTT is available for a flow, it can be used to properly dose congestion event signaling for non-ECT flows in the way described at the end of Section 2.4 of gen-info [1].  It may also be useful to derive an upper limit for the timing of retransmissions (and the latency variance these will introduce).

But I would argue that getting that end-to-end RTT measurement is not a function of LOOPS — it is a function that implementations could provide to improve the LOOPS function of congestion signaling.  (Besides, that measurement is per-flow, not a per-tunnel wholesale measurement.)

As Yizhou listed [2], we need a number of different per-tunnel measurements, and we need them to be available for different functions of the LOOPS protocol.  Specifically:

— per-tunnel RTT, to time retransmissions
— per-tunnel one-way latency, to detect congestion and sort congestion from non-congestion losses
— per-tunnel loss rate (and other statistical properties of the loss), for parameterizing FEC

These differ in how they can be achieved, where the data is produced and where it is needed, and that should guide the design.  E.g., moving the timestamps into the forward information makes latency variation immediately available at the egress, which is also the place where the congestion signaling to non-ECT transports will need to happen.

Grüße, Carsten

[1]: https://tools.ietf.org/html/draft-welzl-loops-gen-info-00#section-2.4
[2]: https://mailarchive.ietf.org/arch/msg/loops/P3LdmvVvMeeYUzv-6r9oB91pkvQ

---

From: Michael Welzl <[email-masked]>
Date: Wed, 26 Jun 2019 08:39:01 +0200

Hi,

Thanks, all of this is very reasonable IMO.
More below:

> On Jun 26, 2019, at 8:12 AM, Carsten Bormann <[email-masked]> wrote:
> 

I see - I didn’t think of this being useful additional information, but I agree it can be.

> These differ in how they can be achieved, where the data is produced and where it is needed, and that should guide the design.  E.g., moving the timestamps into the forward information makes latenc…

I agree with all that. Just a side question: what makes you say “non-ECT transports”?  How is this different for ECT ones?

Cheers,
Michael

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 26 Jun 2019 10:50:38 +0200

On Jun 26, 2019, at 08:39, Michael Welzl <[email-masked]> wrote:
> 
>> E.g., moving the timestamps into the forward information makes latency variation immediately available at the egress, which is also the place where the congestion signaling to non-ECT transports will need to happen.
> 

Indeed, congestion signaling has to happen for either.
It is just massively simpler for ECT transports, as we can simply hand through loss/CE events in the CE marking.
For non-ECT transports, we don’t want to hand through each loss (there would be no point in recovering them if we then have to drop all of them).
(This requires per-flow state, which definitely should remain optional in a LOOPS implementation.)

Grüße, Carsten

---

From: Michael Welzl <[email-masked]>
Date: Wed, 26 Jun 2019 10:52:09 +0200

> On Jun 26, 2019, at 10:50 AM, Carsten Bormann <[email-masked]> wrote:
> 

Ahh, sure

> (This requires per-flow state, which definitely should remain optional in a LOOPS implementation.)

Yes. Got it, thanks!

Cheers,
Michael

---

From: "Zhouxingwang (Joe)" <[email-masked]>
Date: Thu, 27 Jun 2019 02:39:24 +0000

Hi,

Generally speaking, maintaining per-flow state in a network node is a heavy solution.
But for this scenario,  the LOOPS node does not need to maintain the state for every flow all the time, the ingress node can start to maintain the state of the flows only for the retransmission packets, then the number of the flows should be much smaller than the total number of flows, based on this, ingress node may make sure every loss-affected flow has one loss event for every RTT.

Thanks
Joe

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Wednesday, June 26, 2019 4:51 PM
To: Michael Welzl <[email-masked]>
Cc: Giuseppe Fioccola <[email-masked]>; Cociglio Mauro <[email-masked]>; [email-masked]; Brian Trammell (IETF) <[email-masked]>
Subject: Re: [LOOPS] Measuring forward latency

Grüße, Carsten

---

From: "Zhouxingwang (Joe)" <[email-masked]>
Date: Fri, 28 Jun 2019 13:02:21 +0000

Hi, 

Who will consume the measurement information? 

For ECN-capable case: it is both OK for ingress or egress node to do the CE or not CE marking in retransmission or FEC mode.
For non ECN- capable case, it is congested and if:
1) the egress node perceives, egress node can stop retransmitting the retransmission packets for retransmission-mode, or stop recovering the packets for FEC-mode;
2) the ingress node perceives, ingress can stop performing the retransmission for retransmission-mode, or stop adding redundant packets for FEC-mode;

It seems option 2) is better because it saves bandwidth compare with option 1).

Thanks
Joe

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Liyizhou
Sent: Wednesday, June 19, 2019 5:41 PM
To: Carsten Bormann <[email-masked]>; [email-masked]
Subject: Re: [LOOPS] Measuring forward latency

Hi Carsten,

Rgds,
Yizhou

-----Original Message-----
From: LOOPS [mailto:[email-masked]] On Behalf Of Carsten Bormann
Sent: Tuesday, June 18, 2019 6:42 AM
To: [email-masked]
Subject: [LOOPS] Measuring forward latency

The proposal in

Grüße, Carsten