Subject: [Detnet] Fwd: review DetNet OAM framework draft

From: Greg Mirsky <[email-masked]>
Date: Tue, 11 May 2021 06:10:41 -0700

Now to the list

Regards,
Greg
---------- Forwarded message ---------
From: Greg Mirsky <[email-masked]>
Date: Sun, May 9, 2021 at 12:12 PM
Subject: Re: review DetNet OAM framework draft
To: Pascal Thubert (pthubert) <[email-masked]>
Cc: [email-masked] <
[email-masked]>, [email-masked] <[email-masked]>

Hi Pascal,
I much appreciate your thorough review and the most helpful comments.
Please find my notes in-lined below under the tag GIM>>.

Regards,
Greg

On Thu, Apr 29, 2021 at 6:07 AM Pascal Thubert (pthubert) <
[email-masked]> wrote:

> Dear authors and all
>
>
>
> I do support this document but do not see that in its present shape it is
> ready for WGLC. Please find some review comments below.
>
>
>
> - Sadly RFC 8655 does not define either “latency” or “delay”. Reading this
> draft, I now see it as an oversight. I’d suggest that we define them in the
> terminology section. it is my sense that RFC 8655 uses “delay”  with the
> meaning of incremental flight duration, and uses “latency” for the total
> time end-to-end time across the network, IOW, minimal_latency + delay <=
> bounded_latency.
>
GIM>> Thank you, Pascal, for bringing this to the discussion. I agree with
you and I strongly believe that it is utterly important to set the
dictionary so that we can discuss technology and use cases based on the
common understanding of terms. I like your suggestion to provide
definitions of "latency" and "delay" in the Terminology section. Before
going into definitions, I'd like us to define the scope of these terms.
Would it be for the draft and all DetNet OAM documents or it can serve as
the clarification of the terminology used in DetNet in general since RFC
8655?
About the definitions. Do you think that the interpretation of "delay" is
close to the definition of the residence time (RFC 8169):
   Residence time is the sum of the difference between
   the time of receipt at an ingress interface and the time of
   transmission from an egress interface for each node along the network
   path from an ingress node to an egress node.
Also, in RFC 8169 we've assumed that the e2e delay, i.e., e2e latency
consists of two parts - fixed propagation delay determined by the link
speed and variable portion, i.e., residence time. In your opinion, could
this model be applied to DetNet?

>
>
>
>
> - Their work -> that work ?
>
> GIM>> Yes, you're right. Thank you.

>
>
> - the abstract and the introduction indicate that the document is a problem statement / functional requirement draft, and indeed the text reads like a requirement document. But the title says framework. Which is it?
>
>
>
> - same question about the intended status (std track). Is that really what we want here?
>
> GIM>> Thank you for pointing this out. Switched to Informational track.

>
>
> - “It is critical for the
>
>       quality of information obtained using an active method that
>
>       generated test packets are in-band with the monitored data flow.
>
> ”. this duplicates text in 3.3 and does not belong in a terminology. Remove?
>
> GIM>> I agree with you. Text in Section 3.3 is stronger but seems out of
place in that section. I think moving it up into Section 3 might be
appropriate. What do you think?

>
>
> - “                                                                                                         OAM represents the
>
>    essential elements of the network operation and necessary for OAM
>
>    resources that need to be accounted for to maintain the network
>
>    operational.
>
> ” does not parse too well
>
> GIM>> I propose the following update:
 OLD TEXT:
   OAM represents the
   essential elements of the network operation and necessary for OAM
   resources that need to be accounted for to maintain the network
   operational.
NEW TEXT:
   Because OAM is an
   essential element of the network operation,  resources, necessary for
   OAM, need to be accounted for in addition to DetNet flows.

Does the update read better?

>
>
> - “information about the state of the network being
>
>    collected
>
> ” is being? Maybe remove that text ? it seems to repeat the first sentence of the paragraph.
>
> GIM>> The purpose of the sentence is in the final part "... and sent to
the controller".Would the following update make it better:
OLD TEXT:
   In either way, information about the state of the network being
   collected and sent to the controller.
NEW TEXT:
   In either way, information is collected and sent to the controller.

>
>
> - “   Also, we can characterize methods of transporting OAM information
>
>    relative to the path of data.  For instance, OAM information may be
>
>    transported out-of-band or in-band with the data flow.
>
> ” this deserves its own subsection with some more explanation and possibly some references to relevant IETF work
>
> GIM>>  Thank you for highlighting this question. I propose adding
references to work at the IPPM WG. Here's the updated text:
NEW TEXT:
   Also, we can characterize methods of transporting OAM information
   relative to the path of data.  For instance, OAM information may be
   transported in-band or out-of-band with the data flow.  In case of
   the former, the telemetry information uses resources allocated for
   the monitored DetNet flow.  If an in-band method of transporting
   telemetry is used, the amount of generated information needs to be
   carefully analyzed, and additional resources must be reserved.
   [I-D.ietf-ippm-ioam-data] defines the in-band transport mechanism
   where telemetry information is collected in the data packet on which
   information is generated.  Two tracing methods are described - end-
   to-end, i.e., from the ingress and egress nodes, and hop-by-hop,
   i.e., like end-to-end with additional information from transit nodes.
   [I-D.ietf-ippm-ioam-direct-export] and
   [I-D.mirsky-ippm-hybrid-two-step] are examples of out-of-band
   telemetry transport.  In the former case, information is transported
   by each node traversed by the data packet of the monitored DetNet
   flow in a specially constructed packet.  In the latter, information
   is collected in a sequence of follow-up packets that traverse the
   same path as the data packet of the monitored DetNet flow.  In both
   methods, transport of the telemetry can avoid using resources
   allocated for the DetNet domain.
>
>
>
> “   Similarly, the destination does not receive packets from different
>
>    flows through its interface.
>
> ” sorry I do not get the intended meaning
>
> GIM>> We're pointing to the distinction between the Continuity Check (CC)
and Connectivity Verification (CV). The former only verifies that a path
between two systems exists, while the latter, in addition to that path,
verifies that the system does not receive packets from any flow other than
one being monitored. Associated with the CV, is a misconnection error state
and conditions of raising and clearing it. I propose removing the paragraph
with that sentence and update the first paragraph as follows:
OLD TEXT:
   In addition to the Continuity Check, DetNet solutions have to verify
   the connectivity.  This verification considers additional
   constraints, i.e., the absence of misconnection.
NEW TEXT:
   In addition to the Continuity Check, DetNet solutions have to verify
   the connectivity.  This verification considers additional
   constraints, i.e., the absence of misconnection.  The misconnection
   error state is entered after several consecutive test packets from
   other DetNet flows received.  The definition of the conditions of
   entry and exit for misconnection error state is outside the scope of
   this document.

>
>
> - section 3.3 –“It is worth noting that the test and data packets MUST follow the
>
>    same path, i.e., the connectivity verification has to be conducted
>
>    in-band without impacting the data traffic.”
>
>  there appears to be a tension between the bandwidth that is reserved for the flow and the extra bandwidth that the OAM needs .
>
> Do we need to know in advance how much OAM there will be? Must that be included in the reservation?
>
> GIM>> Yes, the impact of OAM, whether active or hybrid, must be evaluated
and accounted for when reserving resources in the DetNet domain. The update
above is intended to clarify that:
   Because OAM is an
   essential element of the network operation,  resources, necessary for
   OAM, need to be accounted for in addition to DetNet flows.
>
>
>
>
>
> - section 3.4. What do we do for PREOF?
>
> GIM>> Excellent question, thank you! I think of two updates:

   - in Section 3.4:

   Also, tracing can be used for the discovery of the Path
   Maximum Transmission Unit or location of elements of
   PREOF for the particular route in the DetNet domain.

   - in Section 6:

   DetNet OAM MUST support the discovery of PREOF along a route in
   the given DetNet domain.

>
> - “The network has isolated and identified the cause of the fault.”
>
> Again I fail to understand the intention.
>
> GIM>> Indeed. How about we change this text as the following:
OLD TEXT:
   The network has isolated and identified the cause of the fault.  For
   instance, the replication process behaves not as expected to a
   specific intermediary router.
NEW TEXT:
   An ability to localize the network defect and provide its
   characterization are necessary elements of network operation.

* Fault localization, a process of deducing the location of a network
failure
   from a set of observed failure indications, might be achieved, for
   example, by tracing the route of the DetNet flow in which the network
   failure was detected.  Another method of fault localization can
   correlate reports of failures from a set of interleaving sessions
   monitoring path continuity.
* Fault characterization is a process of identifying the root cause
  of the problem.  For instance, misconfiguration or malfunction of
  PREOF elements can be the cause of erroneous packet replication or
  extra packets being flooded in the DetNet domain.

>
>
> -“                                  One of the advantages of the use of
>
>    AMM in a DetNet domain with the IP data plane is that the marking is
>
>    applied to a data flow, thus ensuring that measured metrics are
>
>    directly applicable to the DetNet flow.
>
> ” isn’t this the case for IPPM in general? Isn’t the benefit of AMM more related to conciseness?
>
> GIM>> You are right.  Do you think that the update below makes the text
better:
OLD TEXT:
   One of the advantages of the use of   AMM in a DetNet domain with
   the IP data plane is that the marking is   applied to a data flow, thus
   ensuring that measured metrics are   directly applicable to the DetNet
   flow.
NEW TEXT:
   As with all on-path telemetry methods,
   AMM in a DetNet domain with the IP data plane is natively in-band in
   respect to the monitored DetNet flow.  Because the marking is applied
   to a data flow, measured metrics are directly applicable to the
   DetNet flow.  AMM minimizes the additional load on the DetNet domain
   by using nodal collection and computation of performance metrics in
   combination with optionally using out-of-band telemetry collection
   for further network analysis.

>
>
> - Maybe we should discuss the tension between RFC 8939 and the capability to use separate packets for OAM, since the flow is routed on the 5/6 tuple. Does that mean that only in-situ is feasible with RFC 8939 (ref draft-ietf-ippm-ioam-ipv6-options)?
>
> GIM>> I agree, that the definition of the DetNet IP flow is an important
aspect when considering the applicability of IOAM in a DetNet domain. That
may be discussed in this document, though, as I think of it, that
discussion seems also in place in draft-ietf-detnet-ip-oam
<https://datatracker.ietf.org/doc/draft-ietf-detnet-ip-oam/>. What do you
think?

>
>
> Section 4. This section needs more thoughts. For instance there are packets like PTP packets for which it is important to know the transit duration in each hop. Isn’t that true for DetNet as well?
>
> GIM>> We can discuss if and how RFC 8169
<https://datatracker.ietf.org/doc/rfc8169/> may be applicable. Perhaps in a
new discussion thread.

> Radio links have numerous metrics that can be reported (think RFC 8175), we need to know that information per hop as opposed to per path/circuit. We need to know if the hop does store-and-forward or passthrough – e.g., in the former case using smaller frames / packets / segments / fragments and/or network coding may improve the latency. Maybe we could dump the WG collective mind with a dedicated thread?
>
> GIM>> I agree, a new discussion thread will certainly help. I've been
thinking of per node metrics too. I think that to be useful, per node
metrics might as well be for the given ordered pair of ingress-egress
interfaces. I understand that blows the amount of information we generate
and collect, but by doing on-node PM calculations exported will be median,
percentiles values. Definetely needs discussion.

>
>
> - “   DetNet needs to implement a self-healing and self-optimization
>
>    approach.”
>
> Though I agree with the intention and the next sentences, I’m not sure this is the best wording. Those words are classical for routing protocols that
>
> (re)compute a path, which is disruptive or consecutive to a disruption. We do not want disruptions in DetNet. There must be enough PREOF to ensure the SLO
>
>  without repairing. The RAW problem is actually NOT to use all the redundancy at all times in order to save battery and spectrum. I’d rather say that
>
> ” in the face of events that impact the network operation (e.g., link up/down, node crash/reboot, flows starting and ending)
>
> the DetNet Controller need to perform repair and re-optimization actions in order to permanently ensure the SLO of all active flows
>
> with minimal waste of resources”.
>
> GIM>> Pascal, thank you for the suggested text. Accepted with gratitude

>
>
> - section 5.2: Is it the intention to include the repair action in OAM work?
>
> GIM>> Thank you for the question. Thinking of it, OAM role is the
detection of a performance metric degradation. Restorative actions, in my
opinion, are taken by the control or management plane. Since the detection
of performance monitoring and SLO are discussed in other sections, the
value of Section 5.2 may be not sufficient to keep it in the document. What
do you think?

>
>
> - section 6. Why only MPLS data plane? It seems that the requirement are not dependent on the underlying technology.
>
> GIM>> That is my cut'n'paste mistake.

>
>
> - I’m not sure about requirement 5, 6, 10, 11, and of the use of “excellent” in 16.
>
> GIM>> Let's remove the "excellent" :)
GIM>> The intention of listing OAM functionality in that particular manner
as requirements is to provide the check list for the gap analysis and
defining extensions. Are you concerned wth the described OAM functions or
the manner requirements are expressed?

>
>
> - I read requirement 16 as the only service assurance requirement. A very important requirement in my view, and I think we want more. We also want to ensure that a scheduled transmission happens as scheduled/when, that the guaranteed resources like buffers are available per reservation..
>
> GIM>> Thank you for the suggestion. I've tried to capture these resource
reservation parameters in the new requirement:
NEW TEXT:
    18.  DetNet OAM MUST support monitoring levels of resources allocated
        for the particular DetNet flow.  Such resources include but not
        limited to buffer utilization, scheduler transmission calendar.
>
>
>
> Many thanks to the authors for all the great work!
>
> GIM>> Many thanks for your great review, much appreciated!

>
>
> Pascal
>
>
>
>
>
> *From:* detnet <[email-masked]> *On Behalf Of *Janos Farkas
> *Sent:* jeudi 29 avril 2021 0:28
> *To:* [email-masked]
> *Subject:* [Detnet] review DetNet OAM framework draft
>
>
>
> WG,
>
>
>
> Please review the DetNet OAM framework draft:
> https://datatracker.ietf.org/doc/draft-ietf-detnet-oam-framework/.
>
>
>
> The authors consider the draft in a pretty good shape, do not see any
> major piece missing.
>
>
>
> WG review and comments would be great in order to develop the draft
> towards WG Last Call.
>
>
>
> Please send your comments to the list before the next informal OAM
> discussion on May 11 or bring it to the informal meeting:
> https://ietf.webex.com/ietf/j.php?MTID=m27bb8ee31f025670e4071bfdfcd47432.
> (More details on the OAM informal meetings:
> https://mailarchive.ietf.org/arch/msg/detnet/4ik3tI-E3uhOFNaKhpkZiWwyiZc/
> .)
>
>
>
> Regards,
>
> Janos
>
>
>

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Tue, 11 May 2021 16:46:56 +0000

Hello Greg

Please see below

On Thu, Apr 29, 2021 at 6:07 AM Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>> wrote:
Dear authors and all

I do support this document but do not see that in its present shape it is ready for WGLC. Please find some review comments below.

- Sadly RFC 8655 does not define either “latency” or “delay”. Reading this draft, I now see it as an oversight. I’d suggest that we define them in the terminology section. it is my sense that RFC 8655 uses “delay”  with the meaning of incremental flight duration, and uses “latency” for the total time end-to-end time across the network, IOW, minimal_latency + delay <= bounded_latency.
GIM>> Thank you, Pascal, for bringing this to the discussion. I agree with you and I strongly believe that it is utterly important to set the dictionary so that we can discuss technology and use cases based on the common understanding of terms. I like your suggestion to provide definitions of "latency" and "delay" in the Terminology section. Before going into definitions, I'd like us to define the scope of these terms. Would it be for the draft and all DetNet OAM documents or it can serve as the clarification of the terminology used in DetNet in general since RFC 8655?
About the definitions. Do you think that the interpretation of "delay" is close to the definition of the residence time (RFC 8169):
   Residence time is the sum of the difference between
   the time of receipt at an ingress interface and the time of
   transmission from an egress interface for each node along the network
   path from an ingress node to an egress node.
Also, in RFC 8169 we've assumed that the e2e delay, i.e., e2e latency consists of two parts - fixed propagation delay determined by the link speed and variable portion, i.e., residence time. In your opinion, could this model be applied to DetNet?

PT> Well the residence time seems like the latency of the network transit, and I see the delay as the difference between the best possible latency and the actual latency experienced by a packet. Like when I say, I’ve been delayed by 30mn in a trip that takes 5 hours when all the lights are green so the residence time in my car (aka latency) is 5H30mn. The DetNet architecture seems consistent with that interpretation, at least as far as I intended to write and can read the language.

”. this duplicates text in 3.3 and does not belong in a terminology. Remove?
GIM>> I agree with you. Text in Section 3.3 is stronger but seems out of place in that section. I think moving it up into Section 3 might be appropriate. What do you think?

PT> Cool

- “                                                                                                         OAM represents the

   operational.

PT> great

   collected

GIM>> The purpose of the sentence is in the final part "... and sent to the controller".Would the following update make it better:

PT> I see now

GIM>>  Thank you for highlighting this question. I propose adding references to work at the IPPM WG. Here's the updated text:

PT> This is where RAW needs a third method, similar to the latter but that collects the information on the reverse path to help the PSE at ingress make its decision

GIM>> We're pointing to the distinction between the Continuity Check (CC) and Connectivity Verification (CV). The former only verifies that a path between two systems exists, while the latter, in addition to that path, verifies that the system does not receive packets from any flow other than one being monitored. Associated with the CV, is a misconnection error state and conditions of raising and clearing it. I propose removing the paragraph with that sentence and update the first paragraph as follows:

PT> WFM

Do we need to know in advance how much OAM there will be? Must that be included in the reservation?
GIM>> Yes, the impact of OAM, whether active or hybrid, must be evaluated and accounted for when reserving resources in the DetNet domain. The update above is intended to clarify that:
   Because OAM is an
   essential element of the network operation,  resources, necessary for
   OAM, need to be accounted for in addition to DetNet flows.

PT> I hoped you would clarify that the DetNet controller must be aware of the amount of resources that OAM needs, and would reserve those resources together with those needed for the flow

  *   in Section 3.4:

  *   in Section 6:

PT> Cool. Do you intend to flood the OAM along the PREOF graph? Can you do local assurance, like, whether an individual node that is supposed to replicate effectively does so, without impacting the end to end flow?

* Fault localization, a process of deducing the location of a network failure

PT> Much clearer and more informative to me, thanks you!

GIM>> You are right.  Do you think that the update below makes the text better:

PT> cool

- Maybe we should discuss the tension between RFC 8939 and the capability to use separate packets for OAM, since the flow is routed on the 5/6 tuple. Does that mean that only in-situ is feasible with RFC 8939 (ref draft-ietf-ippm-ioam-ipv6-options)?
GIM>> I agree, that the definition of the DetNet IP flow is an important aspect when considering the applicability of IOAM in a DetNet domain. That may be discussed in this document, though, as I think of it, that discussion seems also in place in draft-ietf-detnet-ip-oam <https://datatracker.ietf.org/doc/draft-ietf-detnet-ip-oam/> . What do you think?

PT> I’d like to see some text explaining the problem and the requirement here. It does not hurt to copy paste a bit in the IP OAM draft if that helps the reader of either doc.

Section 4. This section needs more thoughts. For instance there are packets like PTP packets for which it is important to know the transit duration in each hop. Isn’t that true for DetNet as well?
GIM>> We can discuss if and how RFC 8169<https://datatracker.ietf.org/doc/rfc8169/> may be applicable. Perhaps in a new discussion thread.

let’s do that. I wish this doc to be high level but holistic, showing all the aspects and angles and not constrained by the drafts we already have. It is the framework for the present capabilities, but also for the future ones.

Radio links have numerous metrics that can be reported (think RFC 8175), we need to know that information per hop as opposed to per path/circuit. We need to know if the hop does store-and-forward or passthrough – e.g., in the former case using smaller frames / packets / segments / fragments and/or network coding may improve the latency. Maybe we could dump the WG collective mind with a dedicated thread?
GIM>> I agree, a new discussion thread will certainly help. I've been thinking of per node metrics too. I think that to be useful, per node metrics might as well be for the given ordered pair of ingress-egress interfaces. I understand that blows the amount of information we generate and collect, but by doing on-node PM calculations exported will be median, percentiles values. Definetely needs discussion.

PT> we’re in sync

- section 5.2: Is it the intention to include the repair action in OAM work?
GIM>> Thank you for the question. Thinking of it, OAM role is the detection of a performance metric degradation. Restorative actions, in my opinion, are taken by the control or management plane. Since the detection of performance monitoring and SLO are discussed in other sections, the value of Section 5.2 may be not sufficient to keep it in the document. What do you think?

PT> confirming what we said at the call, I think it does not belong. There’s a spectrum of things that can be done based on the OAM information, from simple dashboard and assurance to network automation (ala MPLS TP) to, at the extreme, the  RAW Path selection. In the extreme cases, there’s an SLO for the reaction, so the OAM flow that feeds the PSE becomes time-sensitive as well.

- I’m not sure about requirement 5, 6, 10, 11, and of the use of “excellent” in 16.
GIM>> Let's remove the "excellent" :)
GIM>> The intention of listing OAM functionality in that particular manner as requirements is to provide the check list for the gap analysis and defining extensions. Are you concerned wth the described OAM functions or the manner requirements are expressed?

PT> Yes, items 5, 6, 10, 11 need more discussion. Maybe we can discuss that in yet another thread?

Take care,

Pascal

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of Janos Farkas
Sent: jeudi 29 avril 2021 0:28
To: [email-masked]<mailto:[email-masked]>
Subject: [Detnet] review DetNet OAM framework draft

WG,

Please review the DetNet OAM framework draft: https://datatracker.ietf.org/doc/draft-ietf-detnet-oam-framework/.

The authors consider the draft in a pretty good shape, do not see any major piece missing.

WG review and comments would be great in order to develop the draft towards WG Last Call.

Please send your comments to the list before the next informal OAM discussion on May 11 or bring it to the informal meeting: https://ietf.webex.com/ietf/j.php?MTID=m27bb8ee31f025670e4071bfdfcd47432. (More details on the OAM informal meetings: https://mailarchive.ietf.org/arch/msg/detnet/4ik3tI-E3uhOFNaKhpkZiWwyiZc/.)

Regards,
Janos

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 12 May 2021 13:14:02 +0000

Hello Fabrice

> -----Original Message-----
> From: Fabrice Theoleyre <[email-masked]>
> Sent: mercredi 12 mai 2021 13:59
> Cc: Greg Mirsky <[email-masked]>; DetNet WG <[email-masked]>;
> DetNet Chairs <[email-masked]>
> Subject: Re: [Detnet] review DetNet OAM framework draft - latency
> 
> Dear Pascal,
> 
> I separated the latency discussion, because I guess it desserves
> additional explanations.
> 
> > - Sadly RFC 8655 does not define either “latency” or “delay”. Reading
> this draft, I now see it as an oversight. I’d suggest that we define them
> in the terminology section. it is my sense that RFC 8655 uses “delay”
> with the meaning of incremental flight duration, and uses “latency” for
> the total time end-to-end time across the network, IOW, minimal_latency +
> delay <= bounded_latency.
> > GIM>> Thank you, Pascal, for bringing this to the discussion. I agree
> with you and I strongly believe that it is utterly important to set the
> > About the definitions. Do you think that the interpretation of "delay"
> is close to the definition of the residence time (RFC 8169):
> > PT> Well the residence time seems like the latency of the network
> transit, and I see the delay as the difference between the best possible
> latency and the actual latency experienced by a packet. Like when I say,
> I’ve been delayed by 30mn in a trip that takes 5 hours when all the
> lights are green so the residence time in my car (aka latency) is 5H30mn.
> The DetNet architecture seems consistent with that interpretation, at
> least as far as I intended to write and can read the language.
> 
> FT> Actually, we reuse the latency as defined in rfc8655, end-to-end,
> from source to destination.
> 
> Thus, along a path S-A-B-C-D, the end-to-end latency is the sum of:
> - the residence time in S, A, B, and C
> - the propagation delay S->A, A->B, B->C, C->D
> 
> Thus, the residence time corresponds to the time a packet stays in a
> buffer for a specific forwarding node.
> 
> I guess your definition slightly differs (it corresponds rather to our
> end-to-end latency).
> 
> If we all agree, we will detail the end-to-end latency, as I described
> above.

I see what you're saying and we're not in full agreement yet. Not that it's pure terminology, not disagreement on how things work.

Agreed that you and RFC 8169 describe the residence time in each node. My example of residence time in the car was misleading. I'd still like to see a term for the residence time in the DetNet network, from ingress of the first DetNet router to egress of the last one, seeing the DetNet network as one big virtual box, if you like. Because that residence time in the network is the only thing that DetNet can guarantee.

What I tried to discuss was the term delay. Whether it is synonymous to "time it takes to do something" or "extra time it takes to do something" beyond the " minimal time it takes to do something".
I's rather use the latter. Meaning that in a zero-jitter network, there's never a delay, there's just a fixed latency. In your example, the propagation time over the wire is called a delay. I've seen that before but do not like it, because that gives us 4 terms to say the same thing (latency, time, duration and delay) and no term to indicate the positive jitter experienced by this packet above best possible transmission time, iow how much delay this packet incurred, how long it was delayed. I believe we need a term for that and I see that "delay" would suit that need just fine. If you scan RFC 8655, you should find that this is consistent. This is how I used it for the pieces I wrote. It occurs to me that we need our terminology. Ideally it will match the art but it does not have to if the art is unclear or overloaded.

As for latency, I see it as total time, and it must be used in cinjunction with the description of the end points, whether it is application - what we would like to guarantee but do not, NIC, DetNet-network or one-box edges - in which case it measures a residence time.

Do we have a path to converge?

Pascal

---

From: "Pascal Thubert (pthubert)" <[email-masked]>
Date: Wed, 12 May 2021 16:27:49 +0000

Hello Fabrice:

Using the term latency to refer to the time between 2 endpoint is consistent with RFC 8655 and avoids the risk of misinterpretation. Having 2 terms for the same thig can only create confusion. So I agree with your proposal not to use delay when latency is what’s intended.

There are some modern environment where the extra delay becomes a notion of importance, e.g., to take flow control actions in the DC fabric. In DetNet, it is significant to the network assurance and it may trigger some automation to ensure that the SLO is continuously met. OAM will typically measure that. So it would be cool to give it a name.

Take care,

Pascal

From: Fabrice Theoleyre <[email-masked]>
Sent: mercredi 12 mai 2021 17:56
To: Pascal Thubert (pthubert) <[email-masked]>
Cc: Greg Mirsky <[email-masked]>; DetNet WG <[email-masked]>; DetNet Chairs <[email-masked]>
Subject: Re: [Detnet] review DetNet OAM framework draft - latency

Hi Pascal,

-----Original Message-----
From: Fabrice Theoleyre <[email-masked]<mailto:[email-masked]>>
Sent: mercredi 12 mai 2021 13:59
To: Pascal Thubert (pthubert) <[email-masked]<mailto:[email-masked]>>
Cc: Greg Mirsky <[email-masked]<mailto:[email-masked]>>; DetNet WG <[email-masked]<mailto:[email-masked]>>;
DetNet Chairs <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Detnet] review DetNet OAM framework draft - latency

Dear Pascal,

  Residence time is the sum of the difference between
  the time of receipt at an ingress interface and the time of
  transmission from an egress interface for each node along the
network

Now, I catch your idea:
-> latency = time difference between entry / exit points (e.g., network (end-to-end) latency)
-> delay = extra time as a result of some cause (i.e., a packet has been *delayed* more than expected).

This distinction seems not very usual for me, but I’m fine if we find an interest for it, and if we have a consensus in the WG.

Another option would be to use uniquely the term ”latency” in the draft.

Best,
Fabrice

---

From: <[email-masked]>
Date: Thu, 13 May 2021 05:28:04 +0800 (CST)

Hi Fabrice and Pascal,

thank you for the great discussion and presenting your thoughts clearly. I've taken the liberty to top-copy Pascal notes on the terms as I think they very well reflect the issue. I've added my notes tagged GIM>> in-line. I hope my notes help the discussion.

GIM>> I think that RFC 8169 uses the residence time (RT) as two metrics - nodal and path, i.e., end-to-end. Nodal RT is measured on each node supporting the specification. Path RT is the sum of all nodal RTs collected in the Scratch Pad field, and is used by the egress node to adjust PTP control message.What I tried to discuss was the term delay. Whether it is synonymous to "time it takes to do something" or "extra time it takes to do something" beyond the " minimal time it takes to do something".I's rather use the latter. Meaning that in a zero-jitter network, there's never a delay, there's just a fixed latency. In your example, the propagation time over the wire is called a delay. I've seen that before but do not like it, because that gives us 4 terms to say the same thing (latency, time, duration and delay) and no term to indicate the positive jitter experienced by this packet above best possible transmission time, iow how much delay this packet incurred, how long it was delayed. I believe we need a term for that and I see that "delay" would suit that need just fine. If you scan RFC 8655, you should find that this is consistent. This is how I used it for the pieces I wrote. It occurs to me that we need our terminology. Ideally it will match the art but it does not have to if the art is unclear or overloaded.

GIM>> It seems that what we really need is "jitter", a.k.a., "delay variation". Perhaps we are not need interpacket delay variation (measured as delta between two consecutive delays) but rather delay variation relative to the delay determined by the law of physics, i.e., minimal possible delay. We can define the new metric and how it is calculated using a set of measured delays. Is that close to what you'd like to have?As for latency, I see it as total time, and it must be used in cinjunction with the description of the end points, whether it is application - what we would like to guarantee but do not, NIC, DetNet-network or one-box edges - in which case it measures a residence time.

GIM>> I think that we need to specify where DetNet MEPs (logically) are. I'd expect that we need to specify in the DetNet OAM Framework where the DetNet MEP is for DetNet forwarding and DetNet service sub-layer respectively.

Best regards,

Greg Mirsky

Sr. Standardization Expert
预研标准部/有线研究院/有线产品经营部 Standard Preresearch Dept./Wireline Product R&D Institute/Wireline Product Operation Division

E: [email-masked] 
www.zte.com.cn

Original Mail

Sender: PascalThubert(pthubert)
To: Fabrice Theoleyre;
CC: Greg Mirsky;DetNet WG;DetNet Chairs;
Date: 2021/05/12 06:14
Subject: Re: [Detnet] review DetNet OAM framework draft - latency

Hello Fabrice

 

 

Pascal

 

 
 
_______________________________________________
detnet mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/detnet

---

From: Balázs Varga A <[email-masked]>
Date: Thu, 13 May 2021 13:51:44 +0000

Hi,

As this is an OAM document and performance measurement uses the terminology “delay”, I would expect
to stick to the OAM related terms..

Frame/packet delay is well defined.
Frame delay (Y.1731)
Frame delay can be expressed as a one-way delay for a frame, where one-way frame delay is
defined as the time elapsed since the start of transmission of the first bit of the frame by a
source node until the reception of the last bit of the same frame by the destination node. …

Similar definition is used for IP.

In my view we should use the term “delay”. (“Latency” when used has the same meaning.)

Cheers
Bala’zs

From: detnet <[email-masked]> On Behalf Of [email-masked]
Sent: Wednesday, May 12, 2021 11:28 PM
To: pthubert=[email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] review DetNet OAM framework draft - latency

Hi Fabrice and Pascal,

GIM>> I think that RFC 8169 uses the residence time (RT) as two metrics - nodal and path, i.e., end-to-end. Nodal RT is measured on each node supporting the specification. Path RT is the sum of all nodal RTs collected in the Scratch Pad field, and is used by the egress node to adjust PTP control message.

GIM>> It seems that what we really need is "jitter", a.k.a., "delay variation". Perhaps we are not need interpacket delay variation (measured as delta between two consecutive delays) but rather delay variation relative to the delay determined by the law of physics, i.e., minimal possible delay. We can define the new metric and how it is calculated using a set of measured delays. Is that close to what you'd like to have?

Best regards,

Greg Mirsky

E: [email-masked]<mailto:[email-masked]>
www.zte.com.cn<https://protect2.fireeye.com/v1/url?k=4a9fa1c7-150498fb-4a9fe15c-86b1886cfa64-c66234771f55db2c&q=1&e=de2f305a-f1e9-4ab5-8a0b-2080bd4cd503&u=http%3A%2F%2Fwww.zte.com.cn%2F>
Original Mail
Sender: PascalThubert(pthubert)
To: Fabrice Theoleyre;
CC: Greg Mirsky;DetNet WG;DetNet Chairs;
Date: 2021/05/12 06:14
Subject: Re: [Detnet] review DetNet OAM framework draft - latency
Hello Fabrice

Pascal

_______________________________________________
detnet mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/detnet

---

From: "BRUNGARD, DEBORAH A" <[email-masked]>
Date: Fri, 14 May 2021 16:46:50 +0000

Hi,

Agree with Greg on this one – there’s lots of terms used – services/SLAs, gamers. I asked my in-house ITU-T/IETF performance expert and he noted the SDOs do not use latency for metrics, the metrics are named “delay”. While latency/delay have been used interchangeably in our RFCs, I don’t think anyone has been confused yet as the context clarifies.

So I agree with Greg, if this OAM document is based on defined metrics, let’s not rename them. If we think we need to invent any new ones, we should consult with other IETF experts. And we should consult as we progress so no negative surprises when go for publication.

Some additional hints: The relevant IETF RFCs for IP-layer: 3393 and 5481. If referring to Y.1731, then Y.1563 should also be looked at.

Thanks,
Deborah

From: detnet <[email-masked]> On Behalf Of Balázs Varga A
Sent: Thursday, May 13, 2021 9:52 AM
To: [email-masked]; pthubert=[email-masked]; [email-masked]
Cc: [email-masked]; [email-masked]; [email-masked]
Subject: Re: [Detnet] review DetNet OAM framework draft - latency

Hi,

Cheers
Bala’zs

From: detnet <[email-masked]<mailto:[email-masked]>> On Behalf Of [email-masked]<mailto:[email-masked]>
Sent: Wednesday, May 12, 2021 11:28 PM
To: pthubert=[email-masked]<mailto:pthubert=[email-masked]>; [email-masked]<mailto:[email-masked]>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [Detnet] review DetNet OAM framework draft - latency

Hi Fabrice and Pascal,

Best regards,

Greg Mirsky

www.zte.com.cn<https://urldefense.com/v3/__https:/protect2.fireeye.com/v1/url?k=4a9fa1c7-150498fb-4a9fe15c-86b1886cfa64-c66234771f55db2c&q=1&e=de2f305a-f1e9-4ab5-8a0b-2080bd4cd503&u=http*3A*2F*2Fwww.zte.com.cn*2F__;JSUlJQ!!BhdT!wRxOFss6QHaLXwUlm5jRz4xkrkG-OxxazzfEUhW7TCgDZsx3fOGjcF43Yb-WQXI$>

Pascal

_______________________________________________
detnet mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/detnet<https://urldefense.com/v3/__https:/www.ietf.org/mailman/listinfo/detnet__;!!BhdT!wRxOFss6QHaLXwUlm5jRz4xkrkG-OxxazzfEUhW7TCgDZsx3fOGjcF43W0qFpvA$>