Subject: [sfc] Tsvart telechat review of draft-ietf-sfc-oam-framework-13

From: Frank Brockners via Datatracker <[email-masked]>
Date: Mon, 04 May 2020 12:05:01 -0700

Reviewer: Frank Brockners
Review result: Ready with Nits

This document has been reviewed as part of the transport area review team's
ongoing effort to review key IETF documents. These comments were written
primarily for the transport area directors, but are copied to the document's
authors and WG to allow them to address any issues raised and also to the IETF
discussion list for information.

When done at the time of IETF Last Call, the authors should consider this
review as part of the last-call comments they receive. Please always CC
[email-masked] if you reply to or forward this review.

This document provides a reference framework for OAM for SFC.

Comments:

Section 3.1.1 SF availability: The text makes explicit reference to multiple
instances of a SF. Consequently, it should be defined how availability of a SF
is computed/determined in case multiple instances are deployed. This further
leads to the question, whether availability is always a "binary" state
(available / not-available), or could a SF be e.g. 99% available? Section 3.1.2
SF performance: What is the impact of a "multiple instance SF deployment" on SF
performance measurement? The section only talks about loss and delay as
performance criteria. It would be good to state that other performance criteria
(e.g. specific to the SF, throughput, etc.) exist. Section 3.2.1 SFC
availability: The current definition is very focused on connectivity
verification, i.e. it tries to answer the question: "Does my SFC transport
packets?". IMHO we should also ask the question "Does my SFC process the
packets correctly?" - because if packets are not processed per the SFC
definition, we might not call the SFC available. While 3.2.2 states that "any
SFC-aware network device should have the ability to make performance
measurements" a similar statement isn't found in 3.2.1. IMHO the ability for
availability checks is probably a prerequisite for performance measurement.
Section 3.2.2 SFC performance measurement: The section only mentions the need
for performance measurement. It misses the definition of what SFC performance
measurement is. Section 3.3. Classifier component: The section mentions the
need for the ability to perform performance measurement of the classifier
component. What is performance measurement of the classifier? What does
performance measurement of the classifier component comprise? Section 3.4. /
3.5. Availability/PM of the underlay and overlay network: It would be good to
add a sentence that states that the mechanisms for availability/PM which are
offered by the technologies used by the overlay/underlay are used, rather than
new methods specifically for SFC would be defined. Section 4. SFC OAM
Functions: It would be good, if examples in section 4 could also include more
"recent" methods such as OWAMP/TWAMP (RFC4656, RFC 5357). Section 4.4.
Performance Measurement: Focus is entirely on the PM of the connectivity,
rather than on the SF. How about covering PM for the SF as well? Section 5.1
OAM Tool Gap Analysis:
 - Not sure what "NVo3 OAM" refers to. Could that be explained below the table
 and in section 1.2.1? - E-OAM needs to be detailed. Is seems that CFM
 (802.1ag) and not 802.3ah is refered to here. - "Trace" in the "Trace" column
 need to be extended on. Is this traceroute? Paris-Traceroute? IOAM-Loopback? -
 IPPM needs to be detailed, because IPPM is not a tool as such but an IETF WG.
 Does this refer to OWAMP/TWAMP/etc. as defined by IPPM?
Section 6.4.3 IOAM:
- The section states that IOAM "may be used to perform various SFC OAM
functions as well". It would be good to expand on this statement: E.g. IOAM
Trace-Option Type could be leveraged for SFC tracing. IOAM Direct-Export Option
Type could be leveraged. - How would we deal with the IOAM Active Flag
(draft-ietf-ippm-ioam-flags-01) when used with SFC OAM? - The text states
"In-Situ OAM could be used with O bit set": Why would IOAM be used with the
overflow bit set for SFC OAM? For details on IOAM's O-bit, see section 4.4.1 in
https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-09. Section 6.4.4 SFC
Traceroute: - This section refers to an expired draft (even calling out the
fact that the draft has exipred), but also mentions that functionality is
available and implemented in OpenDaylight. Consider removing the references to
the expired draft and rather add references to OpenDaylight documents. - IOAM
Loopback (see draft-ietf-ippm-ioam-flags-01) could apply SFC Traceroute as well.

Detailed set of nits that I encountered while reading through the document ([x]
references line number x) – hope that they are helpful in further improving the
doc:

[global] s/an SF/a SF/ -- and similarly SFC/SFF
[176] "OAM Controller" not defined
[202] Why just Virtual Machines and no containers? Suggest to make things
generic and talk about virtual and physical entities.
      This comment applies throughout the document.
[216] Ethernet OAM: Add reference. Do you refer to physical layer Ethernet OAM
(802.3ah) or CFM (802.1ag)? [243] s/uses the overlay network/uses the overlay
network layer/ [246] Could we add a few examples of "various overlay network
technologies"? For the underlay network layer several examples are listed.
[248] What does "mostly transparent" mean? [254] What does "tight coupling"
between the link layer and the physical technology mean? [255] Suggest to avoid
terms like "popular" - popularity can change, standards stay [256] Acronyms
"POS" and "DWDM" are not defined [274] Link start/end-points don't seem to
always align with the underlay network in the diagram [287] s/may comprise
of/may consist of/ [288] s/but not shown/but is not shown/ [307]
s/devices/device/ [308] What is a "controller"? [314] s/includes/include/ [319]
Add hSFC to list of acronyms in section 1.2.1 [320] Add IBN to list of acronyms
in section 1.2.1 [325] s/includes/include/ [359] The function/term "controller"
requires definition. [383] s/?./?/ [398] s/get the got/got/ [461]
s/devices/device/ [469] Does it have to be equal cost multipath at the service
layer, or could unequal cost multipath also be an option for load-balancing?
[521] Not sure whether the overlay network establishes the service plane. Isn't
it that the overlay network establishes connectivity for the SFC-related
functions in the service plane? [531] s/components/component/ [545] remove
"underlay" [595] s/devices/device/ [600] s/action/an action/ [601] Expand on
"TTL or other means" (TTL also needs to be added to acronyms in 1.2.1). Is this
specific to NSH? Or specific to IPv4? [630] Mention that for "approximation of
packet loss for a given SFC can be derived" to be applicable, SFC OAM packets
would need to be forwarded the same as live user traffic. [636] Is uppercase
"MUST" applicable to an informational document? Especially given that
RFC2119/RFC8174 is explicitly referenced by the draft. [666] Add MPLS, TRILL to
acronyms in 1.2.1 [678] s/exhaustive/exhaustive./ [705] What are "underlying
layers"? [720] Is uppercase "SHOULD" applicable to an informational document?
Especially given that RFC2119/RFC8174 is explicitly referenced by the draft.
[722] Is uppercase "MAY" applicable to an informational document? Especially
given that RFC2119/RFC8174 is explicitly referenced by the draft. [754]
s/packet/packets/ [755] s/to next node/to the next node/ [771] How does this
requirement align with the earlier paragraph, e.g. in case a node sends an ICMP
reply? It would probably make sense to scope the statement to e.g. NSH. [806]
s/function/functions/ [809] s/from relevant node/from the relevant node/ [810]
s/generate ICMP/generate an ICMP/ [812] s/from last/from the last/ [830]
s/perform continuity/perform the continuity/ [834] s/with relevant/with the
relevant [835] s/perform partial SFC availability./perform a partial SFC
availability check./ [851] For "In-Situ OAM data fields" add a normative
reference to draft-ietf-ippm-ioam-data [905] Add "CLI" to section 1.2.1
acronyms [920] Add a reference for NETCONF ->RFC6241

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Sat, 16 May 2020 14:16:15 +0000

Hi Frank,

Thank you for the review. Please see inline for the response..

    

    Comments:
    
    Section 3.1.1 SF availability: The text makes explicit reference to multiple
    instances of a SF. Consequently, it should be defined how availability of a SF
    is computed/determined in case multiple instances are deployed. 

<Nagendra> This is already clarified in the section as below:

"For cases where
   multiple instances of an SF are used to realize a given SF for the
   purpose of load sharing, SF availability can be performed by checking
   the availability of any one of those instances, or the availability
   check may be targeted at a specific instance."

This further
    leads to the question, whether availability is always a "binary" state
    (available / not-available), or could a SF be e.g. 99% available? 

<Nagendra>The availability is measured as binary state. I am not sure what is 99% available. If it means getting 99 responses for 100 probes sent, I think it falls under packet loss category which in turn is performance measurement.

Section 3.1.2
    performance measurement? 

<Nagendra>I think we covered this in SF availability but not here. Does the below updated text look better?

OLD:
On the one hand, the performance of any specific SF can be quantified
   by measuring the loss and delay metrics of the traffic from SFF to
   the respective SF, while on the other hand, the performance can be
   measured by leveraging the loss and delay metrics from the respective
   SFs.  The latter requires SF involvement to perform the measurement
   while the former does not.

NEW:
On the one hand, the performance of any specific SF can be quantified
   by measuring the loss and delay metrics of the traffic from SFF to
   the respective SF, while on the other hand, the performance can be
   measured by leveraging the loss and delay metrics from the respective
   SFs.  The latter requires SF involvement to perform the measurement
   while the former does not. For cases where
   multiple instances of an SF are used to realize a given SF for the
   purpose of load sharing, SF performance can be quantified by measuring 
   the metrics for any one instance of SF or by measuring the metrics for 
   a specific instance.

The section only talks about loss and delay as
    performance criteria. It would be good to state that other performance criteria
    (e.g. specific to the SF, throughput, etc.) exist. 

<Nagendra> We can add the below to Section 3.1.2:

NEW:
"The metrics measured to quantify the performance of the SF component is 
not just limited to loss and delay. Other metrics such as throughout also exist 
and the choice of metrics for performance measurement is outside the scope 
of this document."

Section 3.2.1 SFC
    definition, we might not call the SFC available. 

<Nagendra> I think this is already handled by SF availability. The end-to-end SFC availability is verified by steering the OAM packet over the ordered set of SFs within the SFC. This is more like daisy chaining the availability of SFs within the SFC to determine end-to-end SFC availability. If the derived solution verifies the SF availability not just based on the uptime but based on the service treatment, it also answers the question "Does my SFC process the packets correctly". Let us know if there is any further clarity required. 

While 3.2.2 states that "any
    SFC-aware network device should have the ability to make performance
    measurements" a similar statement isn't found in 3.2.1. IMHO the ability for
    availability checks is probably a prerequisite for performance measurement.

<Nagendra> The ability to perform end-to-end or partial SFC availability verification is already mentioned in section 3.2.1 as below:

" In order to perform service connectivity verification of an SFC/SFP,
   the OAM functions could be initiated from any SFC-aware network
   devices of an SFC-enabled domain for end-to-end paths, or partial
   paths terminating on a specific SF, within the SFC/SFP"

Please let us know if you have any suggestion to improve if there is a lack of clarity.

    measurement is. 

<Nagendra>

Section 3.3. Classifier component: The section mentions the
    need for the ability to perform performance measurement of the classifier
    component. What is performance measurement of the classifier? What does
    performance measurement of the classifier component comprise? 

<Nagendra>We can add the below text:

OLD:
Any SFC-aware network device should have the ability to perform
   performance measurement of the classifier component for each SFC.

NEW:
Any SFC-aware network device should have the ability to perform
   performance measurement of the classifier component for each SFC.
    The performance can be quantified by measuring the performance metrics of the 
     traffic from the classifier for each SFC/SFP.

Section 3.4. /
    3.5. Availability/PM of the underlay and overlay network: It would be good to
    add a sentence that states that the mechanisms for availability/PM which are
    offered by the technologies used by the overlay/underlay are used, rather than
    new methods specifically for SFC would be defined. 

<Nagendra>Yes, that makes sense. Please check the below text:

OLD:
Any SFC-aware network device may have the ability to perform
   availability check or performance measurement of the overlay network.

NEW:
Any SFC-aware network device may have the ability to perform
   availability check or performance measurement of the overlay network. Any
   existing OAM tools and techniques can be leveraged for this purpose.

Section 4. SFC OAM
    Functions: It would be good, if examples in section 4 could also include more
    "recent" methods such as OWAMP/TWAMP (RFC4656, RFC 5357). 

<Nagendra> 

OLD:
Delay within an SFC could be measured based on the time it takes for
   a packet to traverse the SFC from the ingress SFC node to the egress
   SFF.  As SFCs are unidirectional in nature, measurement of one-way
   delay [RFC7679] is important.  In order to measure one-way delay,
   time synchronization MUST be supported by means such as NTP, PTP,
   GPS, etc.

NEW:
Delay within an SFC could be measured based on the time it takes for
   a packet to traverse the SFC from the ingress SFC node to the egress
   SFF.  Measurement protocols such as One-way Active Measurement 
    Protocol (OWAMP) [RFC4656], Two-way Active Measurement Protocol
   (TWAMP) [RFC5357] can be used to measure the characteristics. As 
   SFCs are unidirectional in nature, measurement of one-way
   delay [RFC7679] is important.  In order to measure one-way delay,
   time synchronization MUST be supported by means such as NTP, Precision Time Protocol (PTP),
   GPS, etc.

Section 4.4.
    Performance Measurement: Focus is entirely on the PM of the connectivity,
    rather than on the SF. How about covering PM for the SF as well? 

<Nagendra> I am not sure I understand what is missing. Do you have any suggestion for the text improvement?.

Section 5.1
    OAM Tool Gap Analysis:
     and in section 1.2.1? 

<Nagendra> Combining this with other below queries as they appears to be related.

- E-OAM needs to be detailed. Is seems that CFM
     (802.1ag) and not 802.3ah is refered to here. 

<Nagendra> Per my understanding, 802.ah is 1-hop while 802.3ag can be more than 1 hop and both uses Ethernet frames. So I think both are applicable here. My response regarding E-OAM details in this section is combined below.

- "Trace" in the "Trace" column
     need to be extended on. Is this traceroute? Paris-Traceroute? IOAM-Loopback? 

<Nagendra> Combining the above queries. 

OLD:
There are various OAM tool sets available to perform OAM functions
   within various layers.  These OAM functions may be used to validate
   some of the underlay and overlay networks.  Tools like ping and trace
   are in existence to perform connectivity check and tracing of
   intermediate hops in a network.  These tools support different
   network types like IP, MPLS, TRILL, etc.  There is also an effort to
   extend the tool set to provide connectivity and continuity checks
   within overlay networks.  BFD is another tool which helps in
   detecting data forwarding failures.  Table 3 below is not exhaustive

NEW:
There are various OAM tool sets available to perform OAM functions
   within various layers.  These OAM functions may be used to validate
   some of the underlay and overlay networks.  Tools like ping and trace
   are used to perform connectivity check and tracing of
   intermediate hops in a network.  These tools are already available for
   different types of networks such as IP, MPLS, TRILL, etc. 
 
E-OAM offers OAM mechanisms such as an Ethernet continuity check for 
Ethernet links. There is an effort around NVO3 OAM
to provide connectivity and continuity checks for networks that use NVO3.  BFD is used
for the detection of data plane forwarding failures.
 
The IPPM framework [RFC 2330] offers tools such as OWAMP [RFC4656] and TWAMP
[RFC5357] (collectively referred as IPPM in this section) to measure various performance
metrics. MPLS Packet Loss Measurement (LM) and Packet Delay Measurement (DM) (collectively
referred as MPLS_PM in this section) [RFC6374] offers the ability to measure
performance metrics in MPLS network.

Table 3 below is not exhaustive.

Section 6.4.3 IOAM:
    (draft-ietf-ippm-ioam-flags-01) when used with SFC OAM? 

<Nagendra> The intention of the section is to highlight the applicability of different OAM toolsets for OAM functions at service layer. I am not sure if we really should try explaining all the possible options within each tool. But I agree that it is worth clarifying the availability of IOAM options for tracing. think we can clarify that different IOAM Option-Types are available for OAM functions such as SFC tracing. Can you check if the below looks ok?

OLD:
[I-D.ietf-sfc-ioam-nsh] defines how In-Situ OAM data fields are
   transported using NSH header.  [I-D.ietf-sfc-proof-of-transit]
   defines a mechanism to perform proof of transit to securely verify if
   a packet traversed the relevant SFP or SFC.  While the mechanism is
   defined inband (i.e., it will be included in data packets), it may be
   used to perform various SFC OAM functions as well.

NEW:
[I-D.ietf-sfc-ioam-nsh] defines how In-Situ OAM data fields are
   transported using NSH header.  [I-D.ietf-sfc-proof-of-transit]
   defines a mechanism to perform proof of transit to securely verify if
   a packet traversed the relevant SFP or SFC.  While the mechanism is
   defined inband (i.e., it will be included in data packets), IOAM Option-Types
  such as IOAM Trace Option-Types can also be used to perform other SFC OAM function 
  such as SFC tracing.

- The text states
    "In-Situ OAM could be used with O bit set": Why would IOAM be used with the
    overflow bit set for SFC OAM? For details on IOAM's O-bit, see section 4.4.1 in
    https://tools.ietf.org/html/draft-ietf-ippm-ioam-data-09. 

<Nagendra> The O bit referred here is not the O bit in IOAM but the one in NSH/Overlay header. To avoid any confusion, this can be updated as below:

OLD:
In-Situ OAM could be used with O bit set to perform SF availability
   and SFC availability or performance measurement.

NEW:
In-Situ OAM could be used with O bit in the overlay header set, to perform SF availability
   and SFC availability or performance measurement.

Section 6.4.4 SFC
    
<Nagendra>Ok. Let me check if I can find some reference for ODL. 

<Nagendra> Yes of course (.

<Nagendra>Other RFCs uses "an SF/SFF". So the draft is updated accordingly. If your suggestion is to substitute "a SF" to "an SF",  it is done (.

<Nagendra>We can change it as below:

OLD:
OAM controllers are assumed to be within the same administrative
   domain as the target SFC enabled domain.

NEW:
OAM controllers are SFC-aware network devices that are capable of 
generating OAM packets. They are assumed to be within the same 
administrative domain as the target SFC enabled domain.

<Nagendra> We changed this as virtual entities.

          This comment applies throughout the document.
    [216] Ethernet OAM: Add reference. Do you refer to physical layer Ethernet OAM
    (802.3ah) or CFM (802.1ag)? 

<Nagendra> The response was provided in the above comment section.

[243] s/uses the overlay network/uses the overlay
    network layer/ 

<Nagendra> Done.

[246] Could we add a few examples of "various overlay network
    technologies"? For the underlay network layer several examples are listed.

<Nagendra> Ok.

    [248] What does "mostly transparent" mean? 

<Nagendra> The data plane elements connecting the overlay layer nodes may not always process the overlay header. 

[254] What does "tight coupling"
    between the link layer and the physical technology mean? 

<Nagendra>I am not sure I understand the nit here. Do you see any difficulty in parsing the sentence?.

[255] Suggest to avoid
    terms like "popular" - popularity can change, standards stay 

<Nagendra> Ok. This is changed as "Ethernet is one such choice..."

[256] Acronyms
    "POS" and "DWDM" are not defined 

<Nagendra> Added.

[274] Link start/end-points don't seem to
    always align with the underlay network in the diagram 

<Nagendra> Fixed it.

[287] s/may comprise
    of/may consist of/ 

<Nagendra>We fixed it as "may comprise"..

[288] s/but not shown/but is not shown/ 

<Nagendra> We fixed this as "intermediate nodes not shown...:

[307]
    s/devices/device/ 

<Nagendra> Done.

[308] What is a "controller"? 

<Nagendra> We discussed this in the above comment section.

[314] s/includes/include/ 

<Nagendra>Done.

[319]
    Add hSFC to list of acronyms in section 1.2.1 

<Nagendra> This is expanded in the respective section. We added it in the acronym section as well.

[320] Add IBN to list of acronyms
    in section 1.2.1 

<Nagendra> Ok, Done.

[325] s/includes/include/ 

<Nagendra> Done.
[359] The function/term "controller"
    requires definition. 

<Nagendra> Done, as mentioned in the above comment section.

[383] s/?./?/ 

[398] s/get the got/got/

<Nagendra> Done.

 [461]
    s/devices/device/

<Nagendra> Done.

 [469] Does it have to be equal cost multipath at the service
    layer, or could unequal cost multipath also be an option for load-balancing?
   
<Nagendra>I didn’t see any discussion specific to ECMP/UCMP in the architecture RFC.

 [521] Not sure whether the overlay network establishes the service plane. Isn't
    it that the overlay network establishes connectivity for the SFC-related
    functions in the service plane? 

<Nagendra> The service layer is established over the overlay network layer. I am not sure if it is right to say overlay network provides connectivity for service layer (.

[531] s/components/component/ [545] remove
    "underlay" 

<Nagendra> Done.

[595] s/devices/device/ 

<Nagendra> Done.

[600] s/action/an action/ 

<Nagendra> Done.

[601] Expand on
    "TTL or other means" (TTL also needs to be added to acronyms in 1.2.1). Is this
    specific to NSH? Or specific to IPv4?

<Nagendra> TTL is listed as well-known abbrev in https://www.rfc-editor.org/materials/abbrev.expansion.txt and so we left it as it is. TTL in this document refers to NSH TTL field.

 [630] Mention that for "approximation of
    packet loss for a given SFC can be derived" to be applicable, SFC OAM packets
    would need to be forwarded the same as live user traffic.

<Nagendra> As it is intending to derive the approximate loss value, I am not sure if we need this additional consideration that the OAM packet would need to follow the live user traffic. Let me know if you think otherwise.

 [636] Is uppercase
    "MUST" applicable to an informational document? Especially given that
    RFC2119/RFC8174 is explicitly referenced by the draft. 

<Nagendra> Based on various reviewer comments, we removed the use of any normative statement.

[666] Add MPLS, TRILL to
    acronyms in 1.2.1 

<Nagendra> Ok. Done.

[678] s/exhaustive/exhaustive./ 

<Nagendra> Done.

[720] Is uppercase "SHOULD" applicable to an informational document?
    Especially given that RFC2119/RFC8174 is explicitly referenced by the draft.
    
<Nagendra> Based on various reviewer comments, we removed the use of any normative statement.

[722] Is uppercase "MAY" applicable to an informational document? Especially
    given that RFC2119/RFC8174 is explicitly referenced by the draft. 

<Nagendra> Based on various reviewer comments, we removed the use of any normative statement.

[754]
    s/packet/packets/ 

[755] s/to next node/to the next node/

 [771] How does this
    requirement align with the earlier paragraph, e.g. in case a node sends an ICMP
    reply? It would probably make sense to scope the statement to e.g. NSH. 

<Nagendra> As mentioned in the statement, the node that initiates the OAM packet must set the marker and so this statement is applicable for the initiating node.

[806]
    s/function/functions/ 

<Nagendra> Done

[809] s/from relevant node/from the relevant node/ 

<Nagendra> Done

[810]
    s/generate ICMP/generate an ICMP/ 

<Nagendra> Done

[812] s/from last/from the last/ 

<Nagendra> Done

[830]
    s/perform continuity/perform the continuity/

<Nagendra> Done

 [834] s/with relevant/with the
    relevant 

<Nagendra> Done

[835] s/perform partial SFC availability./perform a partial SFC
    availability check./ 

<Nagendra> Done

[851] For "In-Situ OAM data fields" add a normative
    reference to draft-ietf-ippm-ioam-data 

[905] Add "CLI" to section 1.2.1
    acronyms 

<Nagendra> Done

[920] Add a reference for NETCONF ->RFC6241
    
<Nagendra> Done

Once again, thanks a lot for the great comments.

Regards,
Nagendra

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Wed, 20 May 2020 17:10:20 +0000

Hi Nagendra,

Thanks for the detailed reply. Please see inline (..FB).

> -----Original Message-----
> From: Nagendra Kumar Nainar (naikumar) <[email-masked]>
> Sent: Samstag, 16. Mai 2020 16:16
> To: Frank Brockners (fbrockne) <[email-masked]>; [email-masked]
> Cc: [email-masked]; [email-masked]; [email-masked]
> Subject: Re: Tsvart telechat review of draft-ietf-sfc-oam-framework-13
> 
> Hi Frank,
> 
> <Nagendra>The availability is measured as binary state. I am not sure what is
> 99% available. If it means getting 99 responses for 100 probes sent, I think it
> falls under packet loss category which in turn is performance measurement.

...FB: Thanks. Though I'm still not entirely following. If availability is binary and I put the statements above together, what would be the availability of the following setup: There is an SF that is made up of 100 instances. 99 of these instances are powered down entirely. And the 1 instance that is "up" is alternating between servicing requests for 10min followed by not servicing requests for 10min. Would the SF be considered "available"?

> "The metrics measured to quantify the performance of the SF component is not
> just limited to loss and delay. Other metrics such as throughout also exist and
> the choice of metrics for performance measurement is outside the scope of this
> document."
> 
> Section 3.2.1 SFC
> <Nagendra> I think this is already handled by SF availability. The end-to-end SFC
> availability is verified by steering the OAM packet over the ordered set of SFs
> within the SFC. This is more like daisy chaining the availability of SFs within the
> SFC to determine end-to-end SFC availability. If the derived solution verifies the
> SF availability not just based on the uptime but based on the service treatment, it
> also answers the question "Does my SFC process the packets correctly". Let us
> know if there is any further clarity required.
> 
> <Nagendra> The ability to perform end-to-end or partial SFC availability
> verification is already mentioned in section 3.2.1 as below:
> 

...FB: Thanks for the suggested updates, which would definitively improve the text. One problem about SFC performance remains though IMHO.
All the text so far is focused on the connectivity within a SFC - not the service itself. I.e. If you'd consider a "laundry service" - we focus a lot on how long it takes to get the clothes shipped to and from the washing machine, but we don't focus on how well the washing machine washes the clothes. 
IMHO we should either expand on the performance of the SFC and SF wrt/ the service (especially given that you define a service layer in section 2) - or clearly state that the framework would just focus on connectivity between SFs. 

> suggestion for the text improvement?.

...FB: See above. This would be about adding a capability to assess how well the washing machine washes my laundry.

> <Nagendra> Per my understanding, 802.ah is 1-hop while 802.3ag can be more
> than 1 hop and both uses Ethernet frames. So I think both are applicable here.
> My response regarding E-OAM details in this section is combined below.

...FB: Maybe I missed it - but I don't see text that refers to CFM or EFM OAM. Where is this covered? IMHO we would need references to IEEE standards to avoid confusion.

> Ethernet links. There is an effort around NVO3 OAM to provide connectivity and
> continuity checks for networks that use NVO3.  BFD is used for the detection of
> data plane forwarding failures.

...FB: Check whether NVO3 WG will indeed deliver a solution and "NVO3 OAM" indeed existis. If in doubt, it might be better to avoid forward looking references. Per my note above, it would be good to explicitly refer to IEEE standards as opposed to introducing a new term like "E-OAM". 

> 
> The IPPM framework [RFC 2330] offers tools such as OWAMP [RFC4656] and
> TWAMP [RFC5357] (collectively referred as IPPM in this section) to measure
> various performance metrics. MPLS Packet Loss Measurement (LM) and Packet
> Delay Measurement (DM) (collectively referred as MPLS_PM in this section)
> [RFC6374] offers the ability to measure performance metrics in MPLS network.
> 
> <Nagendra> The intention of the section is to highlight the applicability of
> different OAM toolsets for OAM functions at service layer. I am not sure if we
> really should try explaining all the possible options within each tool. But I agree
> that it is worth clarifying the availability of IOAM options for tracing. think we
> can clarify that different IOAM Option-Types are available for OAM functions
> such as SFC tracing. Can you check if the below looks ok?
> 
> OLD:
> <Nagendra> The O bit referred here is not the O bit in IOAM but the one in
> NSH/Overlay header. To avoid any confusion, this can be updated as below:
> 
> OLD:

... FB: Ah, ok. Given that this section is about IOAM and not NSH, I'd rather explicitly refer to NSH here. E.g. If SFC is realized using NSH, then the O-bit in the NSH header could be used to indicated OAM traffic. You could refer to https://tools.ietf.org/html/draft-ietf-sfc-ioam-nsh-03#section-4.2 explicitly. 

> <Nagendra>Other RFCs uses "an SF/SFF". So the draft is updated accordingly. If
> your suggestion is to substitute "a SF" to "an SF",  it is done (.
> 
> OAM controllers are SFC-aware network devices that are capable of generating
> OAM packets. They are assumed to be within the same administrative domain as
> the target SFC enabled domain.
> 

...FB: How about we explain this in the document?

> <Nagendra>I am not sure I understand the nit here. Do you see any difficulty in
> parsing the sentence?

...FB: Not sure what "tight coupling" means here. Could you clarify what is "tight coupling" vs. "not tight coupling"?

> <Nagendra>I didn’t see any discussion specific to ECMP/UCMP in the

...FB: Hmm. I did not see that RFC7665 is only about equal cost multipath. 
> <Nagendra> The service layer is established over the overlay network layer. I am
> not sure if it is right to say overlay network provides connectivity for service
> layer (.

...FB: Overlay network is one component of the service layer, isn't it. So it is required but not sufficient.

> <Nagendra> TTL is listed as well-known abbrev in https://www.rfc-
> editor.org/materials/abbrev.expansion.txt and so we left it as it is. TTL in this
> document refers to NSH TTL field.

...FB: Let's ensure we refer to NSH TTL in this case. Given that SFC can be done with other means than NSH, implicit reference to NSH might be a problem. 
> <Nagendra> As it is intending to derive the approximate loss value, I am not sure
> if we need this additional consideration that the OAM packet would need to
> follow the live user traffic. Let me know if you think otherwise.

...FB: IMHO we should - given that it is one potential complication. 

> <Nagendra> As mentioned in the statement, the node that initiates the OAM
> packet must set the marker and so this statement is applicable for the initiating
> node.
> 
> [806]
>     s/function/functions/
> 
> <Nagendra> Done
> 

Thanks again for considering the comments in great detail. Much appreciated.

Cheers, Frank

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 20 May 2020 13:20:54 -0400

Frank, regarding your comment about SF performance, I thought the 
document was pretty clear that we consider that out of scope (c.f. the 
discussions with the various ADs.)

If you can see a place to add text, please propose text.

Thank you,
Joel

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 20 May 2020 17:33:27 +0000

Frank,

> IMHO we should either expand on the performance of the SFC and SF wrt/ the service (especially given that you define a service layer in section 2) - or clearly state that the framework would just f…
And
> ...FB: See above. This would be about adding a capability to assess how well the washing machine washes my laundry.

Please follow the discussion at these threads:
https://mailarchive.ietf.org/arch/msg/sfc/QlZKr42XbDImA3HxlEXhxDBv6gU/ <https://mailarchive.ietf.org/arch/msg/sfc/QlZKr42XbDImA3HxlEXhxDBv6gU/>
https://mailarchive.ietf.org/arch/msg/sfc/JT9MPKMqBdNx3S8m_yhdIUGaTEQ/ <https://mailarchive.ietf.org/arch/msg/sfc/JT9MPKMqBdNx3S8m_yhdIUGaTEQ/>

Some of the resulting text is at:
https://mailarchive.ietf.org/arch/msg/sfc/1ZsLw6m1OeJRJfHW2TygxOyKPJk/ <https://mailarchive.ietf.org/arch/msg/sfc/1ZsLw6m1OeJRJfHW2TygxOyKPJk/>

Best,

Carlos.

> 2020/05/20 午後1:10、Frank Brockners (fbrockne) <[email-masked]>のメール:
> 
> Hi Nagendra,
> 
> ... FB: Ah, ok. Given that this section is about IOAM and not NSH, I'd rather explicitly refer to NSH here. E.g. If SFC is realized using NSH, then the O-bit in the NSH header could be used to indicated OAM traffic. You could refer to https://tools.ietf.org/html/draft-ietf-sfc-ioam-nsh-03#section-4.2 <https://tools.ietf.org/html/draft-ietf-sfc-ioam-nsh-03#section-4.2> explicitly.
> 
>> 
>> Section 6.4.4 SFC

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Wed, 20 May 2020 17:41:11 +0000

Thanks Joel. Per what I mentioned below, let's be clear that SF performance is out of scope for the doc.
And I think this was Alvaro's point as well.

Cheers, Frank

> From: Joel M. Halpern <[email-masked]>
> Sent: Mittwoch, 20. Mai 2020 19:21
> To: Frank Brockners (fbrockne) <[email-masked]>; Nagendra Kumar Nainar
> (naikumar) <[email-masked]>; [email-masked]
> Frank, regarding your comment about SF performance, I thought the document
> was pretty clear that we consider that out of scope (c.f. the discussions with the
> various ADs.)
> 
> >> Cc: [email-masked]; [email-masked];
> >> [email-masked]
> >> <Nagendra>The availability is measured as binary state. I am not sure
> >> what is 99% available. If it means getting 99 responses for 100
> >> probes sent, I think it falls under packet loss category which in turn is
> performance measurement.
> >
> > ...FB: Thanks. Though I'm still not entirely following. If availability is binary and
> I put the statements above together, what would be the availability of the
> following setup: There is an SF that is made up of 100 instances. 99 of these
> instances are powered down entirely. And the 1 instance that is "up" is
> alternating between servicing requests for 10min followed by not servicing
> requests for 10min. Would the SF be considered "available"?
> >
> >>
> >> Section 3.1.2
> >> "The metrics measured to quantify the performance of the SF component
> >> is not just limited to loss and delay. Other metrics such as
> >> throughout also exist and the choice of metrics for performance
> >> measurement is outside the scope of this document."
> >>
> >> Section 3.2.1 SFC
> >> <Nagendra> I think this is already handled by SF availability. The
> >> end-to-end SFC availability is verified by steering the OAM packet
> >> over the ordered set of SFs within the SFC. This is more like daisy
> >> chaining the availability of SFs within the SFC to determine
> >> end-to-end SFC availability. If the derived solution verifies the SF
> >> availability not just based on the uptime but based on the service
> >> treatment, it also answers the question "Does my SFC process the packets
> correctly". Let us know if there is any further clarity required.
> >>
> >> <Nagendra> The ability to perform end-to-end or partial SFC
> >> availability verification is already mentioned in section 3.2.1 as below:
> >>
> > ...FB: Thanks for the suggested updates, which would definitively improve the
> text. One problem about SFC performance remains though IMHO.
> > All the text so far is focused on the connectivity within a SFC - not the service
> itself. I.e. If you'd consider a "laundry service" - we focus a lot on how long it
> takes to get the clothes shipped to and from the washing machine, but we don't
> focus on how well the washing machine washes the clothes.
> > IMHO we should either expand on the performance of the SFC and SF wrt/ the
> service (especially given that you define a service layer in section 2) - or clearly
> state that the framework would just focus on connectivity between SFs.
> >
> >
> >>
> Any
> >> <Nagendra> I am not sure I understand what is missing. Do you have
> >> any suggestion for the text improvement?.
> >
> >> <Nagendra> Per my understanding, 802.ah is 1-hop while 802.3ag can be
> >> more than 1 hop and both uses Ethernet frames. So I think both are
> applicable here.
> > ...FB: Maybe I missed it - but I don't see text that refers to CFM or EFM OAM.
> Where is this covered? IMHO we would need references to IEEE standards to
> avoid confusion.
> >
> >>
> >>       need to be extended on. Is this traceroute? Paris-Traceroute?
> >> IOAM- Loopback?
> >>
> >> Ethernet links. There is an effort around NVO3 OAM to provide
> >> connectivity and continuity checks for networks that use NVO3.  BFD
> >> is used for the detection of data plane forwarding failures.
> >
> > ...FB: Check whether NVO3 WG will indeed deliver a solution and "NVO3 OAM"
> indeed existis. If in doubt, it might be better to avoid forward looking
> references. Per my note above, it would be good to explicitly refer to IEEE
> standards as opposed to introducing a new term like "E-OAM".
> >
> >>
> >> The IPPM framework [RFC 2330] offers tools such as OWAMP [RFC4656]
> >> and TWAMP [RFC5357] (collectively referred as IPPM in this section)
> >> to measure various performance metrics. MPLS Packet Loss Measurement
> >> (LM) and Packet Delay Measurement (DM) (collectively referred as
> >> MPLS_PM in this section) [RFC6374] offers the ability to measure
> >> <Nagendra> The intention of the section is to highlight the
> >> applicability of different OAM toolsets for OAM functions at service
> >> layer. I am not sure if we really should try explaining all the
> >> possible options within each tool. But I agree that it is worth
> >> clarifying the availability of IOAM options for tracing. think we can
> >> clarify that different IOAM Option-Types are available for OAM functions
> >> <Nagendra> The O bit referred here is not the O bit in IOAM but the
> >> one in NSH/Overlay header. To avoid any confusion, this can be updated as
> below:
> >>
> >> OLD:
> > ... FB: Ah, ok. Given that this section is about IOAM and not NSH, I'd rather
> explicitly refer to NSH here. E.g. If SFC is realized using NSH, then the O-bit in the
> NSH header could be used to indicated OAM traffic. You could refer to
> https://tools.ietf.org/html/draft-ietf-sfc-ioam-nsh-03#section-4.2 explicitly.
> >
> >>
> >> Section 6.4.4 SFC
> >> <Nagendra>Other RFCs uses "an SF/SFF". So the draft is updated
> >> accordingly. If your suggestion is to substitute "a SF" to "an SF",  it is done (.
> >>
> >> <Nagendra> The data plane elements connecting the overlay layer nodes
> >> may not always process the overlay header.
> >
> >> <Nagendra>I am not sure I understand the nit here. Do you see any
> >> difficulty in parsing the sentence?
> >
> > ...FB: Not sure what "tight coupling" means here. Could you clarify what is
> "tight coupling" vs. "not tight coupling"?
> >
> >>
> >> [255] Suggest to avoid
> >> <Nagendra> The service layer is established over the overlay network
> >> layer. I am not sure if it is right to say overlay network provides
> >> connectivity for service layer (.
> >
> >> editor.org/materials/abbrev.expansion.txt and so we left it as it is.
> >> TTL in this document refers to NSH TTL field.
> >
> > ...FB: Let's ensure we refer to NSH TTL in this case. Given that SFC can be done
> with other means than NSH, implicit reference to NSH might be a problem.
> >>
> >> <Nagendra> As it is intending to derive the approximate loss value, I
> >> am not sure if we need this additional consideration that the OAM
> >> packet would need to follow the live user traffic. Let me know if you think
> otherwise.
> >
> >> <Nagendra> As mentioned in the statement, the node that initiates the
> >> OAM packet must set the marker and so this statement is applicable
> >> for the initiating node.
> >>
> >> [806]
> >>      s/function/functions/
> >>
> >> <Nagendra> Done
> >>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Wed, 20 May 2020 14:16:53 -0400

So the question now is whether the text Murray suggested suffices for 
you?  (We are still waiting to hear from Alvaro.)

Yours,
Joel

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Wed, 20 May 2020 20:58:50 +0000

Hi Frank,

Thank you for the response. Please see inline for <Nagendra2>

On 5/20/20, 1:10 PM, "Frank Brockners (fbrockne)" <[email-masked]> wrote:

    Hi Nagendra,

    
    
<Nagendra2> As mentioned in the draft, SF availability can be performed by checking the availability of any one instance (if the intention is to verify if there is at least one instance of the SF is up) or to one or more specific instances of the SF. If you intend to verify all 100, the availability should be performed for each such specific instance.

  
<Nagendra2> I am skipping this as this point is being discussed in a different thread.

    

<Nagendra2>Ok. Instead of defining the specific clauses, we can refer Y1731. Hope that is fine with you.

    
<Nagendra2> Ok. Noted.

    
<Nagendra2> Ok. Below is the updated text capturing the same:

NEW:
In-Situ OAM could be used with O bit in the overlay header set, to perform SF availability
 and SFC availability or performance measurement. For example, if SFC is realized using NSH, 
the O-bit in the NSH header could be set to indicate the OAM traffic as defined in 
Section 4.2 [I-D.ietf-sfc-ioam-nsh].

    
<Nagendra2>Below is the updated text:

OLD:
This layer is mostly
      transparent to the SFC data plane elements .

NEW:
This layer is mostly
      transparent to the SFC data plane elements as not all the 
data plane elements process the overlay header.

    
<Nagendra2> "tightly vs loosely" coupled normally denotes the dependency (. Based on another reviewer's comment, we agreed to modify the relevant text as shown below. Does it look better?

NEW:
o  The link layer, which is tightly coupled with the physical layer
      technology used.  Ethernet is a popular choice for this layer, but
      other alternatives are deployed (e.g.  POS, DWDM).  In a virtual environment,
      virtualized I/O technologies such as SR-IOV or similar are applicable for this layer.
      The same or distinct link layer technologies may be used in each leg shown in
      Figure 1.

    
    
<Nagendra2> Ok. We can modify as below:

OLD:
  The overlay network establishes the service plane between the SFC
   components and are mostly transparent to the SFC data plane elements.

NEW:
  The overlay network provides connectivity for service plane between the SFC
   components and are mostly transparent to the SFC data plane elements.

<Nagendra2>Ok.

    
<Nagendra2> Ok. We can update as below:

OLD:

SFC OAM should provide the ability to measure packet loss for an SFC.
   On-demand measurement can be used to estimate packet loss using
   statistical methods.  Measuring the loss of OAM packets, an
   approximation of packet loss for a given SFC can be derived.

NEW:
SFC OAM should provide the ability to measure packet loss for an SFC.
   On-demand measurement can be used to estimate packet loss using
   statistical methods.  Measuring the loss of OAM packets that share the 
fate with data traffic, an approximation of packet loss for a given SFC can be derived.

    Cheers, Frank

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 22 May 2020 07:56:55 +0000

Hi Nagendra,

Thanks - please see inline (...FB2)

> Sent: Mittwoch, 20. Mai 2020 22:59
> <Nagendra2> As mentioned in the draft, SF availability can be performed by
> checking the availability of any one instance (if the intention is to verify if there
> is at least one instance of the SF is up) or to one or more specific instances of
> the SF. If you intend to verify all 100, the availability should be performed for
> each such specific instance.
> 
>     >
>     > Section 3.1.2
>     >     SF performance: What is the impact of a "multiple instance SF
> deployment" on
>     > SF
>     >     performance measurement?
>     >
> not
>     >     Section 3.2.2 SFC performance measurement: The section only mentions
> the
>     > need
>     >     The performance can be quantified by measuring the performance metrics
> of
>     > the
> more

...FB2: We can add Y.1731 as well. But let's make sure we also refer to the related IEEE standards.

>     ...FB: Check whether NVO3 WG will indeed deliver a solution and "NVO3
> OAM" indeed existis. If in doubt, it might be better to avoid forward looking
> and
>     >   such as IOAM Trace Option-Types can also be used to perform other SFC
> OAM
>     > function
>     >   such as SFC tracing.
>     >
>     > - The text states
>     > In-Situ OAM could be used with O bit in the overlay header set, to perform
> SF
>     > availability
> In-Situ OAM could be used with O bit in the overlay header set, to perform SF
> availability  and SFC availability or performance measurement. For example, if
> SFC is realized using NSH, the O-bit in the NSH header could be set to indicate
> the OAM traffic as defined in Section 4.2 [I-D.ietf-sfc-ioam-nsh].

...FB2: IMHO the first sentence is still confusing, because you implicitly assumes that the overlay has an "O-Bit" (which is the case if you use NSH - but that does not have to be the case). Plus, the fact that IOAM also has an O-Bit with a very different definition, adds to potential confusion (you at least confused me).

How about:

In-Situ OAM could be leveraged to perform SF availability  and SFC availability or performance measurement. For example, if SFC is realized using NSH, the O-bit in the NSH header could be set to indicate the OAM traffic as defined in Section 4.2 [I-D.ietf-sfc-ioam-nsh].

>     >     available and implemented in OpenDaylight. Consider removing the
> references
>     > to
>     >     Loopback (see draft-ietf-ippm-ioam-flags-01) could apply SFC Traceroute
> as
>     > well.
>     >
>     >     Detailed set of nits that I encountered while reading through the
> document
>     > ([x]
> accordingly. If
> generating
>     >     [216] Ethernet OAM: Add reference. Do you refer to physical layer
> Ethernet
>     > OAM
>       transparent to the SFC data plane elements as not all the data plane
> elements process the overlay header.
> 
>     >
> <Nagendra2> "tightly vs loosely" coupled normally denotes the dependency (.
> Based on another reviewer's comment, we agreed to modify the relevant text as
> shown below. Does it look better?
> 
> NEW:

...FB2: Hmm. We're again back to using terms such as "popular". Per my earlier note, popularity might change, standards stay.

How about:
The link layer, which is associate to the physical layer technology used.  Example technologies for this layer include Ethernet, 
POS, DWDM.  In a virtual environment, virtualized I/O technologies such as SR-IOV or similar are applicable for this layer.

>    statistical methods.  Measuring the loss of OAM packets that share the fate
> with data traffic, an approximation of packet loss for a given SFC can be derived.

...FB: Hmm. Not sure that this really covers the case where the OAM packets are treated differently than the customer data packets.

How about:
SFC OAM should provide the ability to measure packet loss for an SFC. On-demand measurement can be used to estimate packet loss using statistical methods.  To ensure accurate estimations, one needs to ensure that OAM packets are treated the same and also share the same fate as regular data traffic. 

Cheers, Frank

>     > <Nagendra> Based on various reviewer comments, we removed the use of
> any
>     > normative statement.
>     >
>     > [666] Add MPLS, TRILL to
>     >     acronyms in 1.2.1
>     >
>     > <Nagendra> Ok. Done.
>     >
>     > <Nagendra> Based on various reviewer comments, we removed the use of
> any
>     > normative statement.
>     >
>     > <Nagendra> Based on various reviewer comments, we removed the use of
> any
>     > normative statement.
>     >
>     > [754]
>     >     s/packet/packets/
>     >

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 22 May 2020 09:33:56 +0000

Joel,

Nagendra suggested "The task of evaluating the true availability of a Service Function is a complex activity, currently having no simple, unified solution.  There is currently no standard means of doing so. Any such mechanism would be far from a typical OAM function, so it is not explored as part of the analysis in Sections 4 and 5." in the related thread for discussion on availability. https://mailarchive.ietf.org/arch/msg/sfc/1ZsLw6m1OeJRJfHW2TygxOyKPJk/

It could well be that my expectations for a framework document differ from that of others. So please take this as a personal perspective.
My expectation for a framework document is that it identifies and describes the components for a solution. It can optionally also hint at potential approaches to a solution (independently whether these solution are already in scope of the WG or not), but does not have to provide these solutions. The document does a good job at most of these - but misses out on those things, where we currently don't really have a solution available, which are questions like
* How do we define availability for a SFC at service level? How do we define availability for a SF?
* How do we define performance for a SFC at service level? How do we define performance for a SF?
Right now the document focuses on those pieces of SF/SFC availability and performance that are connectivity related ("Packets traverse it"). 

A potential approach could be to decompose the problem, using the structure the document lays out - and put this into the context of SLA definitions.

Why can we just say for e.g. SFC performance that SFC performance is a composite of 
* link layer performance, 
* underlay performance, 
* overlay performance, 
* service layer performance.
A SFC OAM solution needs to consider the performance measures across all those layers.
Consequently, for the performance of a SF, one needs to consider the aspects at each layer:
* link layer - Example "how well is the SF connected at e.g. Ethernet layer?" -> example: Leverage ITU-T Y.1731
* underlay - Example "how well is the SF connected at e.g. IP layer?" -> example: Leverage OWAMP/TWAMP
* overlay - Example "how well is the SF connected at e.g. NSH layer?"
* service layer - Example "how well does the SF meet the criteria of an SLA?"
So e.g. 3.1.2 and 3.2.2 could explicitly expand on these aspects and state that apart from connectivity level measures (loss, throughput), service level criteria, typically defined as part of an SLA, are included in SFC performance characterization. IMHO this is a better approach, than either say "SF availability is a hard problem" (which is what Nagendra says in the above statement - and which is of course true) or just define the service aspects as out of scope for SFC OAM ("how clean are the clothes when returned from the laundry?").

My 2cents..

Cheers, Frank

> Sent: Mittwoch, 20. Mai 2020 20:17
> So the question now is whether the text Murray suggested suffices for you?  (We
> are still waiting to hear from Alvaro.)
> 
> Yours,
> Joel
> 
> >> document was pretty clear that we consider that out of scope (c.f.
> >> the discussions with the various ADs.)
> >>
> >>>>       This document has been reviewed as part of the transport area
> >>>> review
> >> team's
> >>>>       ongoing effort to review key IETF documents. These comments
> >>>> were
> >> written
> >>>>       primarily for the transport area directors, but are copied to
> >>>> the
> >> document's
> >>>>       instances of a SF. Consequently, it should be defined how
> >>>> availability of a
> >> SF
> >>>> <Nagendra>The availability is measured as binary state. I am not
> >>>> sure what is 99% available. If it means getting 99 responses for
> >>>> 100 probes sent, I think it falls under packet loss category which
> >>>> in turn is
> >> performance measurement.
> >>>
> >> I put the statements above together, what would be the availability
> >> of the following setup: There is an SF that is made up of 100
> >> instances. 99 of these instances are powered down entirely. And the 1
> >> instance that is "up" is alternating between servicing requests for
> >> 10min followed by not servicing requests for 10min. Would the SF be
> considered "available"?
> >>>
> >>>>
> >>>> Section 3.1.2
> >>>> deployment" on SF
> >>>> "The metrics measured to quantify the performance of the SF
> >>>> component is not just limited to loss and delay. Other metrics such
> >>>> as throughout also exist and the choice of metrics for performance
> >>>>       packets?". IMHO we should also ask the question "Does my SFC
> >>>> process
> >> the
> >>>> end-to-end SFC availability. If the derived solution verifies the
> >>>> SF availability not just based on the uptime but based on the
> >>>> service treatment, it also answers the question "Does my SFC
> >>>> process the packets
> >>>>       measurements" a similar statement isn't found in 3.2.1. IMHO
> >>>> the ability
> >> for
> >>>>       availability checks is probably a prerequisite for
> >>>> performance
> >> measurement.
> >>>>
> >> itself. I.e. If you'd consider a "laundry service" - we focus a lot
> >> on how long it takes to get the clothes shipped to and from the
> >> washing machine, but we don't focus on how well the washing machine
> washes the clothes.
> >> service (especially given that you define a service layer in section
> >> 2) - or clearly state that the framework would just focus on connectivity
> between SFs.
> >>>
> >>>
> >>>>
> >>>>       component. What is performance measurement of the classifier?
> >>>> What
> >> does
> >>>>       3.5. Availability/PM of the underlay and overlay network: It
> >>>> would be good
> >> to
> >>>>       add a sentence that states that the mechanisms for
> >>>> availability/PM which
> >> are
> >>>>       Functions: It would be good, if examples in section 4 could
> >>>> also include
> >> more
> >>> ...FB: See above. This would be about adding a capability to assess
> >>> how well
> >> the washing machine washes my laundry.
> >>>
> >>>>
> >>>> Section 5.1
> >>>>       OAM Tool Gap Analysis:
> >>>> <Nagendra> Per my understanding, 802.ah is 1-hop while 802.3ag can
> >>>> be more than 1 hop and both uses Ethernet frames. So I think both
> >>>> are
> >> applicable here.
> >> Where is this covered? IMHO we would need references to IEEE
> >> standards to avoid confusion.
> >>>
> >>>>
> >>>> E-OAM offers OAM mechanisms such as an Ethernet continuity check
> >>>> for Ethernet links. There is an effort around NVO3 OAM to provide
> OAM"
> >> indeed existis. If in doubt, it might be better to avoid forward
> >> looking references. Per my note above, it would be good to explicitly
> >> refer to IEEE standards as opposed to introducing a new term like "E-OAM".
> >>>
> >>>>
> >>>> applicability of different OAM toolsets for OAM functions at
> >>>> service layer. I am not sure if we really should try explaining all
> >>>> the possible options within each tool. But I agree that it is worth
> >>>> clarifying the availability of IOAM options for tracing. think we
> >>>>       "In-Situ OAM could be used with O bit set": Why would IOAM be
> >>>> used with
> >> the
> >>>>       overflow bit set for SFC OAM? For details on IOAM's O-bit,
> >>>> see section
> >> 4.4.1 in
> >> explicitly refer to NSH here. E.g. If SFC is realized using NSH, then
> >> the O-bit in the NSH header could be used to indicated OAM traffic.
> >> You could refer to
> >>>>       references line number x) – hope that they are helpful in
> >>>> further improving
> >> the
> >>>>       doc:
> >>>>
> >>>>       [202] Why just Virtual Machines and no containers? Suggest to
> >>>> make
> >> things
> >>>> <Nagendra> The data plane elements connecting the overlay layer
> >>>> nodes may not always process the overlay header.
> >>>
> >>>>       layer, or could unequal cost multipath also be an option for
> >>>> load-
> >> balancing?
> >>>>
> >>>>    [521] Not sure whether the overlay network establishes the service
> plane.
> >> Isn't
> >>>> <Nagendra> The service layer is established over the overlay
> >>>> network layer. I am not sure if it is right to say overlay network
> >>>> provides connectivity for service layer (.
> >>>
> >>> ...FB: Overlay network is one component of the service layer, isn't
> >>> it. So it is
> >> required but not sufficient.
> >>>
> >>>>
> >>>>       "TTL or other means" (TTL also needs to be added to acronyms
> >>>> in 1.2.1). Is
> >> this
> >>>>       packet loss for a given SFC can be derived" to be applicable,
> >>>> SFC OAM
> >> packets
> >>>> <Nagendra> As it is intending to derive the approximate loss value,
> >>>> I am not sure if we need this additional consideration that the OAM
> >>>>       Especially given that RFC2119/RFC8174 is explicitly
> >>>> referenced by the
> >> draft.
> >>>>
> >>>> <Nagendra> As mentioned in the statement, the node that initiates
> >>>> the OAM packet must set the marker and so this statement is
> >>>> applicable for the initiating node.
> >>>>
> >>>> [806]
> >>>>       s/function/functions/
> >>>>
> >>>> <Nagendra> Done
> >>>>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Fri, 22 May 2020 10:48:51 -0400

My basic concern is that if we go down the path you outline, it would 
suggest to the reader that the SFC working group expected to address the 
SF dimensions of the problem.  I grant that they are real problems.  But 
the SFC working group has neither the charter nor the expertise to 
address those problems.  As far as I can tell, ART may have the 
expertise, and may not.  And has shown no interest in the problem.  So I 
am very reluctant to put a lot of verbiage into the RFC on SF monitoring.

Given the above, can you live with Nagendra's text that you quote?

Yours,
Joel

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 22 May 2020 15:03:27 +0000

Per what I mentioned below - the below is my personal take, taking the position of a reader, who might not be aware of WG structures, WG scopes etc. Personally, I don't think that a reader of the SFC OAM framework document would expect the SFC WG to address all of the dimensions of SFC OAM, even if the document lays them out.

That said, I do understand your concern about WG structures and scope. Looks like we (sadly) need to follow Conway's law ...
So in general terms, I'm ok with Nagendra's approach.

Cheers, Frank

> Sent: Freitag, 22. Mai 2020 16:49
> My basic concern is that if we go down the path you outline, it would suggest to
> the reader that the SFC working group expected to address the SF dimensions of
> the problem.  I grant that they are real problems.  But the SFC working group has
> neither the charter nor the expertise to address those problems.  As far as I can
> tell, ART may have the expertise, and may not.  And has shown no interest in the
> problem.  So I am very reluctant to put a lot of verbiage into the RFC on SF
> monitoring.
> 
> > Nagendra suggested "The task of evaluating the true availability of a
> > Service Function is a complex activity, currently having no simple,
> > unified solution.  There is currently no standard means of doing so.
> > Any such mechanism would be far from a typical OAM function, so it is
> > not explored as part of the analysis in Sections 4 and 5." in the
> > related thread for discussion on availability.
> > https://mailarchive.ietf.org/arch/msg/sfc/1ZsLw6m1OeJRJfHW2TygxOyKPJk/
> >
> > It could well be that my expectations for a framework document differ from
> that of others. So please take this as a personal perspective.
> > My expectation for a framework document is that it identifies and
> > describes the components for a solution. It can optionally also hint
> > at potential approaches to a solution (independently whether these
> > solution are already in scope of the WG or not), but does not have to
> > provide these solutions. The document does a good job at most of these
> > - but misses out on those things, where we currently don't really have
> > a solution available, which are questions like
> > Right now the document focuses on those pieces of SF/SFC availability and
> performance that are connectivity related ("Packets traverse it").
> >
> > A potential approach could be to decompose the problem, using the structure
> the document lays out - and put this into the context of SLA definitions.
> >
> > * link layer - Example "how well is the SF connected at e.g. Ethernet
> > layer?" -> example: Leverage ITU-T Y.1731
> > So e.g. 3.1.2 and 3.2.2 could explicitly expand on these aspects and state that
> apart from connectivity level measures (loss, throughput), service level criteria,
> typically defined as part of an SLA, are included in SFC performance
> characterization. IMHO this is a better approach, than either say "SF availability
> is a hard problem" (which is what Nagendra says in the above statement - and
> which is of course true) or just define the service aspects as out of scope for SFC
> OAM ("how clean are the clothes when returned from the laundry?").
> >
> > My 2cents..
> >
> > Cheers, Frank
> >
> >
> >>> Thanks Joel. Per what I mentioned below, let's be clear that SF
> >>> performance is
> >> out of scope for the doc.
> >>>>>>        This document has been reviewed as part of the transport
> >>>>>> area review
> >>>> team's
> >>>>>>        ongoing effort to review key IETF documents. These
> >>>>>> comments were
> >>>> written
> >>>>>>        primarily for the transport area directors, but are copied
> >>>>>> to the
> >>>> document's
> >>>>>>        When done at the time of IETF Last Call, the authors
> >>>>>> should consider
> >> this
> >>>>>>        Section 3.1.1 SF availability: The text makes explicit
> >>>>>> reference to
> >> multiple
> >>>>>> 100 probes sent, I think it falls under packet loss category
> >>>>>> which in turn is
> >>>> performance measurement.
> >>>>>
> >>>> instances. 99 of these instances are powered down entirely. And the
> >>>> 1 instance that is "up" is alternating between servicing requests
> >>>> for 10min followed by not servicing requests for 10min. Would the
> >>>> SF be
> >> considered "available"?
> >>>>>
> >>>>>>
> >>>>>> Section 3.1.2
> >>>>>> component is not just limited to loss and delay. Other metrics
> >>>>>> such as throughout also exist and the choice of metrics for
> >>>>>> performance measurement is outside the scope of this document."
> >>>>>>
> >>>>>> Section 3.2.1 SFC
> >>>>>>        packets?". IMHO we should also ask the question "Does my
> >>>>>> SFC process
> >>>> the
> >>>>>> <Nagendra> I think this is already handled by SF availability.
> >>>>>> The end-to-end SFC availability is verified by steering the OAM
> >>>>>> packet over the ordered set of SFs within the SFC. This is more
> >>>>>> like daisy chaining the availability of SFs within the SFC to
> >>>>>> determine end-to-end SFC availability. If the derived solution
> >>>>>> verifies the SF availability not just based on the uptime but
> >>>>>> based on the service treatment, it also answers the question
> >>>>>> "Does my SFC process the packets
> >>>>>>        measurements" a similar statement isn't found in 3.2.1.
> >>>>>> IMHO the ability
> >>>> for
> >>>>>>        need for the ability to perform performance measurement of
> >>>>>> the
> >> classifier
> >>>>>>        3.5. Availability/PM of the underlay and overlay network:
> >>>>>> It would be good
> >>>> to
> >>>>>>        Functions: It would be good, if examples in section 4
> >>>>>> could also include
> >>>> more
> >>>>>>        Performance Measurement: Focus is entirely on the PM of
> >>>>>> the
> >>>> connectivity,
> >>>>>> <Nagendra> I am not sure I understand what is missing. Do you
> >>>>>> have any suggestion for the text improvement?.
> >>>>>
> >>>>> ...FB: See above. This would be about adding a capability to
> >>>>> assess how well
> >>>>>> <Nagendra> Per my understanding, 802.ah is 1-hop while 802.3ag
> >>>>>> can be more than 1 hop and both uses Ethernet frames. So I think
> >>>>>> both are
> >>>> applicable here.
> >>>>> ...FB: Maybe I missed it - but I don't see text that refers to CFM
> >>>>> or EFM
> >> OAM.
> >>>>>> connectivity and continuity checks for networks that use NVO3.
> >>>>>> BFD is used for the detection of data plane forwarding failures.
> >>>>>
> >>>> looking references. Per my note above, it would be good to
> >>>> explicitly refer to IEEE standards as opposed to introducing a new term like
> "E-OAM".
> >>>>>
> >>>>>>
> >>>>>> The IPPM framework [RFC 2330] offers tools such as OWAMP
> >>>>>> [RFC4656] and TWAMP [RFC5357] (collectively referred as IPPM in
> >>>>>> this section) to measure various performance metrics. MPLS Packet
> >>>>>> Loss Measurement
> >>>>>>        - The section states that IOAM "may be used to perform
> >>>>>> various SFC
> >> OAM
> >>>>>>        functions as well". It would be good to expand on this statement:
> E.g.
> >>>> IOAM
> >>>>>>        Type could be leveraged. - How would we deal with the IOAM
> >>>>>> Active
> >> Flag
> >>>>>> service layer. I am not sure if we really should try explaining
> >>>>>> all the possible options within each tool. But I agree that it is
> >>>>>> worth clarifying the availability of IOAM options for tracing.
> >>>>>> think we can clarify that different IOAM Option-Types are
> >>>>>> available for OAM functions
> >>>>>>       defined inband (i.e., it will be included in data packets),
> >>>>>> IOAM Option-
> >> Types
> >>>>>>        "In-Situ OAM could be used with O bit set": Why would IOAM
> >>>>>> be used with
> >>>> the
> >>>>>> <Nagendra> The O bit referred here is not the O bit in IOAM but
> >>>>>> the one in NSH/Overlay header. To avoid any confusion, this can
> >>>>>> be updated as
> >>>> below:
> >>>>>>
> >>>>>> OLD:
> >>>> explicitly refer to NSH here. E.g. If SFC is realized using NSH,
> >>>> then the O-bit in the NSH header could be used to indicated OAM traffic.
> >>>> You could refer to
> >>>>>>        Traceroute: - This section refers to an expired draft
> >>>>>> (even calling out
> >> the
> >>>>>> accordingly. If your suggestion is to substitute "a SF" to "an
> >>>>>> SF",  it is done
> >> (.
> >>>>>>
> >>>>>>        [202] Why just Virtual Machines and no containers? Suggest
> >>>>>> to make
> >>>> things
> >>>>>>        technologies"? For the underlay network layer several
> >>>>>> examples are
> >> listed.
> >>>>>>
> >>>>>> <Nagendra> Ok.
> >>>>>>
> >>>>>> network layer. I am not sure if it is right to say overlay
> >>>>>> network provides connectivity for service layer (.
> >>>>>
> >>>>> ...FB: Overlay network is one component of the service layer,
> >>>>> isn't it. So it is
> >>>>>>        "TTL or other means" (TTL also needs to be added to
> >>>>>> acronyms in 1.2.1). Is
> >>>> this
> >>>>>>        packet loss for a given SFC can be derived" to be
> >>>>>> applicable, SFC OAM
> >>>> packets
> >>>>>> <Nagendra> As it is intending to derive the approximate loss
> >>>>>> value, I am not sure if we need this additional consideration
> >>>>>> that the OAM packet would need to follow the live user traffic.
> >>>>>> Let me know if you think
> >>>> otherwise.
> >>>>>

---

From: Joel Halpern Direct <[email-masked]>
Date: Fri, 22 May 2020 11:17:48 -0400

Thanks Frank.  Both for the review, and the discussion.
Yours,
Joel

---

From: "Nagendra Kumar Nainar (naikumar)" <[email-masked]>
Date: Sat, 23 May 2020 12:18:25 +0000

Hi Frank,

Fixed all the updates. I will submit a new revision soon.

Thanks,
Nagendra

On 5/22/20, 3:56 AM, "Frank Brockners (fbrockne)" <[email-masked]> wrote:

    Hi Nagendra,

    How about:

    How about:

    How about:

    Cheers, Frank