Subject: [ippm] Progressing draft-ietf-ippm-ioam-conf-state

From: <[email-masked]>
Date: Fri, 10 Dec 2021 15:00:30 +0800 (CST)

Hi IPPM WG,

The -02 version of draft-ietf-ippm-ioam-conf-state has been posted.
There are mainly two changes, one is on IOAM Tracing Capabilities Objects to make them applicable to ICMPv6 extensions, another one is on IOAM Proof-of-Transit Capabilities Object to make it aligned with the updated IOAM-Data document.
Also note that I've had an offline discussion with wg chairs and Frank Brockners, my conclusion is that Frank's concern raised at IETF 112 has been addressed. If that's not the case, please speak up.
With that said, I think this draft is ready for WGLC.

Best Regards,
Xiao Min

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 16 Dec 2021 12:54:09 +0000

Hi Xiao Min,

Thanks for posting draft-ietf-ippm-ioam-conf-state-02. I read through the updated version and looked at the diff to the 01 version (https://www.ietf.org/rfcdiff?url1=draft-ietf-ippm-ioam-conf-state-01&url2=draft-ietf-ippm-ioam-conf-state-02). Different from what you state below, I don't see any of my comments reflected. 
The two main points I mentioned in the last WG meeting were about (a) alignment with draft-ietf-ippm-ioam-yang and (b) enhancements to the security section, reflecting that the protocol you are defining is a network management protocol, and needs to be secured as such. 

More specifically: 
* Alignment with draft-ietf-ippm-ioam-yang: 
The IPPM WG is in the process of defining a YANG module for IOAM: draft-ietf-ippm-ioam-yang. We should have a single, comprehensive model for config information for IOAM. That model can then be rendered into different transports (be it JSON, XML, or yet another format - to then be carried over a e.g. ICMP in your case). Right now draft-ietf-ippm-ioam-conf-state heads down a path of defining a new set of management objects - many are similar to what draft-ietf-ippm-ioam-yang already defines, some they are less comprehensive, some hint at additional information that draft-ietf-ippm-ioam-yang does not cover yet: E.g.,  you define a "Pre-allocated Tracing Capabilities Object" where draft-ietf-ippm-ioam-yang  has a "Preallocated Tracing Profile" defined. You define ingress interface fields, which is information, which is more comprehensively defined by the ioam-filter grouping. You define specific fields to describe the timestamp format used by a node, which is information that should be described as part of the ioam-info container - and which is currently missing in draft-ietf-ippm-ioam-yang; point well taken :-). It is interesting to note that draft-ietf-ippm-ioam-conf-state does not even reference draft-ietf-ippm-ioam-yang.
From my perspective we should have one single data model for IOAM configuration - and that is the YANG module defined in draft-ietf-ippm-ioam-yang. Let's make sure that this model covers all the information required to properly manage IOAM. Then draft-ietf-ippm-ioam-conf-state would be solely focused on defining how that YANG module (from draft-ietf-ippm-ioam-yang) would be rendered into e.g., ICMP as a carrier protocol. 

* Security: Revision -02 does not seem to update the security section. IMHO, section 6 on security considerations should be enhanced to clearly articulate that we're dealing with very sensitive information. Consider loaning from e.g., https://datatracker.ietf.org/doc/html/rfc6241#section-9. As part of the discussion, it would be good to see an explanation where you'd expect  draft-ietf-ippm-ioam-conf-state to be used. Given the discussion in the Introduction section, you seem to assume an environment that does not have a central network management station / controller in place. Or in other terms, you seem to target a deployment, which isn't a limited domain per RFC 8799. In that case, I would expect that we have normative language that requires the use of strong authentication and encryption between the nodes (i.e., MUST use ICMP with AH and ESP..).

In addition to the above, I struggle to understand the "Operational Guide" in section 4: Could you shed a bit more light on how you expect things to work - and what a target deployment environment would look like? It seems that you assume that you don't know the network nor the destination addresses in your network: Do you expect that you would do regular "ICMP echo sweeps" in your network? Do you expect that, while doing the expanding ring search, an ICMP time exceeded message would also carry the "IOAM Capabilities Response Container Header", so that the capabilities container would not only be carried in the echo reply message but also in the time exceeded message? 

Thanks, Frank

(BTW - Note that due to the upcoming holiday season, replies might be delayed).

> -----Original Message-----
> From: ippm <[email-masked]> On Behalf Of [email-masked]
> Sent: Friday, 10 December 2021 08:01
> To: [email-masked]
> Subject: [ippm] Progressing draft-ietf-ippm-ioam-conf-state
> 
> Hi IPPM WG,
> 
> There are mainly two changes, one is on IOAM Tracing Capabilities Objects to
> make them applicable to ICMPv6 extensions, another one is on IOAM Proof-of-
> Transit Capabilities Object to make it aligned with the updated IOAM-Data
> document.
> Also note that I've had an offline discussion with wg chairs and Frank Brockners,
> my conclusion is that Frank's concern raised at IETF 112 has been addressed. If
> that's not the case, please speak up.
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm

---

From: <[email-masked]>
Date: Fri, 17 Dec 2021 10:28:35 +0800 (CST)

Hi Frank,

Thanks for your thorough review and thoughtful comments.
I apologize if I misinterpreted your mind.
Considering that your holiday season is coming, before digging into technical details, I suggest that we discuss the IOAM deployment environment first.
As you may have noticed while reading through draft-ietf-ippm-ioam-conf-state-02, in the introduction section it says
"A centralized controller which owns the enabled IOAM capabilities of each IOAM device could be used in some IOAM deployments.  The IOAM encapsulating node can discover the enabled IOAM capabilities infomation from the centralized controller, using, for example, NETCONF/YANG, PCEP, or BGP.  In the IOAM deployment scenario where there is no centralized controller, NETCONF/YANG or IGP may be used by the IOAM encapsulating node to discover these IOAM capabilities information."
I know you're the primary author of both IOAM-Data and IOAM-Deployment documents, so I have a fundamental question to you: Is the above statement correct or not?
More specifically, if you can confirm that "the IOAM deployment scenario where there is no centralized controller" doesn't exist at all, on both the mailing list and the IOAM-Deployment document, then I suggest IPPM WG to abandon draft-ietf-ippm-ioam-conf-state, that would save the energy of the whole wg including you and me.

Best Regards,
Xiao Min
------------------原始邮件------------------
发件人：FrankBrockners(fbrockne)
收件人：肖敏10093570;[email-masked];
抄送人：Tommy Pauly;
日 期 ：2021年12月16日 20:56
主 题 ：RE: [ippm] Progressing draft-ietf-ippm-ioam-conf-state
Hi Xiao Min,
Thanks for posting draft-ietf-ippm-ioam-conf-state-02. I read through the updated version and looked at the diff to the 01 version (https://www.ietf.org/rfcdiff?url1=draft-ietf-ippm-ioam-conf-state-01&url2=draft-ietf-ippm-ioam-conf-state-02). Different from what you state below, I don't see any of my comments reflected.
The two main points I mentioned in the last WG meeting were about (a) alignment with draft-ietf-ippm-ioam-yang and (b) enhancements to the security section, reflecting that the protocol you are defining is a network management protocol, and needs to be secured as such.
More specifically:
* Alignment with draft-ietf-ippm-ioam-yang:
The IPPM WG is in the process of defining a YANG module for IOAM: draft-ietf-ippm-ioam-yang. We should have a single, comprehensive model for config information for IOAM. That model can then be rendered into different transports (be it JSON, XML, or yet another format - to then be carried over a e.g. ICMP in your case). Right now draft-ietf-ippm-ioam-conf-state heads down a path of defining a new set of management objects - many are similar to what draft-ietf-ippm-ioam-yang already defines, some they are less comprehensive, some hint at additional information that draft-ietf-ippm-ioam-yang does not cover yet: E.g.,  you define a "Pre-allocated Tracing Capabilities Object" where draft-ietf-ippm-ioam-yang  has a "Preallocated Tracing Profile" defined. You define ingress interface fields, which is information, which is more comprehensively defined by the ioam-filter grouping. You define specific fields to describe the timestamp format used by a node, which is information that should be 
 described as part of the ioam-info container - and which is currently missing in draft-ietf-ippm-ioam-yang; point well taken :-). It is interesting to note that draft-ietf-ippm-ioam-conf-state does not even reference draft-ietf-ippm-ioam-yang.

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Fri, 17 Dec 2021 09:45:29 +0000

Hi Xiao Min,

per https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-data-17#section-4, IOAM is focused on "limited domains" as defined in [RFC8799], which is per RFC 8799 the same as what is also referred to as a "controlled environment". Personally I understand "Controlled environment" as an environment where an operator does have control over the nodes in the network and their configuration, i.e., the operator has control over his nodes, knows what network elements he is dealing with and what the config of the nodes is. How this control is implemented - whether there is one or several "controllers" or "network managers" or similar - is an implementation detail IMHO.

Cheers, Frank

> From: [email-masked] <[email-masked]>
> Sent: Friday, 17 December 2021 03:29
> To: Frank Brockners (fbrockne) <[email-masked]>
> Cc: [email-masked]; [email-masked]
> Subject: Re:[ippm] Progressing draft-ietf-ippm-ioam-conf-state
> 
> Hi Frank,
> 
> Considering that your holiday season is coming, before digging into technical
> details, I suggest that we discuss the IOAM deployment environment first.
> As you may have noticed while reading through draft-ietf-ippm-ioam-conf-
> state-02, in the introduction section it says "A centralized controller which owns
> the enabled IOAM capabilities of each IOAM device could be used in some IOAM
> deployments.  The IOAM encapsulating node can discover the enabled IOAM
> capabilities infomation from the centralized controller, using, for example,
> NETCONF/YANG, PCEP, or BGP.  In the IOAM deployment scenario where there
> is no centralized controller, NETCONF/YANG or IGP may be used by the IOAM
> encapsulating node to discover these IOAM capabilities information."
> I know you're the primary author of both IOAM-Data and IOAM-Deployment
> documents, so I have a fundamental question to you: Is the above statement
> correct or not?
> More specifically, if you can confirm that "the IOAM deployment scenario where
> there is no centralized controller" doesn't exist at all, on both the mailing list and
> the IOAM-Deployment document, then I suggest IPPM WG to abandon draft-
> ietf-ippm-ioam-conf-state, that would save the energy of the whole wg
> including you and me.
> 
> Best Regards,
> Xiao Min
> 主 题 ：RE: [ippm] Progressing draft-ietf-ippm-ioam-conf-state Hi Xiao Min,
> Thanks for posting draft-ietf-ippm-ioam-conf-state-02. I read through the
> updated version and looked at the diff to the 01 version
> (https://www.ietf.org/rfcdiff?url1=draft-ietf-ippm-ioam-conf-state-
> 01&url2=draft-ietf-ippm-ioam-conf-state-02). Different from what you state
> below, I don't see any of my comments reflected.
> The two main points I mentioned in the last WG meeting were about (a)
> alignment with draft-ietf-ippm-ioam-yang and (b) enhancements to the security
> section, reflecting that the protocol you are defining is a network management
> protocol, and needs to be secured as such.
> More specifically:
> The IPPM WG is in the process of defining a YANG module for IOAM: draft-ietf-
> ippm-ioam-yang. We should have a single, comprehensive model for config
> information for IOAM. That model can then be rendered into different
> transports (be it JSON, XML, or yet another format - to then be carried over a
> e.g. ICMP in your case). Right now draft-ietf-ippm-ioam-conf-state heads down
> a path of defining a new set of management objects - many are similar to what
> draft-ietf-ippm-ioam-yang already defines, some they are less comprehensive,
> some hint at additional information that draft-ietf-ippm-ioam-yang does not
> cover yet: E.g.,  you define a "Pre-allocated Tracing Capabilities Object" where
> draft-ietf-ippm-ioam-yang  has a "Preallocated Tracing Profile" defined. You
> define ingress interface fields, which is information, which is more
> comprehensively defined by the ioam-filter grouping. You define specific fields
> to describe the timestamp format used by a node, which is information that
> should be  described as part of the ioam-info container - and which is currently
> missing in draft-ietf-ippm-ioam-yang; point well taken :-). It is interesting to
> note that draft-ietf-ippm-ioam-conf-state does not even reference draft-ietf-
> ippm-ioam-yang.
> From my perspective we should have one single data model for IOAM
> configuration - and that is the YANG module defined in draft-ietf-ippm-ioam-
> yang. Let's make sure that this model covers all the information required to
> properly manage IOAM. Then draft-ietf-ippm-ioam-conf-state would be solely
> focused on defining how that YANG module (from draft-ietf-ippm-ioam-yang)
> would be rendered into e.g., ICMP as a carrier protocol.
> * Security: Revision -02 does not seem to update the security section. IMHO,
> section 6 on security considerations should be enhanced to clearly articulate
> that we're dealing with very sensitive information. Consider loaning from e.g.,
> https://datatracker.ietf.org/doc/html/rfc6241#section-9. As part of the
> discussion, it would be good to see an explanation where you'd expect  draft-
> ietf-ippm-ioam-conf-state to be used. Given the discussion in the Introduction
> section, you seem to assume an environment that does not have a central
> network management station / controller in place. Or in other terms, you seem
> to target a deployment, which isn't a limited domain per RFC 8799. In that case,
> I would expect that we have normative language that requires the use of strong
> authentication and encryption between the nodes (i.e., MUST use ICMP with AH
> and ESP..).
> In addition to the above, I struggle to understand the "Operational Guide" in
> section 4: Could you shed a bit more light on how you expect things to work -
> and what a target deployment environment would look like? It seems that you
> assume that you don't know the network nor the destination addresses in your
> network: Do you expect that you would do regular "ICMP echo sweeps" in your
> network? Do you expect that, while doing the expanding ring search, an ICMP
> time exceeded message would also carry the "IOAM Capabilities Response
> Container Header", so that the capabilities container would not only be carried
> in the echo reply message but also in the time exceeded message?
> Thanks, Frank
> > There are mainly two changes, one is on IOAM Tracing Capabilities
> > Objects to make them applicable to ICMPv6 extensions, another one is
> > on IOAM Proof-of- Transit Capabilities Object to make it aligned with
> > the updated IOAM-Data document.
> > Also note that I've had an offline discussion with wg chairs and Frank
> > Brockners, my conclusion is that Frank's concern raised at IETF 112
> > has been addressed. If that's not the case, please speak up.

---

From: <[email-masked]>
Date: Mon, 20 Dec 2021 10:01:08 +0800 (CST)

Hi Frank,

Thank you for the quick reply.
I believe you've provided your answer to my question implicitly, however what I requested is a clear conclusion, on both the mailing list and the draft-ietf-ippm-ioam-deployment.
If I understand you correctly this time, you meant that the IOAM deployment must include a centralized controller or network manager or similar, then I have a hard time trying to understand why four types of IOAM are needed in section 4 of draft-ietf-ippm-ioam-deployment (https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-deployment#section-4). IMHO only one type of IOAM is necessary and enough, that's Direct Export IOAM described in section 4.4, would you please explain elaborately?

Best Regards,
Xiao Min
------------------原始邮件------------------
发件人：FrankBrockners(fbrockne)
收件人：肖敏10093570;
抄送人：[email-masked];[email-masked];
日 期 ：2021年12月17日 17:47
主 题 ：RE: Re:[ippm] Progressing draft-ietf-ippm-ioam-conf-state
Hi Xiao Min,
per https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-data-17#section-4, IOAM is focused on "limited domains" as defined in [RFC8799], which is per RFC 8799 the same as what is also referred to as a "controlled environment". Personally I understand "Controlled environment" as an environment where an operator does have control over the nodes in the network and their configuration, i.e., the operator has control over his nodes, knows what network elements he is dealing with and what the config of the nodes is. How this control is implemented - whether there is one or several "controllers" or "network managers" or similar - is an implementation detail IMHO.
Cheers, Frank

---

From: "Chengli (Cheng Li)" <[email-masked]>
Date: Wed, 22 Dec 2021 02:42:07 +0000

I would like to echo with the security part, cause I have the same considerations with Frank's. 

The most important question should be answered by the authors should be: how to use this mechanism is what environment? And how to ensure the security? 

P2P IPSec connection needs to be built among your network, so N nodes will need N*(N-1)/2*2 connections are needed(full-mesh and bidirectional). Is it worth to do it only for getting iOAM capabilities info? Or may be we can collect it by a controller via NETCONF/YANG or BGP-LS?  Why do we do it in data plane?

Respect,
Cheng

'''
'''

-----Original Message-----
From: ippm [mailto:[email-masked]] On Behalf Of Frank Brockners (fbrockne)
Sent: Thursday, December 16, 2021 8:54 PM
To: [email-masked]; [email-masked]
Subject: Re: [ippm] Progressing draft-ietf-ippm-ioam-conf-state

Hi Xiao Min,

More specifically: 

Thanks, Frank

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 28 Dec 2021 15:19:12 +0000

Hi Xiao Min,

Please see inline ("...FB")

> Sent: Monday, 20 December 2021 03:01
> I believe you've provided your answer to my question implicitly, however what I
> requested is a clear conclusion, on both the mailing list and the draft-ietf-ippm-
> ioam-deployment.
> If I understand you correctly this time, you meant that the IOAM deployment
> must include a centralized controller or network manager or similar, 

...FB: Let me restate what I said below: The current IOAM drafts assume that IOAM is deployed in a limited domain (RFC8799), also called controlled domain, i.e., a deployment where an operator does have control over the nodes in the network and their configuration. IMHO the use of the term "controller" does not help the discussion, because "controller" means different things to different people. The key aspect is that in a limited/controlled domain, the operator has control over the nodes in the limited domain and their configuration.

Wrt/ DEX: On top of the ability to control the nodes and their configuration, DEX requires you to have a solution in place which allows you to retrieve, ingest, process, and correlate per-packet IOAM data from every node in the limited domain. This requirement to retrieve, ingest, process, and correlate per-packet IOAM data from every node in the limited domain is specific to the DEX IOAM-Option-Type and not required by the other IOAM-Option-Types. For example, the two IOAM Trace-Option-Types automatically correlate the information from the different nodes by assembling the info in the node-list array while the packet traverses the network. 

What is required from my perspective for draft-ietf-ippm-ioam-conf-state, is a crisp definition of the target deployment environment draft-ietf-ippm-ioam-conf-state targets and the personas acting in that deployment environment. In a limited/controlled domain, we can assume that the operator knows the configuration of the nodes. The problem definition of draft-ietf-ippm-ioam-conf-state seems to be that "someone" who does not know the configuration of the nodes in the deployment environment wants to retrieve the configuration of the nodes.

Cheers, Frank

> then I have
> a hard time trying to understand why four types of IOAM are needed in section 4
> of draft-ietf-ippm-ioam-deployment
> (https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-
> deployment#section-4). IMHO only one type of IOAM is necessary and enough,
> that's Direct Export IOAM described in section 4.4, would you please explain
> elaborately?
> 
> Best Regards,
> Xiao Min
> 主 题 ：RE: Re:[ippm] Progressing draft-ietf-ippm-ioam-conf-state Hi Xiao Min,
> per https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-data-
> 17#section-4, IOAM is focused on "limited domains" as defined in [RFC8799],
> which is per RFC 8799 the same as what is also referred to as a "controlled
> environment". Personally I understand "Controlled environment" as an
> environment where an operator does have control over the nodes in the
> network and their configuration, i.e., the operator has control over his nodes,
> knows what network elements he is dealing with and what the config of the
> nodes is. How this control is implemented - whether there is one or several
> "controllers" or "network managers" or similar - is an implementation detail
> IMHO.
> Cheers, Frank
> > Considering that your holiday season is coming, before digging into
> > technical details, I suggest that we discuss the IOAM deployment environment
> first.
> > As you may have noticed while reading through
> > draft-ietf-ippm-ioam-conf- state-02, in the introduction section it
> > says "A centralized controller which owns the enabled IOAM
> > capabilities of each IOAM device could be used in some IOAM
> > deployments.  The IOAM encapsulating node can discover the enabled
> > IOAM capabilities infomation from the centralized controller, using,
> > for example, NETCONF/YANG, PCEP, or BGP.  In the IOAM deployment
> > scenario where there is no centralized controller, NETCONF/YANG or IGP may
> be used by the IOAM encapsulating node to discover these IOAM capabilities
> information."
> > documents, so I have a fundamental question to you: Is the above
> > statement correct or not?
> > More specifically, if you can confirm that "the IOAM deployment
> > scenario where there is no centralized controller" doesn't exist at
> > all, on both the mailing list and the IOAM-Deployment document, then I
> > suggest IPPM WG to abandon draft- ietf-ippm-ioam-conf-state, that
> > would save the energy of the whole wg including you and me.
> >
> > Best Regards,
> > Xiao Min
> > 主 题 ：RE: [ippm] Progressing draft-ietf-ippm-ioam-conf-state Hi Xiao
> > Min, Thanks for posting draft-ietf-ippm-ioam-conf-state-02. I read
> > through the updated version and looked at the diff to the 01 version
> > 01&url2=draft-ietf-ippm-ioam-conf-state-02). Different from what you
> > state below, I don't see any of my comments reflected.
> > alignment with draft-ietf-ippm-ioam-yang and (b) enhancements to the
> > security section, reflecting that the protocol you are defining is a
> > network management protocol, and needs to be secured as such.
> > More specifically:
> > The IPPM WG is in the process of defining a YANG module for IOAM:
> > draft-ietf- ippm-ioam-yang. We should have a single, comprehensive
> > model for config information for IOAM. That model can then be rendered
> > into different transports (be it JSON, XML, or yet another format - to
> > then be carried over a e.g. ICMP in your case). Right now
> > draft-ietf-ippm-ioam-conf-state heads down a path of defining a new
> > set of management objects - many are similar to what
> > draft-ietf-ippm-ioam-yang already defines, some they are less
> > comprehensive, some hint at additional information that
> > draft-ietf-ippm-ioam-yang does not cover yet: E.g.,  you define a
> > "Pre-allocated Tracing Capabilities Object" where
> > draft-ietf-ippm-ioam-yang  has a "Preallocated Tracing Profile"
> > defined. You define ingress interface fields, which is information,
> > which is more comprehensively defined by the ioam-filter grouping. You
> > define specific fields to describe the timestamp format used by a
> > node, which is information that should be  described as part of the
> > ioam-info container - and which is currently missing in draft-ietf-ippm-ioam-
> yang; point well taken :-). It is interesting to note that draft-ietf-ippm-ioam-
> conf-state does not even reference draft-ietf- ippm-ioam-yang.
> > configuration - and that is the YANG module defined in
> > draft-ietf-ippm-ioam- yang. Let's make sure that this model covers all
> > the information required to properly manage IOAM. Then
> > draft-ietf-ippm-ioam-conf-state would be solely focused on defining
> > how that YANG module (from draft-ietf-ippm-ioam-yang) would be rendered
> into e.g., ICMP as a carrier protocol.
> > * Security: Revision -02 does not seem to update the security section.
> > IMHO, section 6 on security considerations should be enhanced to
> > clearly articulate that we're dealing with very sensitive information.
> > Consider loaning from e.g.,
> > https://datatracker.ietf.org/doc/html/rfc6241#section-9. As part of
> > the discussion, it would be good to see an explanation where you'd
> > expect  draft- ietf-ippm-ioam-conf-state to be used. Given the
> > discussion in the Introduction section, you seem to assume an
> > environment that does not have a central network management station /
> > controller in place. Or in other terms, you seem to target a
> > deployment, which isn't a limited domain per RFC 8799. In that case, I
> > would expect that we have normative language that requires the use of strong
> > In addition to the above, I struggle to understand the "Operational
> > Guide" in section 4: Could you shed a bit more light on how you expect
> > things to work - and what a target deployment environment would look
> > like? It seems that you assume that you don't know the network nor the
> > destination addresses in your
> > network: Do you expect that you would do regular "ICMP echo sweeps" in
> > your network? Do you expect that, while doing the expanding ring
> > search, an ICMP time exceeded message would also carry the "IOAM
> > Capabilities Response Container Header", so that the capabilities
> > container would not only be carried in the echo reply message but also in the
> time exceeded message?
> > Thanks, Frank
> > > on IOAM Proof-of- Transit Capabilities Object to make it aligned
> > > with the updated IOAM-Data document.
> > > Also note that I've had an offline discussion with wg chairs and
> > > Frank Brockners, my conclusion is that Frank's concern raised at
> > > IETF 112 has been addressed. If that's not the case, please speak up.

---

From: <[email-masked]>
Date: Wed, 29 Dec 2021 16:15:02 +0800 (CST)

Hi Frank,

Thanks for replying to my email during your holiday season.
Please see inline ("XM>>>").

Best Regards,
Xiao Min
------------------原始邮件------------------
日 期 ：2021年12月28日 23:21
> must include a centralized controller or network manager or similar,
..FB: Let me restate what I said below: The current IOAM drafts assume that IOAM is deployed in a limited domain (RFC8799), also called controlled domain, i.e., a deployment where an operator does have control over the nodes in the network and their configuration. IMHO the use of the term "controller" does not help the discussion, because "controller" means different things to different people. The key aspect is that in a limited/controlled domain, the operator has control over the nodes in the limited domain and their configuration.
XM>>> In a limited domain, the operator has control over the nodes in the limited domain and their configuration, doesn't mean there must be a network entity (e.g. controller or network manager or similar) which has control over the nodes in the limited domain and their configuration, is my understanding correct or not?
Wrt/ DEX: On top of the ability to control the nodes and their configuration, DEX requires you to have a solution in place which allows you to retrieve, ingest, process, and correlate per-packet IOAM data from every node in the limited domain. This requirement to retrieve, ingest, process, and correlate per-packet IOAM data from every node in the limited domain is specific to the DEX IOAM-Option-Type and not required by the other IOAM-Option-Types. For example, the two IOAM Trace-Option-Types automatically correlate the information from the different nodes by assembling the info in the node-list array while the packet traverses the network.
XM>>> This requirement to retrieve, ingest, process, and correlate per-packet IOAM data from every node in the limited domain is not required by some IOAM-Option-Types, that means there may be a limited domain that doesn't meet the requirement that there is a network entity (e.g. controller or network manager or similar) which has direct connection to every node, is my understanding correct or not?
What is required from my perspective for draft-ietf-ippm-ioam-conf-state, is a crisp definition of the target deployment environment draft-ietf-ippm-ioam-conf-state targets and the personas acting in that deployment environment. In a limited/controlled domain, we can assume that the operator knows the configuration of the nodes. The problem definition of draft-ietf-ippm-ioam-conf-state seems to be that "someone" who does not know the configuration of the nodes in the deployment environment wants to retrieve the configuration of the nodes.
XM>>> Let me give a simplified example of the target deployment environment and the personas acting in that deployment environment. Assuming that the limited domain is an IPv6 domain, specifically an enterprise campus using physical connections between devices (as described in section 4 of draft-ietf-ippm-ioam-data-17), the operator knows the configuration of every node within the enterprise campus, however there is no a network entity (e.g. controller or network manager or similar) that knows the configuration of every node. In order to deploy IOAM Pre-allocated Tracing, the operator instructs the IOAM encapsulating node to add IOAM header into data packets meeting a rule of DA+SA+Flow Label, through Command Line or SNMP or NETCONF or similar, the IOAM encapsulating node firstly does traceroute using DA+SA+Flow Label and secondly sends ICMPv6 Echo Request to every node discovered by traceroute, the IOAM transit and decapsulating nodes respond with ICMPv6 Echo Reply, and then the IOA
 M encapsulating node knows what the IOAM header looks like, e.g. including how many node data lists and how much space allocated for each node data list. Is this example clear or not?
Cheers, Frank

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Wed, 29 Dec 2021 10:40:10 +0000

Hi Xiao Min,

Please see inline...

> Sent: Wednesday, 29 December 2021 09:15
> > I believe you've provided your answer to my question implicitly,
> > however what I requested is a clear conclusion, on both the mailing
> > list and the draft-ietf-ippm- ioam-deployment.
> > If I understand you correctly this time, you meant that the IOAM
> > deployment must include a centralized controller or network manager or
> > similar,
> ..FB: Let me restate what I said below: The current IOAM drafts assume that
> IOAM is deployed in a limited domain (RFC8799), also called controlled domain,
> i.e., a deployment where an operator does have control over the nodes in the
> network and their configuration. IMHO the use of the term "controller" does not
> help the discussion, because "controller" means different things to different
> people. The key aspect is that in a limited/controlled domain, the operator has
> control over the nodes in the limited domain and their configuration.
> XM>>> In a limited domain, the operator has control over the nodes in the
> limited domain and their configuration, doesn't mean there must be a network
> entity (e.g. controller or network manager or similar) which has control over the
> nodes in the limited domain and their configuration, is my understanding correct
> or not?

...FB: The operator has control over the nodes of the limited domain - i.e., the operator is the entity that owns the control of the network. Operators use a variety of tools and methods to operationalize the control. From what I know, there is no crisp definition of how the control is operationalized - and TBH, it would be quite challenging to create a comprehensive list, which would include all types of approaches to configure and operate devices.  

> Wrt/ DEX: On top of the ability to control the nodes and their configuration, DEX
> requires you to have a solution in place which allows you to retrieve, ingest,
> process, and correlate per-packet IOAM data from every node in the limited
> domain. This requirement to retrieve, ingest, process, and correlate per-packet
> IOAM data from every node in the limited domain is specific to the DEX IOAM-
> Option-Type and not required by the other IOAM-Option-Types. For example,
> the two IOAM Trace-Option-Types automatically correlate the information from
> the different nodes by assembling the info in the node-list array while the packet
> traverses the network.
> XM>>> This requirement to retrieve, ingest, process, and correlate per-packet
> IOAM data from every node in the limited domain is not required by some IOAM-
> Option-Types, that means there may be a limited domain that doesn't meet the
> requirement that there is a network entity (e.g. controller or network manager
> or similar) which has direct connection to every node, is my understanding
> correct or not?

...FB: The operator will have a way to control the nodes in the limited domain. I'm not sure what a "direct connection" is - but if you refer to "direct connection" as a means that allows the operator to access and configure the device, then yes, there would be a "direct connection". 

> What is required from my perspective for draft-ietf-ippm-ioam-conf-state, is a
> crisp definition of the target deployment environment draft-ietf-ippm-ioam-
> conf-state targets and the personas acting in that deployment environment. In a
> limited/controlled domain, we can assume that the operator knows the
> configuration of the nodes. The problem definition of draft-ietf-ippm-ioam-
> conf-state seems to be that "someone" who does not know the configuration of
> the nodes in the deployment environment wants to retrieve the configuration of
> the nodes.
> XM>>> Let me give a simplified example of the target deployment
> XM>>> environment and the personas acting in that deployment
> XM>>> environment. Assuming that the limited domain is an IPv6 domain,
> XM>>> specifically an enterprise campus using physical connections
> XM>>> between devices (as described in section 4 of
> XM>>> draft-ietf-ippm-ioam-data-17), the operator knows the
> XM>>> configuration of every node within the enterprise campus, however
> XM>>> there is no a network entity (e.g. controller or network manager
> XM>>> or similar) that knows the configuration of every node. In order
> XM>>> to deploy IOAM Pre-allocated Tracing, the operator instructs the
> XM>>> IOAM encapsulating node to add IOAM header into data packets
> XM>>> meeting a rule of DA+SA+Flow Label, through Command Line or SNMP
> XM>>> or NETCONF or similar, the IOAM encapsulating node firstly does
> XM>>> traceroute using DA+SA+Flow Label and secondly sends ICMPv6 Echo
> XM>>> Request to every node discovered by traceroute, the IOAM transit
> XM>>> and decapsulating nodes respond with ICMPv6 Echo Reply, and then
> XM>>> the IOA
>  M encapsulating node knows what the IOAM header looks like, e.g. including
> how many node data lists and how much space allocated for each node data list.
> Is this example clear or not?

...FB: Thanks for the example. Unfortunately I don't really follow. If you assume a limited domain - which you do - then the operator has access to all the nodes in his network. The operator knows the configuration of the nodes, including the IOAM configuration of the nodes. Or in other terms, the operator would configure encap/decap/transit nodes according the objectives. Why would the operator would want to use ICMP time exceeded and echo replies, given that the operator knows the network and the configuration of the nodes? 

Cheers, Frank

> Cheers, Frank
> > then I have
> > a hard time trying to understand why four types of IOAM are needed in
> > section 4 of draft-ietf-ippm-ioam-deployment
> > deployment#section-4). IMHO only one type of IOAM is necessary and
> > enough, that's Direct Export IOAM described in section 4.4, would you
> > please explain elaborately?
> >
> > Best Regards,
> > Xiao Min
> > 主 题 ：RE: Re:[ippm] Progressing draft-ietf-ippm-ioam-conf-state Hi Xiao
> > Min, per
> > https://datatracker.ietf.org/doc/html/draft-ietf-ippm-ioam-data-
> > 17#section-4, IOAM is focused on "limited domains" as defined in
> > [RFC8799], which is per RFC 8799 the same as what is also referred to
> > as a "controlled environment". Personally I understand "Controlled
> > environment" as an environment where an operator does have control
> > over the nodes in the network and their configuration, i.e., the
> > operator has control over his nodes, knows what network elements he is
> > dealing with and what the config of the nodes is. How this control is
> > implemented - whether there is one or several "controllers" or
> > "network managers" or similar - is an implementation detail IMHO.
> > Cheers, Frank
> > be used by the IOAM encapsulating node to discover these IOAM
> > capabilities information."
> > > I know you're the primary author of both IOAM-Data and
> > > IOAM-Deployment documents, so I have a fundamental question to you:
> > > Is the above statement correct or not?
> > > all, on both the mailing list and the IOAM-Deployment document, then
> > > I suggest IPPM WG to abandon draft- ietf-ippm-ioam-conf-state, that
> > > The two main points I mentioned in the last WG meeting were about
> > > (a) alignment with draft-ietf-ippm-ioam-yang and (b) enhancements to
> > > the security section, reflecting that the protocol you are defining
> > > is a network management protocol, and needs to be secured as such.
> > > More specifically:
> > > model for config information for IOAM. That model can then be
> > > rendered into different transports (be it JSON, XML, or yet another
> > > format - to then be carried over a e.g. ICMP in your case). Right
> > > now draft-ietf-ippm-ioam-conf-state heads down a path of defining a
> > > new set of management objects - many are similar to what
> > > which is more comprehensively defined by the ioam-filter grouping.
> > > You define specific fields to describe the timestamp format used by
> > > a node, which is information that should be  described as part of
> > > the ioam-info container - and which is currently missing in
> > > draft-ietf-ippm-ioam-
> > yang; point well taken :-). It is interesting to note that
> > draft-ietf-ippm-ioam- conf-state does not even reference draft-ietf- ippm-
> ioam-yang.
> > > draft-ietf-ippm-ioam- yang. Let's make sure that this model covers
> > > all the information required to properly manage IOAM. Then
> > > environment that does not have a central network management station
> > > / controller in place. Or in other terms, you seem to target a
> > > deployment, which isn't a limited domain per RFC 8799. In that case,
> > authentication and encryption between the nodes (i.e., MUST use ICMP
> > with AH and ESP..).
> > > Guide" in section 4: Could you shed a bit more light on how you
> > > expect things to work - and what a target deployment environment
> > > would look like? It seems that you assume that you don't know the
> > > network nor the destination addresses in your
> > > network: Do you expect that you would do regular "ICMP echo sweeps"
> > > in your network? Do you expect that, while doing the expanding ring
> > > (BTW - Note that due to the upcoming holiday season, replies might
> > > be
> > delayed).
> > > > Objects to make them applicable to ICMPv6 extensions, another one
> > > > is on IOAM Proof-of- Transit Capabilities Object to make it
> > > > aligned with the updated IOAM-Data document.

---

From: <[email-masked]>
Date: Thu, 30 Dec 2021 10:17:53 +0800 (CST)

Hi Frank,

Please see inline...

Best Regards,
Xiao Min
------------------原始邮件------------------
日 期 ：2021年12月29日 18:42
> nodes in the limited domain and their configuration, is my understanding correct
..FB: The operator has control over the nodes of the limited domain - i.e., the operator is the entity that owns the control of the network. Operators use a variety of tools and methods to operationalize the control. From what I know, there is no crisp definition of how the control is operationalized - and TBH, it would be quite challenging to create a comprehensive list, which would include all types of approaches to configure and operate devices.
XM-2>>> To be more specific, in a limited domain, except for human being, is there must be a single network entity which has control over all the nodes and their configuration?
> or similar) which has direct connection to every node, is my understanding
..FB: The operator will have a way to control the nodes in the limited domain. I'm not sure what a "direct connection" is - but if you refer to "direct connection" as a means that allows the operator to access and configure the device, then yes, there would be a "direct connection".
XM-2>>> To be more specific, in a limited domain, except for human being, is there must be a single network entity which has one-hop IP connection to every node?
> Is this example clear or not?
..FB: Thanks for the example. Unfortunately I don't really follow. If you assume a limited domain - which you do - then the operator has access to all the nodes in his network. The operator knows the configuration of the nodes, including the IOAM configuration of the nodes. Or in other terms, the operator would configure encap/decap/transit nodes according the objectives. Why would the operator would want to use ICMP time exceeded and echo replies, given that the operator knows the network and the configuration of the nodes?
XM-2>>> There are mainly two reasons. The first one is that the operator might not know the transport path of data packet, draft-ietf-ippm-ioam-conf-state provides an *on-path* IOAM capabilities discovery mechanism; The second one is that the operator doesn't want to provision too much parameters for one feature, my take away from talking with some operators is that they like any mechanism that can reduce their burden of manual configuration. In a limited domain, the operator can even provision routing table at every node, getting rid of IGP and BGP, however the operator rarely does that way.
Cheers, Frank

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 30 Dec 2021 10:31:16 +0000

Hi Xiao Min,

Please see inline...

> Sent: Thursday, 30 December 2021 03:18
> > Min, Please see inline ("...FB")
> > > deployment must include a centralized controller or network manager
> > > or similar,
> > ..FB: Let me restate what I said below: The current IOAM drafts assume
> > that IOAM is deployed in a limited domain (RFC8799), also called
> > controlled domain, i.e., a deployment where an operator does have
> > control over the nodes in the network and their configuration. IMHO
> > the use of the term "controller" does not help the discussion, because
> > "controller" means different things to different people. The key
> > aspect is that in a limited/controlled domain, the operator has control over the
> nodes in the limited domain and their configuration.
> > XM>>> In a limited domain, the operator has control over the nodes in
> > XM>>> the
> > limited domain and their configuration, doesn't mean there must be a
> > network entity (e.g. controller or network manager or similar) which
> > has control over the nodes in the limited domain and their
> > configuration, is my understanding correct or not?
> ..FB: The operator has control over the nodes of the limited domain - i.e., the
> operator is the entity that owns the control of the network. Operators use a
> variety of tools and methods to operationalize the control. From what I know,
> there is no crisp definition of how the control is operationalized - and TBH, it
> would be quite challenging to create a comprehensive list, which would include
> all types of approaches to configure and operate devices.
> XM-2>>> To be more specific, in a limited domain, except for human being, is
> there must be a single network entity which has control over all the nodes and
> their configuration?

...FB2: For the definition of a limited domain, it is best to simply refer to RFC 8799 :-).
The main point is that a limited or controlled domain is one where you *know* your nodes and their config.
I don't think that RFC 8799 lists a requirement for a single network entity that has control over all the nodes - and TBH, I don't think that it matters for the current discussion. 

> > Wrt/ DEX: On top of the ability to control the nodes and their
> > configuration, DEX requires you to have a solution in place which
> > allows you to retrieve, ingest, process, and correlate per-packet IOAM
> > data from every node in the limited domain. This requirement to
> > retrieve, ingest, process, and correlate per-packet IOAM data from
> > every node in the limited domain is specific to the DEX IOAM-
> > Option-Type and not required by the other IOAM-Option-Types. For
> > example, the two IOAM Trace-Option-Types automatically correlate the
> > information from the different nodes by assembling the info in the node-list
> array while the packet traverses the network.
> > XM>>> This requirement to retrieve, ingest, process, and correlate
> > XM>>> per-packet
> > IOAM data from every node in the limited domain is not required by
> > some IOAM- Option-Types, that means there may be a limited domain that
> > doesn't meet the requirement that there is a network entity (e.g.
> > controller or network manager or similar) which has direct connection
> > to every node, is my understanding correct or not?
> ..FB: The operator will have a way to control the nodes in the limited domain.
> I'm not sure what a "direct connection" is - but if you refer to "direct
> connection" as a means that allows the operator to access and configure the
> device, then yes, there would be a "direct connection".
> XM-2>>> To be more specific, in a limited domain, except for human being, is
> there must be a single network entity which has one-hop IP connection to every
> node?

...FB2: See above.

> > What is required from my perspective for
> > draft-ietf-ippm-ioam-conf-state, is a crisp definition of the target
> > deployment environment draft-ietf-ippm-ioam- conf-state targets and
> > the personas acting in that deployment environment. In a
> > configuration of the nodes. The problem definition of
> > draft-ietf-ippm-ioam- conf-state seems to be that "someone" who does
> > not know the configuration of the nodes in the deployment environment
> wants to retrieve the configuration of the nodes.
> > XM>>> configuration of every node within the enterprise campus,
> > XM>>> however there is no a network entity (e.g. controller or network
> > XM>>> manager or similar) that knows the configuration of every node.
> > XM>>> In order to deploy IOAM Pre-allocated Tracing, the operator
> > XM>>> instructs the IOAM encapsulating node to add IOAM header into
> > XM>>> data packets meeting a rule of DA+SA+Flow Label, through Command
> > XM>>> Line or SNMP or NETCONF or similar, the IOAM encapsulating node
> > XM>>> firstly does traceroute using DA+SA+Flow Label and secondly
> > XM>>> sends ICMPv6 Echo Request to every node discovered by
> > XM>>> traceroute, the IOAM transit and decapsulating nodes respond
> > XM>>> with ICMPv6 Echo Reply, and then the IOA
> >  M encapsulating node knows what the IOAM header looks like, e.g.
> > including how many node data lists and how much space allocated for each
> node data list.
> ..FB: Thanks for the example. Unfortunately I don't really follow. If you assume
> a limited domain - which you do - then the operator has access to all the nodes
> in his network. The operator knows the configuration of the nodes, including the
> IOAM configuration of the nodes. Or in other terms, the operator would
> configure encap/decap/transit nodes according the objectives. Why would the
> operator would want to use ICMP time exceeded and echo replies, given that
> the operator knows the network and the configuration of the nodes?
> XM-2>>> There are mainly two reasons. The first one is that the operator might
> not know the transport path of data packet, draft-ietf-ippm-ioam-conf-state
> provides an *on-path* IOAM capabilities discovery mechanism; The second one
> is that the operator doesn't want to provision too much parameters for one
> feature, my take away from talking with some operators is that they like any
> mechanism that can reduce their burden of manual configuration. In a limited
> domain, the operator can even provision routing table at every node, getting rid
> of IGP and BGP, however the operator rarely does that way.

...FB2: TBH, I'm not sure what you mean with "on-path" IOAM capability discovery. 
Section 4 of your draft proposes to use an ICMP sweep to discover the nodes in the network. On top of that, a discovered node would expose a portion of its IOAM configuration as part of the response to the ICMP sweep.  Do you call the fact that you're doing an expanding ring search "on-path"? With draft-ietf-ippm-ioam-conf-state, you still wouldn't know which acl/interface combinations IOAM is applied to, nor would you know which timestamp format a transit node uses. I.e., you're only discovering a subset of the IOAM capabilities - for what reason? Why wouldn't one discover all the capabilities - which would lead to the afore asked alignment with draft-ietf-ippm-ioam-yang?
In addition, could you elaborate how your approach would reduce the amount of "manual configuration" - compared to approaches that leverage draft-ietf-ippm-ioam-yang?

All these questions point back to the original question of "what is the deployment environment that you have in mind?" and what is the associated problem that you're trying to solve. You seem to target a deployment, where the operator does not know all the nodes in the network, nor their configuration - but where at the same time, ICMP sweeps are feasible, and you have security associations available to all the nodes (to the nodes that you don't know), so that you can protect ICMP payload with ESP and AH. 
It would be good if you could add a paragraph to the draft that answers those questions clearly, ideally with a set of examples. 

Cheers, Frank

> Cheers, Frank
> > Cheers, Frank
> > > then I have
> > > a hard time trying to understand why four types of IOAM are needed
> > > in section 4 of draft-ietf-ippm-ioam-deployment
> > > enough, that's Direct Export IOAM described in section 4.4, would
> > > you please explain elaborately?
> > >
> > > Best Regards,
> > > Xiao Min
> > > 主 题 ：RE: Re:[ippm] Progressing draft-ietf-ippm-ioam-conf-state Hi
> > > Xiao Min, per
> > > [RFC8799], which is per RFC 8799 the same as what is also referred
> > > to as a "controlled environment". Personally I understand
> > > "Controlled environment" as an environment where an operator does
> > > have control over the nodes in the network and their configuration,
> > > i.e., the operator has control over his nodes, knows what network
> > > elements he is dealing with and what the config of the nodes is. How
> > > this control is implemented - whether there is one or several
> > > > Considering that your holiday season is coming, before digging
> > > > into technical details, I suggest that we discuss the IOAM
> > > > deployment environment
> > > first.
> > > > draft-ietf-ippm-ioam-conf- state-02, in the introduction section
> > > > it says "A centralized controller which owns the enabled IOAM
> > > > IOAM capabilities infomation from the centralized controller,
> > > > using, for example, NETCONF/YANG, PCEP, or BGP.  In the IOAM
> > > > deployment scenario where there is no centralized controller,
> > > > NETCONF/YANG or IGP may
> > > > scenario where there is no centralized controller" doesn't exist
> > > > at all, on both the mailing list and the IOAM-Deployment document,
> > > > then I suggest IPPM WG to abandon draft-
> > > > 主 题 ：RE: [ippm] Progressing draft-ietf-ippm-ioam-conf-state Hi
> > > > Xiao Min, Thanks for posting draft-ietf-ippm-ioam-conf-state-02. I
> > > > read through the updated version and looked at the diff to the 01
> > > > version
> > > > (https://www.ietf.org/rfcdiff?url1=draft-ietf-ippm-ioam-conf-state
> > > > - 01&url2=draft-ietf-ippm-ioam-conf-state-02). Different from what
> > > > you state below, I don't see any of my comments reflected.
> > > > (a) alignment with draft-ietf-ippm-ioam-yang and (b) enhancements
> > > > to the security section, reflecting that the protocol you are
> > > > defining is a network management protocol, and needs to be secured as
> such.
> > > > More specifically:
> > > > rendered into different transports (be it JSON, XML, or yet
> > > > another format - to then be carried over a e.g. ICMP in your
> > > > case). Right now draft-ietf-ippm-ioam-conf-state heads down a path
> > > > of defining a new set of management objects - many are similar to
> > > > what draft-ietf-ippm-ioam-yang already defines, some they are less
> > > > defined. You define ingress interface fields, which is
> > > > information, which is more comprehensively defined by the ioam-filter
> grouping.
> > > > You define specific fields to describe the timestamp format used
> > > > by a node, which is information that should be  described as part
> > > > of the ioam-info container - and which is currently missing in
> > > > draft-ietf-ippm-ioam-
> > > > draft-ietf-ippm-ioam-conf-state would be solely focused on
> > > > defining how that YANG module (from draft-ietf-ippm-ioam-yang)
> > > > would be rendered
> > > > https://datatracker.ietf.org/doc/html/rfc6241#section-9. As part
> > > > of the discussion, it would be good to see an explanation where
> > > > you'd expect  draft- ietf-ippm-ioam-conf-state to be used. Given
> > > > the discussion in the Introduction section, you seem to assume an
> > > > environment that does not have a central network management
> > > > station / controller in place. Or in other terms, you seem to
> > > > target a deployment, which isn't a limited domain per RFC 8799. In
> > > > that case, I would expect that we have normative language that
> > > > requires the use of strong
> > > > In addition to the above, I struggle to understand the
> > > > "Operational Guide" in section 4: Could you shed a bit more light
> > > > on how you expect things to work - and what a target deployment
> > > > environment would look like? It seems that you assume that you
> > > > don't know the network nor the destination addresses in your
> > > > in your network? Do you expect that, while doing the expanding
> > > > ring search, an ICMP time exceeded message would also carry the
> > > > "IOAM Capabilities Response Container Header", so that the
> > > > capabilities container would not only be carried in the echo reply
> > > > message but also in the
> > > time exceeded message?
> > > > Thanks, Frank
> > > > > There are mainly two changes, one is on IOAM Tracing
> > > > > Capabilities Objects to make them applicable to ICMPv6
> > > > > extensions, another one is on IOAM Proof-of- Transit
> > > > > Capabilities Object to make it aligned with the updated IOAM-Data
> document.

---

From: "Chengli (Cheng Li)" <[email-masked]>
Date: Fri, 31 Dec 2021 02:19:18 +0000

Hi xiao Mi and Frank,

"""
"""

All the questions come from this. If IOAM is performed in a limited domain, then the controller or the operator can know the capabilities of nodes. Why the operator can establish IPSec connections among nodes but has no idea of the IOAM capabilities? If this is for a node to detect the capabilities of nodes on a path from outside a domain, then this is another story, and I guess this is not the case of IOAM. 

Respect,
Cheng

-----Original Message-----
From: ippm [mailto:[email-masked]] On Behalf Of Frank Brockners (fbrockne)
Sent: Thursday, December 30, 2021 6:31 PM
To: [email-masked]
Cc: [email-masked]
Subject: Re: [ippm] Progressing draft-ietf-ippm-ioam-conf-state

Hi Xiao Min,

Please see inline...

> Min, Please see inline...
> > Xiao Min, Please see inline ("...FB")
> > > however what I requested is a clear conclusion, on both the 
> > > mailing list and the draft-ietf-ippm- ioam-deployment.
> > > deployment must include a centralized controller or network 
> > > manager or similar,
> > ..FB: Let me restate what I said below: The current IOAM drafts 
> > assume that IOAM is deployed in a limited domain (RFC8799), also 
> > called controlled domain, i.e., a deployment where an operator does 
> > have control over the nodes in the network and their configuration. 
> > IMHO the use of the term "controller" does not help the discussion, 
> > because "controller" means different things to different people. The 
> > key aspect is that in a limited/controlled domain, the operator has 
> > control over the
> > XM>>> In a limited domain, the operator has control over the nodes 
> > XM>>> in the
> ..FB: The operator has control over the nodes of the limited domain - 
> i.e., the operator is the entity that owns the control of the network. 
> Operators use a variety of tools and methods to operationalize the 
> control. From what I know, there is no crisp definition of how the 
> control is operationalized - and TBH, it would be quite challenging to 
> create a comprehensive list, which would include all types of approaches to configure and operate devices.
> XM-2>>> To be more specific, in a limited domain, except for human 
> XM-2>>> being, is
> there must be a single network entity which has control over all the 
> nodes and their configuration?

> > allows you to retrieve, ingest, process, and correlate per-packet 
> > IOAM data from every node in the limited domain. This requirement to 
> > some IOAM- Option-Types, that means there may be a limited domain 
> > that doesn't meet the requirement that there is a network entity (e.g.
> > controller or network manager or similar) which has direct 
> > connection to every node, is my understanding correct or not?
> I'm not sure what a "direct connection" is - but if you refer to 
> "direct connection" as a means that allows the operator to access and 
> configure the device, then yes, there would be a "direct connection".
> XM-2>>> To be more specific, in a limited domain, except for human 
> XM-2>>> being, is
> there must be a single network entity which has one-hop IP connection 
> to every node?

...FB2: See above.

> > XM>>> environment. Assuming that the limited domain is an IPv6 
> > XM>>> domain, specifically an enterprise campus using physical 
> > XM>>> connections between devices (as described in section 4 of 
> > XM>>> however there is no a network entity (e.g. controller or 
> > XM>>> network manager or similar) that knows the configuration of every node.
> > XM>>> data packets meeting a rule of DA+SA+Flow Label, through 
> > XM>>> Command Line or SNMP or NETCONF or similar, the IOAM 
> > XM>>> encapsulating node firstly does traceroute using DA+SA+Flow 
> > XM>>> Label and secondly sends ICMPv6 Echo Request to every node 
> > XM>>> discovered by traceroute, the IOAM transit and decapsulating 
> > XM>>> nodes respond with ICMPv6 Echo Reply, and then the IOA
> ..FB: Thanks for the example. Unfortunately I don't really follow. If 
> you assume a limited domain - which you do - then the operator has 
> access to all the nodes in his network. The operator knows the 
> configuration of the nodes, including the IOAM configuration of the 
> nodes. Or in other terms, the operator would configure 
> encap/decap/transit nodes according the objectives. Why would the 
> operator would want to use ICMP time exceeded and echo replies, given that the operator knows the network and the configuration of the nodes?
> XM-2>>> There are mainly two reasons. The first one is that the 
> XM-2>>> operator might
> not know the transport path of data packet, 
> draft-ietf-ippm-ioam-conf-state provides an *on-path* IOAM 
> capabilities discovery mechanism; The second one is that the operator 
> doesn't want to provision too much parameters for one feature, my take 
> away from talking with some operators is that they like any mechanism 
> that can reduce their burden of manual configuration. In a limited 
> domain, the operator can even provision routing table at every node, getting rid of IGP and BGP, however the operator rarely does that way.

Cheers, Frank

> > > have control over the nodes in the network and their 
> > > configuration, i.e., the operator has control over his nodes, 
> > > knows what network elements he is dealing with and what the config 
> > > of the nodes is. How this control is implemented - whether there 
> > > is one or several "controllers" or "network managers" or similar - 
> > > is an implementation detail
> IMHO.
> > > Cheers, Frank
> > > > deployments.  The IOAM encapsulating node can discover the 
> > > > enabled IOAM capabilities infomation from the centralized 
> > > > controller, using, for example, NETCONF/YANG, PCEP, or BGP.  In 
> > > > the IOAM deployment scenario where there is no centralized 
> > > > controller, NETCONF/YANG or IGP may
> > > > at all, on both the mailing list and the IOAM-Deployment 
> > > > document, then I suggest IPPM WG to abandon draft- 
> > > > Xiao Min, Thanks for posting draft-ietf-ippm-ioam-conf-state-02. 
> > > > I read through the updated version and looked at the diff to the 
> > > > 01 version 
> > > > - 01&url2=draft-ietf-ippm-ioam-conf-state-02). Different from 
> > > > what you state below, I don't see any of my comments reflected.
> > > > (a) alignment with draft-ietf-ippm-ioam-yang and (b) 
> > > > enhancements to the security section, reflecting that the 
> > > > protocol you are defining is a network management protocol, and 
> > > > needs to be secured as
> such.
> > > > More specifically:
> > > > draft-ietf- ippm-ioam-yang. We should have a single, 
> > > > comprehensive model for config information for IOAM. That model 
> > > > can then be rendered into different transports (be it JSON, XML, 
> > > > or yet another format - to then be carried over a e.g. ICMP in 
> > > > your case). Right now draft-ietf-ippm-ioam-conf-state heads down 
> > > > a path of defining a new set of management objects - many are 
> > > > similar to what draft-ietf-ippm-ioam-yang already defines, some 
> > > > they are less comprehensive, some hint at additional information 
> > > > that draft-ietf-ippm-ioam-yang does not cover yet: E.g.,  you 
> > > > define a "Pre-allocated Tracing Capabilities Object" where 
> > > > by a node, which is information that should be  described as 
> > > > part of the ioam-info container - and which is currently missing 
> > > > in
> > > > draft-ietf-ippm-ioam-
> > > draft-ietf-ippm-ioam- conf-state does not even reference 
> > > draft-ietf-
> > > ippm-
> > ioam-yang.
> > > > From my perspective we should have one single data model for 
> > > > IOAM configuration - and that is the YANG module defined in
> > > > draft-ietf-ippm-ioam- yang. Let's make sure that this model 
> > > > covers all the information required to properly manage IOAM. 
> > > > Then draft-ietf-ippm-ioam-conf-state would be solely focused on 
> > > > the discussion in the Introduction section, you seem to assume 
> > > > an environment that does not have a central network management 
> > > > target a deployment, which isn't a limited domain per RFC 8799. 
> > > > In that case, I would expect that we have normative language 
> > > > that requires the use of strong
> > > authentication and encryption between the nodes (i.e., MUST use 
> > > ICMP with AH and ESP..).
> > > > "Operational Guide" in section 4: Could you shed a bit more 
> > > > light on how you expect things to work - and what a target 
> > > > deployment environment would look like? It seems that you assume 
> > > > that you don't know the network nor the destination addresses in 
> > > > your
> > > > capabilities container would not only be carried in the echo 
> > > > reply message but also in the
> > > time exceeded message?
> > > > Thanks, Frank
> > > > (BTW - Note that due to the upcoming holiday season, replies 
> > > > might be
> > > delayed).
> > > > > Also note that I've had an offline discussion with wg chairs 
> > > > > and Frank Brockners, my conclusion is that Frank's concern 
> > > > > raised at IETF 112 has been addressed. If that's not the case, please speak up.

---

From: <[email-masked]>
Date: Fri, 31 Dec 2021 16:30:14 +0800 (CST)

Hi Frank,

Please see inline ("XM-3>>>").
Happy New Year to you and all IPPM folks!

Best Regards,
Xiao Min
------------------原始邮件------------------
日 期 ：2021年12月30日 18:33
> there must be a single network entity which has control over all the nodes and
..FB2: For the definition of a limited domain, it is best to simply refer to RFC 8799 :-).
The main point is that a limited or controlled domain is one where you *know* your nodes and their config.
I don't think that RFC 8799 lists a requirement for a single network entity that has control over all the nodes - and TBH, I don't think that it matters for the current discussion.
XM-3>>> Thanks for your confirmation. The reason why I regarded it as a fundamental question is that if I got the counter confirmation, then I would suggest the IPPM WG to abandon draft-ietf-ippm-ioam-conf-state which triggered this discussion thread.
> there must be a single network entity which has one-hop IP connection to every
..FB2: See above.
> of IGP and BGP, however the operator rarely does that way.
..FB2: TBH, I'm not sure what you mean with "on-path" IOAM capability discovery.
XM-3>>> The IOAM encapsulating node discovers IOAM capabilities of the nodes that are on the transport path of data packet to which IOAM header is added, not all nodes within the limited domain.

Section 4 of your draft proposes to use an ICMP sweep to discover the nodes in the network. On top of that, a discovered node would expose a portion of its IOAM configuration as part of the response to the ICMP sweep.  Do you call the fact that you're doing an expanding ring search "on-path"? With draft-ietf-ippm-ioam-conf-state, you still wouldn't know which acl/interface combinations IOAM is applied to, nor would you know which timestamp format a transit node uses. I.e., you're only discovering a subset of the IOAM capabilities - for what reason? Why wouldn't one discover all the capabilities - which would lead to the afore asked alignment with draft-ietf-ippm-ioam-yang?
XM-3>>> I guess you misinterpret section 4 of draft-ietf-ippm-ioam-conf-state, let me explain a bit more here. The mechanism defined in draft-ietf-ippm-ioam-conf-state is applied after the IOAM encapsulating node knows the ACL IOAM is applied to, not before that. Here a somewhat complicated thing is that, after the IOAM encapsulating node knows the ACL IOAM is applied to, its behavior differs between ICMPv6 and LSP Ping. If it's IPv6 domain, the IOAM encapsulating node does firsly traceroute and secondly IOAM capabilites discovery. If it's MPLS domain, the IOAM encapsulating node does traceroute and IOAM capabilities discovery synchronously. Furthermore, there is possibility that the IOAM encapsulating node has known the nodes on the path (e.g. by SR Policy), then the traceroute process can be omitted. With that said, you'll find draft-ietf-ippm-ioam-conf-state has nothing to do with draft-ietf-ippm-ioam-yang, they're orthogonal. There are some RFCs that define mechanisms of capabili
 ties discovery or OAM configuration, such as RFC 9088/9089 and RFC 7759, they don't refer to any YANG Models.

In addition, could you elaborate how your approach would reduce the amount of "manual configuration" - compared to approaches that leverage draft-ietf-ippm-ioam-yang?
XM-3>>> In the example I provided in my previous email, assuming the operator knows the transport path of data packet IOAM header is added to, the operator can input the parameters such as the number of node data lists and the bitmap of IOAM-Trace-Type manually, however draft-ietf-ippm-ioam-conf-state can help the operator to omit the mentioned manual configuration.
All these questions point back to the original question of "what is the deployment environment that you have in mind?" and what is the associated problem that you're trying to solve. You seem to target a deployment, where the operator does not know all the nodes in the network, nor their configuration - but where at the same time, ICMP sweeps are feasible, and you have security associations available to all the nodes (to the nodes that you don't know), so that you can protect ICMP payload with ESP and AH.
It would be good if you could add a paragraph to the draft that answers those questions clearly, ideally with a set of examples.
XM-3>>> Please see above for the detailed explanation and let me know which part you think appropriate to be added to the draft.
Cheers, Frank