Subject: [Ioam] Questions to IOAM proposed charter

From: Linda Dunbar <[email-masked]>
Date: Mon, 30 Jan 2017 23:10:52 +0000

Frank:
The first sentence of the proposed charter says:
In-situ OAM (IOAM) provides real-time telemetry of individual data packets and flows. It is based on telemetry information which is embedded along within data packets

So the key differences between In-situ OAM and BFD is that BFD uses synthetic data packets (i.e. dedicated OAM packets), whereas In-situ OAM are encoded into the user data packets. Correct?

Does it mean "Telemetry information that doesn't have user payload" is out of the scope?

Thanks, Linda Dunbar

From: "Frank Brockners (fbrockne)" <fbrockne at cisco.com<mailto:[email-masked]>>
________________________________

Following the discussions and general support of in-situ OAM on the OPSAWG mailing list and the guidance from Ops and Routing ADs, we are to propose a dedicated working group to progress normative work on in-situ OAM. Below you'll find a first draft of a working group charter for an in-situ OAM WG that several of us drafted. The scope is intentionally crisp and focused, so that we can built on the past discussions and work as well as make quick progress.

We appreciate your thoughts and comments.

Regards, Frank, Tal, Shwetha, Carlos

IOAM working group charter
In-situ OAM provides real-time telemetry of individual data packets and flows. It is based on telemetry information which is embedded along within data packets. The IOAM WG intends to define data formats and associated procedures for in-situ OAM, including mechanisms for capturing path and path-traversal related information as well as procedures to employ, configure, trigger, and export the embedded telemetry information.

The IOAM WG will define the following in-situ OAM components:

* Data-records for in-situ OAM to cover path-tracing and path verification information (node-ids, ingress/egress interfaces), timestamps, transit-delay, transit jitter, sequence numbers, application-defined metadata. In-situ OAM data-records are defined using an in-situ OAM namespace.
* Procedures to add, update, remove, retrieve and export data-records for in-situ OAM to live traffic and active probing.  In case of live traffic a classifier to select subset of live traffic for addition, update, removal and retrieval of in-situ OAM data-records. In case of active probing procedure to return the in-situ OAM data records to the source of the probe.
*  Scope of in-situ OAM operation. In-Situ OAM operations are defined for a specific operational domain. In-situ OAM data-records are added and removed at domain boundaries and updated by nodes within the in-situ OAM domain. Procedure to deal with various challenges in packet forwarding and error handling such as ECMP processing, path MTU and ICMP message handling when in-situ OAM is enabled in an in-situ OAM domain.
*  Data-records for in-situ OAM are to be defined in a way that makes them independent from the transport protocol used. Data-records for in-situ OAM will need to be embedded into transport protocols such as IPv4, IPv6, VXLAN-GPE, LISP, NSH, SRv6, Geneve.
* Procedures and data-records optimized for software dataplane and hardware dataplane implementations of in-situ OAM.
* In-situ OAM to support layering. If several transport protocols (e.g. in case of tunneling) are stacked on top of each other, in-situ OAM data-records could be present in every transport protocol layer.

The IOAM working group intends to work on and publish:
* Definition of the data-type formats used in in-situ OAM and namespaces for in-situ OAM.
* Definition of procedures that in-situ OAM enabled nodes will perform on data traffic that carries in-situ OAM information (e.g., introducing, removing, processing, modifying, and exporting the telemetry information from the associated data packets).
In-situ OAM data records could be embedded into a variety of transports. The transports are expected to be defined in the respective working group(s) in consultation with the IOAM working group documents.

Milestones

April 2018 - submit data format and associated procedures document to IESG.
March 2018 - WGLC for data format and associated procedures document.
April 2017 - working group adoption of data format and associated procedures document.

Huawei USA IP Technology Lab
5340 Legacy Drive,
Plano, TX 75024
Tel: +1 469-277 - 5840
Fax: +1 469 -277 - 5900

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 31 Jan 2017 10:45:07 +0000

Frank

We have other insitu approaches in flight and I hope that there is going 
to be no resultant confusion.

There are proposals in MPLS and IPPM to perform loss, delay and jitter 
measurements on live packets using packet marking. The important 
difference between the approach taken there and this approach is that we 
have imposed a constraint that the forwarding plane cannot be modified.

It would be useful to clarify that there are other ongoing IETF 
approaches to OAM on live traffic that are out of scope for this WG.

- Stewart

On 30/01/2017 23:10, Linda Dunbar wrote:
> In-situ OAM (IOAM) provides real-time telemetry of individual data 
> packets and flows. It is based on telemetry information which is 
> embedded along within data packets
>
> So the key differences between In-situ OAM and BFD is that BFD uses 
> synthetic data packets (i.e. dedicated OAM packets), whereas In-situ 
> OAM are encoded into the user data packets. Correct?
>
> Does it mean “Telemetry information that doesn’t have user payload” is 
> out of the scope?
>
> Thanks, Linda Dunbar
>
> //
>
> //
>
> /From/: "Frank Brockners (fbrockne)" <fbrockne at cisco.com 
> <mailto:[email-masked]>>
>
> ------------------------------------------------------------------------
>
> Following the discussions and general support of in-situ OAM on the 
> OPSAWG mailing list and the guidance from Ops and Routing ADs, we are 
> to propose a dedicated working group to progress normative work on 
> in-situ OAM. Below you’ll find a first draft of a working group 
> charter for an in-situ OAM WG that several of us drafted. The scope is 
> intentionally crisp and focused, so that we can built on the past 
> discussions and work as well as make quick progress.
>
> In-situ OAM provides real-time telemetry of individual data packets 
> and flows. It is based on telemetry information which is embedded 
> along within data packets. The IOAM WG intends to define data formats 
> and associated procedures for in-situ OAM, including mechanisms for 
> capturing path and path-traversal related information as well as 
> procedures to employ, configure, trigger, and export the embedded 
> telemetry information.
>
> * Data-records for in-situ OAM to cover path-tracing and path 
> verification information (node-ids, ingress/egress interfaces), 
> timestamps, transit-delay, transit jitter, sequence numbers, 
> application-defined metadata. In-situ OAM data-records are defined 
> using an in-situ OAM namespace.
>
> * Procedures to add, update, remove, retrieve and export data-records 
> for in-situ OAM to live traffic and active probing.  In case of live 
> traffic a classifier to select subset of live traffic for addition, 
> update, removal and retrieval of in-situ OAM data-records. In case of 
> active probing procedure to return the in-situ OAM data records to the 
> source of the probe.
>
> *  Scope of in-situ OAM operation. In-Situ OAM operations are defined 
> for a specific operational domain. In-situ OAM data-records are added 
> and removed at domain boundaries and updated by nodes within the 
> in-situ OAM domain. Procedure to deal with various challenges in 
> packet forwarding and error handling such as ECMP processing, path MTU 
> and ICMP message handling when in-situ OAM is enabled in an in-situ 
> OAM domain.
>
> *  Data-records for in-situ OAM are to be defined in a way that makes 
> them independent from the transport protocol used. Data-records for 
> in-situ OAM will need to be embedded into transport protocols such as 
> IPv4, IPv6, VXLAN-GPE, LISP, NSH, SRv6, Geneve.
>
> * Procedures and data-records optimized for software dataplane and 
> hardware dataplane implementations of in-situ OAM.
>
> * In-situ OAM to support layering. If several transport protocols 
> (e.g. in case of tunneling) are stacked on top of each other, in-situ 
> OAM data-records could be present in every transport protocol layer.
>
> * Definition of procedures that in-situ OAM enabled nodes will perform 
> on data traffic that carries in-situ OAM information (e.g., 
> introducing, removing, processing, modifying, and exporting the 
> telemetry information from the associated data packets).
>
> In-situ OAM data records could be embedded into a variety of 
> transports. The transports are expected to be defined in the 
> respective working group(s) in consultation with the IOAM working 
> group documents.
>
> Milestones
>
> *Huawei USA IP Technology Lab***
>
> *5340 Legacy Drive, *
>
> *Plano, TX 75024*
>
> *Tel: +1 469-277 - 5840 *
>
> *Fax: +1 469 -277 - 5900 *
>
>
>
> _______________________________________________
> Ioam mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ioam

---

From: "Alvaro Retana (aretana)" <[email-masked]>
Date: Tue, 31 Jan 2017 13:56:49 +0000

Stewart:

Good point!

While clarifying what is out of scope, we should also explicitly mention that this new-WG will also coordinate with other related WGs (list them!) as needed.

Thanks!

Alvaro.

On 1/31/17, 5:45 AM, "Ioam on behalf of Stewart Bryant" <[email-masked]<mailto:[email-masked]> on behalf of [email-masked]<mailto:[email-masked]>> wrote:

We have other insitu approaches in flight and I hope that there is going to be no resultant confusion.

There are proposals in MPLS and IPPM to perform loss, delay and jitter measurements on live packets using packet marking. The important difference between the approach taken there and this approach is that we have imposed a constraint that the forwarding plane cannot be modified.

It would be useful to clarify that there are other ongoing IETF approaches to OAM on live traffic that are out of scope for this WG.

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 31 Jan 2017 14:27:31 +0000

On 31/01/2017 13:56, Alvaro Retana (aretana) wrote:

> While clarifying what is out of scope, we should also explicitly 
> mention that this new-WG will also coordinate with other related WGs 
> (list them!) as needed.
>

Yes. 6man and MPLS being obvious examples since they will be greatly 
interested in discussing the impact on their forwarding planes.

- Stewart

---

From: Ram Krishnan <[email-masked]>
Date: Tue, 31 Jan 2017 07:08:15 -0800

Good discussion. IPPM is another relevant WG group since this work has
direct impact on performance measurement.

Thanks,
Ramki

On Tue, Jan 31, 2017 at 6:27 AM, Stewart Bryant <[email-masked]>
wrote:

> While clarifying what is out of scope, we should also explicitly mention
> that this new-WG will also coordinate with other related WGs (list them!)
> as needed.
>
>

-- 
Thanks,
Ramki

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 31 Jan 2017 10:34:16 -0500

I believe I sent this question to the requirements authors, but now that 
there is a dedicated list for the topic, it seems appropriate to ask it 
here.

The In-Situ OAM Work looks interesting.
However, it also looks like a new attempt to standardize a mechanism 
whose close relatives have failed before.

Why should we expect In-Situ OAM to work when earlier efforts such as 
record route have ended up ineffective at best?

Yours,
Joel

---

From: Stewart Bryant <[email-masked]>
Date: Tue, 31 Jan 2017 15:42:44 +0000

On 31/01/2017 15:34, Joel M. Halpern wrote:
> I believe I sent this question to the requirements authors, but now 
> that there is a dedicated list for the topic, it seems appropriate to 
> ask it here.
>

I also wonder whether applications that care that much about the detail 
of the path shouldn't be using
SR to force the path. Indeed I wonder whether SR couldn't help with the 
recoding of transit,
at least in the V6 case where each segment identifier could be replaced 
with an opaque token
of transit as the packet passed through that intermediate node.

Stewart

---

From: Linda Dunbar <[email-masked]>
Date: Tue, 31 Jan 2017 16:25:14 +0000

SFC is another relevant WG as the metadata in NSH can carry the needed telemetry and OAM information.

Linda

From: Ioam [mailto:[email-masked]] On Behalf Of Ram Krishnan
Sent: 2017年1月31日 9:08
To: Stewart Bryant <[email-masked]>
Cc: Alvaro Retana (aretana) <[email-masked]>; [email-masked]; Frank Brockners (fbrockne) <[email-masked]>
Subject: Re: [Ioam] Questions to IOAM proposed charter

Good discussion. IPPM is another relevant WG group since this work has direct impact on performance measurement.

Thanks,
Ramki

On Tue, Jan 31, 2017 at 6:27 AM, Stewart Bryant <[email-masked]<mailto:[email-masked]>> wrote:

Yes. 6man and MPLS being obvious examples since they will be greatly interested in discussing the impact on their forwarding planes.

- Stewart

_______________________________________________
Ioam mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ioam

--
Thanks,
Ramki

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 31 Jan 2017 17:48:46 +0000

Good discussion. The draft charter already mentions the need to work with other working groups, especially when it comes to defining how in-situ OAM data records would be carried. That work would naturally be done by 6man, NSH, etc. rather than the new IOAM working group (“In-situ OAM data records could be embedded into a variety of transports. The transports are expected to be defined in the respective working group(s) in consultation with the IOAM working group documents.”). And I agree it would be good to be more specific about which WGs IOAM WG would likely work with, e.g. “In-situ OAM data records could be embedded into a variety of transports. The transports are expected to be defined in the respective working group(s) like 6man, NSH, MPLS, etc. in consultation with the IOAM working group documents.”

Frank

From: Linda Dunbar [mailto:[email-masked]]
Sent: Dienstag, 31. Januar 2017 17:25
To: Ram Krishnan <[email-masked]>; Stewart Bryant <[email-masked]>
Cc: Alvaro Retana (aretana) <[email-masked]>; [email-masked]; Frank Brockners (fbrockne) <[email-masked]>
Subject: RE: [Ioam] Questions to IOAM proposed charter

Linda

From: Ioam [mailto:[email-masked]] On Behalf Of Ram Krishnan
Sent: 2017年1月31日 9:08
To: Stewart Bryant <[email-masked]<mailto:[email-masked]>>
Cc: Alvaro Retana (aretana) <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>
Subject: Re: [Ioam] Questions to IOAM proposed charter

Thanks,
Ramki

- Stewart

--
Thanks,
Ramki

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Tue, 31 Jan 2017 17:56:11 +0000

Linda,

please see inline (...FB)

From: Linda Dunbar [mailto:[email-masked]]
Sent: Dienstag, 31. Januar 2017 00:11
To: [email-masked]; Frank Brockners (fbrockne) <[email-masked]>
Subject: Questions to IOAM proposed charter

Frank:

...FB: Yes, partially. Key qualities of "in-situ OAM" is that the OAM data-records are carried within the user-traffic, that in-situ OAM applies to a specific domain, and that nodes within the domain might update the in-situ OAM data-records as the traffic traverses the network. There are several more attributes of in-situ OAM, which is why the scope statement of the proposed IOAM WG a bit long.

...FB: Given that we're still in the process of defining the scope, let me turn the question back to you: Should it be included?

Thanks, Frank

Thanks, Linda Dunbar

Milestones

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 1 Feb 2017 03:37:13 +0000

Hi, Joel,

> On Jan 31, 2017, at 10:34 AM, Joel M. Halpern <[email-masked]> wrote:
> 
> I believe I sent this question to the requirements authors, but now that there is a dedicated list for the topic, it seems appropriate to ask it here.
> 

Indeed. Thanks.

> However, it also looks like a new attempt to standardize a mechanism whose close relatives have failed before.
> 
> Why should we expect In-Situ OAM to work when earlier efforts such as record route have ended up ineffective at best?
> 

That statement and question, as phrased, seem to both overgeneralize and oversimplify.

Efforts like record route did not have the same objective as In-situ OAM does. Consequently, success / effectiveness of those different goals is measured under different and incomparable bars.

Here’s a fundamental difference: I have used first-hand Record Route in real large-scale provider network troubleshooting — but only to force a packet to be process-switched to differentiate from a fast path and debug. However, it is not hard to see how dataplane technology and silicon improved since the 1981 from RFC 791, and how that’s no longer the case necessarily.

Today’s data path can fast-switch with these features, in a way that the penalty is less than a traditional tunneling encapsulation.

Net-net, 1981’s record route seems to be not a useful comparison, certainly not anything to correlate effectiveness with.

Thanks!

— Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 1 Feb 2017 03:41:27 +0000

Hello, Stewart,

> On Jan 31, 2017, at 10:42 AM, Stewart Bryant <[email-masked]> wrote:
> 
> 
> 
> I also wonder whether applications that care that much about the detail of the path shouldn't be using
> SR to force the path.

If an application cares for learning the path of a packet or flow, measure its delay, or perform capacity planning, does *not* mean that said application would want to control or force that path.

SR is the wrong hammer here.

On the other hand, in-situ OAM can well play with SR for the cases that SR is the right path-steering hammer, to not only verify that path but also provide additional telemetry.

> Indeed I wonder whether SR couldn't help with the recoding of transit,
> at least in the V6 case where each segment identifier could be replaced with an opaque token

One of the key benefits of in-situ OAM is that it is independent of the transport — meaning, you can use it with SR, also with VxLAN, also with native v6, etc., but do not need to force the selection of the transport.

Thanks,

— Carlos.

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 1 Feb 2017 03:46:51 +0000

I agree, all those are very relevant WGs.

The proposed charter text at: “The transports are expected to be defined in the respective working group(s) in consultation with the IOAM working group documents.”

Should provide more precision or some examples based on this thread.

Thanks,

—
Carlos Pignataro, [email-masked]<mailto:[email-masked]>

“Sometimes I use big words that I do not fully understand, to make myself sound more photosynthesis."

On Jan 31, 2017, at 11:25 AM, Linda Dunbar <[email-masked]<mailto:[email-masked]>> wrote:

Linda

Thanks,
Ramki

- Stewart

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 1 Feb 2017 03:54:28 +0000

Stewart,

Please see inline.

—
Carlos Pignataro, [email-masked]

> On Jan 31, 2017, at 5:45 AM, Stewart Bryant <[email-masked]> wrote:
> 
> 
> Frank
> 

Do you mean other “active” approaches? Or other “in-situ” approaches? The former being modifying and using existing fields, the latter being embedding additional information.

> There are proposals in MPLS and IPPM to perform loss, delay and jitter measurements on live packets using packet marking.

This is a good point.

Those approaches are active OAM to provide end-to-end performance measurements. 

A good addition to the charter text can be the granularity of in-situ OAM regarding intermediate hop stamping, measurements, etc. It is slightly implied in the first paragraph under “including mechanisms for capturing path and path-traversal related information”. 

That text can be augmented with “per-hop, per-selected-hop, as well as end-to-end telemetry scope"

> The important difference between the approach taken there and this approach is that we have imposed a constraint that the forwarding plane cannot be modified.
> 

I believe the important difference is what the goal is (path record, per-hop data, verification of path, etc) and not the constraint.

> It would be useful to clarify that there are other ongoing IETF approaches to OAM on live traffic that are out of scope for this WG.

I agree that would be a useful clarification. However, let’s use more precision than “approaches to OAM on live traffic”. 

Thanks,

— Carlos.

>> Does it mean “Telemetry information that doesn’t have user payload” is out of the scope?
>>  
>> Thanks, Linda Dunbar
>>  
>>  
>> From: "Frank Brockners (fbrockne)" <fbrockne at cisco.com>
>>  
>> Following the discussions and general support of in-situ OAM on the OPSAWG mailing list and the guidance from Ops and Routing ADs, we are to propose a dedicated working group to progress normative work on in-situ OAM. Below you’ll find a first draft of a working group charter for an in-situ OAM WG that several of us drafted. The scope is intentionally crisp and focused, so that we can built on the past discussions and work as well as make quick progress.
>>

---

From: "Joel M. Halpern" <[email-masked]>
Date: Tue, 31 Jan 2017 23:31:27 -0500

Your comment about fast-path processing seems almost as over-generalized 
as my question.

Can some hardware process some potential in-situ messages in the fast 
path?  Of course.
Is there likely to be hardware that can process all possible in-situ 
messages in the fast path?  Seem unlikely.
Can existing hardware that handles some of these tunnels handle in-situ 
in the fast path?  Probably not.

Can we define a useful sdt of in-situ functions that can be processed 
efficiently, can avoid giving out inappropriate information to external 
parties, and are useful?  Probably.
But unless we are explicit about the constraints we will not achieve that.

As an example, some of the in-situ proposals have called for each device 
in the path to add information to the packet, expanding the packet. 
While it is theoretically possible to do that in a fast path, it usually 
does not work out.

Separately, you and I may have seen different things.  The usage I saw 
people wanting for record-route was to get what we eventually delivered 
in a messier fashion with traceroute.  I will agree that the need to 
slow-path the packets is a significant contributor to the failure.

Yours,
Joel

---

From: "Carlos Pignataro (cpignata)" <[email-masked]>
Date: Wed, 1 Feb 2017 04:46:57 +0000

Hi, Joel,

> On Jan 31, 2017, at 11:31 PM, Joel M. Halpern <[email-masked]> wrote:
> 
> Your comment about fast-path processing seems almost as over-generalized as my question.

Only to show that correlating potential success of in-situ OAM to that of Record Route IPv4 option was a red herring.

My point is that hardware and software can now do things not possible in 1981 — I did not mean to over-generalize, only to show one counter-example.

> 
> Can some hardware process some potential in-situ messages in the fast path?  Of course.
> Is there likely to be hardware that can process all possible in-situ messages in the fast path?  Seem unlikely.

This is a good point — that is why there are various formats for things like node data. It is not an expectation that every piece of hardware and software can process every possible element and format.

But… we are going into solutions (prompted by the question) and at this point we should stick to charter.

> Can existing hardware that handles some of these tunnels handle in-situ in the fast path?  Probably not.
> 

This is again 

> Can we define a useful sdt of in-situ functions that can be processed efficiently, can avoid giving out inappropriate information to external parties, and are useful?  Probably.

Agreed. Success will depend upon the goals and constraints. Let’s work this into the charter.

> 
> As an example, some of the in-situ proposals have called for each device in the path to add information to the packet, expanding the packet. While it is theoretically possible to do that in a fast path, it usually does not work out.
> 

One thing that would be useful is to share demos in a live meeting.

> Separately, you and I may have seen different things.  The usage I saw people wanting for record-route was to get what we eventually delivered in a messier fashion with traceroute.  I will agree that the need to slow-path the packets is a significant contributor to the failure.

We are on the same page here, Joel. 

As a first step, I recommend we codify what’s missing re- goals, constraints, and scope in the draft charter. In this context, do you have specific text recommendations?

Thanks again!

— Carlos.