Subject: [ippm] co-design the post card telemetry

From: Tianran Zhou <[email-masked]>
Date: Thu, 7 Mar 2019 08:12:13 +0000

A forked thread.
As many folks have interest about this.
Let’s discuss and co-design the post card telemetry. Here is our draft.
https://tools.ietf.org/html/draft-song-ippm-postcard-based-telemetry-01.txt

Please review and comment.

Thanks,
Tianran

From: ippm [mailto:[email-masked]] On Behalf Of Tianran Zhou
Sent: Thursday, March 07, 2019 3:12 PM
To: Shwetha Bhandari (shwethab) <[email-masked]>; Greg Mirsky <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [ippm] [IPPM] Status and plans for draft-ietf-ippm-ioam-data

Hi Shwetha,

On your mentioned proposal, we specially deal with the post card telemetry.
https://tools.ietf.org/html/draft-song-ippm-postcard-based-telemetry-01.txt

We want to work as a separate document. Can we find a way to complement the existing ioam-data draft, while not put everything within one document?

Best,
Tianran

From: ippm [mailto:[email-masked]] On Behalf Of Shwetha Bhandari (shwethab)
Sent: Thursday, March 07, 2019 12:10 PM
To: Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] [IPPM] Status and plans for draft-ietf-ippm-ioam-data

Hi Greg,

How is it out of scope?

  *   Immediate export of the IOAM data at any node

This has come up in multiple discussions when IOAM was presented in earlier meetings.
This solves a problem in some of the protocols and deployments where IOAM is encapsulated and there is not enough space to carry entire trace or issues such as ECMP hashing due to L4 header located deeper in the packet when IOAM trace is large.
Do look at this proposal too:
https://tools.ietf.org/html/draft-song-ippm-postcard-based-telemetry-01.txt<https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-song-ippm-postcard-based-telemetry-01.txt&data=02%7C01%7Cgbarak%40mellanox.com%7C8971a1455508481ee4f608d6a26080f5%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636874932692388048&sdata=10a8fra97ltja4KK%2FDni5nVNbwu9hp7yrHQUrlIGW54%3D&reserved=0>
Given IOAM data draft is not finalized this addition helps in improving the data draft that becomes the basis for IOAM encapsulation work. It also helps in deployment of IOAM with multiple encapsulation and in diverse networks.
If you see a technical issue with the proposed flag and its use please do share.

  *    A flag bit in trace option to indicate "active OAM"  ...
Please refer to - https://mailarchive.ietf.org/arch/msg/ippm/I6d96i67zmgj9_kE0E4TewfwNrc on how this solves the problem of now being able to detect if the IOAM data is collected for a crafted probe or a packet returned due to Loopback flag

Thanks,
Shwetha

From: Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Date: Thursday, March 7, 2019 at 9:06 AM
To: Shwetha bhandari <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [IPPM] Status and plans for draft-ietf-ippm-ioam-data

Hi Shwetha,
thank you for your prompt response. I think that the proposed addition of:

  *   Immediate export of the IOAM data at any node ...
  *    A flag bit in trace option to indicate "active OAM"  ...
is outside the scope of the document as was adopted by the WG.

Regards,
Greg

On Wed, Mar 6, 2019 at 5:08 PM Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>> wrote:
Hi Greg,

> I encourage the editors to present for discussion by WG before adding definitions of new functions to the WG document.
We are proposing these changes as part of addressing comments and hence this discussion, we have not published the -05 version yet.

> I don't see how these new behaviors fit in the scope of this document.
Can you be specific..Which of the change(s) are not in the scope?
Thanks,
Shwetha

From: Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Date: Thursday, March 7, 2019 at 3:56 AM
To: Shwetha bhandari <[email-masked]<mailto:[email-masked]>>
Cc: "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>, "[email-masked]<mailto:[email-masked]>" <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] [IPPM] Status and plans for draft-ietf-ippm-ioam-data

Hi Shwetha,
you've said, "In addition to this, based on some of the face to face and side meetings at IETF we are introducing the following ..."
I encourage the editors to present for discussion by WG before adding definitions of new functions to the WG document. From the brief description you've shared, I don't see how these new behaviors fit in the scope of this document.

Regards,
Greg

On Thu, Feb 28, 2019 at 4:20 AM Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>> wrote:

Hi Adrian, All,

Here is a summary of comments received for draft-ietf-ippm-ioam-data-04 and corresponding github issues the authors are maintaining to address them in -05:

Reviewer

Brief description of the comment

Link to github tickets and discussion

1

Tianran Zhou<[email-masked]<mailto:[email-masked]>>

queue depth and buffer occupancy<https://mailarchive.ietf.org/arch/msg/ippm/IpKxrhvQCL9H7Jxl6q0rjx5VJRc>

https://github.com/inband-oam/ietf/issues/99

2

Robin <[email-masked]<mailto:[email-masked]>>

Drop reference to Segment routing examples in the introduction <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/100

3

Robin <[email-masked]<mailto:[email-masked]>>

"While, for example, the IOAM node identifier (Node-ID) does not have to be unique in a deployment , the combination of Node-ID and Namespace-ID will always be unique.  " Why assume that the node id will not be unique? It is only to increase complexity.

https://github.com/inband-oam/ietf/issues/101

4

Robin <[email-masked]<mailto:[email-masked]>>

RFC2119  language<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/102

5

Robin <[email-masked]<mailto:[email-masked]>>

IOAM-Trace-Type is still being referred to as 16 bit identifier instead of 24bit<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/103

6

Robin <[email-masked]<mailto:[email-masked]>>

  IOAM-Trace-Type Bit 12-22 must be zero - why not simply ignore it<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/104

7

Robin <[email-masked]<mailto:[email-masked]>>

The first element of the node data list (node data list [0]) contains the last node of the path while the last node data of the node data list (node data list[n]) contains the first node data of the path traced.<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/105

8

Robin <[email-masked]<mailto:[email-masked]>>

Since some customized data maybe encapsulated in the IOAM header, maybe it is not always aligned with 4-octet or 8-octet. Is it necessary to introduce some padding mechanisms? <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/106

9

Robin <[email-masked]<mailto:[email-masked]>>

Though it is necessary to save space it is a little strange for 3-octet node-id comparing popular 4-octet and 16-octet node id in other protocols. The 3-octet node id remove the possibility of reuse and consistency. It may introduce the unnecessary confu<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>sion

https://github.com/inband-oam/ietf/issues/107

10

Robin <[email-masked]<mailto:[email-masked]>>

The queue length is not enough since there may be multiple queues for a specific outgoing interface.  If there is H-QoS, it will be more complex.<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/108

11

Robin <[email-masked]<mailto:[email-masked]>>

units of the buffer occupancy bytes or percentage<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/109

12

Robin <[email-masked]<mailto:[email-masked]>>

Fix IOAM-Trace-type length in the examples<https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>

https://github.com/inband-oam/ietf/issues/110

13

John Lemon <[email-masked]<mailto:[email-masked]>>

Packets that are looped back do not have an indicator

https://github.com/inband-oam/ietf/pull/95/commits/320841ee187139a1bfe9cf4b42de05fcc1629cd9

14

Tal Mizrahi <[email-masked]<mailto:[email-masked]>>

Clarify Encapsulating vs. Decapsulating node description

https://github.com/inband-oam/ietf/pull/90/commits/69e6eb496abb6ad3702710f24377a729315336b4

15

Haoyu song <[email-masked]<mailto:[email-masked]>>

Use a Data Template ID to replace the IOAM Trace Type

https://github.com/inband-oam/ietf/issues/111

16

Haoyu song <[email-masked]<mailto:[email-masked]>>

Carry a Sequence Number (SN) field in the IOAM trace option header

https://github.com/inband-oam/ietf/issues/112

In addition to this, based on some of the face to face and side meetings at IETF we are introducing the following:

- Immediate export of the IOAM data at any node by setting an IOAM flag in the trace option to indicate this:

Adding immediate export flag: tell the network node to immediately export IOAM requested data to the collector. By that, it can help solve problems such as insufficient allocated space in the packets' headers or large IOAM header leading to inefficient lookup of data e.g. L4 header that follows.

Proposed update: https://github.com/inband-oam/ietf/pull/96

- A flag bit in trace option to indicate "active OAM", for example

probes and cloned packets.

When "active OAM" is indicated, at the IOAM decapsulating node the packet

is discarded rather than forwarded, after retrieval of the IOAM metadata.

Proposed update: https://github.com/inband-oam/ietf/pull/97

The authors have discussed and added notes/comments in github for each issue listed above and welcome the workgroup to comment there, or we can continue discussing it here.

Thanks,

Shwetha

"Adrian Farrel" <[email-masked]<mailto:[email-masked]>> Sat, 23 February 2019 10:27 UTC

Hi,

I wonder whether the authors can supply some information on the status and

plans for this draft.

It seems to me that there was a deal of discussion after the last version

(2018-10-20) in Bangkok and on the mailing list. Some of the mailing list

threads resulted in agreement, some had discussions that started but don't

appear to have reached a conclusion, and some never even got an answer.

I know we are all very busy, and I don't expect immediate responses, but

with 12 front-page authors it should be possible to find someone to respond

to the questions and propose resolution to the points raised. Or should we

add a thirteenth author who has a little more time?

Thanks,

Adrian

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: Mickey Spiegel <[email-masked]>
Date: Thu, 7 Mar 2019 02:15:03 -0800

> Please review and comment.

I have several concerns with the approach proposed here.

The primary concern is simple:
Why does postcard need to be defined or specified at the IETF?
How does PBT-M as described in this draft improve interoperability?

I am one of the implementers of a postcard implementation that works
on native IPv4 packets. Postcard has several advantages when it comes
to interoperability, primarily that data plane transports are unaffected,
and that different implementations need not do the same thing. As long
as the collector is aware of the different implementations, it can
normalize what it receives from different nodes and put the pieces
together. There is no strong need for uniformity of implementations.

It is possible to ease the task of the collector by defining a uniform
export format, from switches to the collector. See
draft-spiegel-ippm-ioam-rawexport-01 for one such proposal.

Other concerns:

1) PBT-M proposes that generation of postcards be triggered by
   packet marking. While that is one possibility, that is not the only
   or the most desirable method. This can be broken down into
   two parts.

   a) Specification of the flows of interest can be done through the
      management and control planes using ACLs at each node,
      rather than just at the head node.

   b) Within a flow of interest, there is a question whether some
      or all packets should trigger report generation. Note that the
      head node does not know what is going to happen to the
      packet as it traverses the network. The promise of data
      plane telemetry solutions is network visibility, continuously
      observing what is going on in the network. Intelligent
      implementations can monitor all packets in the flows of
      interest and use event detection to determine when to export
      telemetry metadata to the collectors. These event detection
      algorithms are best left to implementations, both to encourage
      experimentation and differentiation in a rapidly evolving field.

   For both aspects, I fail to see what is to be gained by specifying
   any of this in a draft.

2) The proposed use of the NSH OAM bit is not consistent with
   the existing definition. This bit was meant for other purposes.
   It is not possible to redefine the bit's meaning for purposes of
   postcard.

3) The proposed use of bits set aside for alternate marking is
   not consistent with the existing definitions, which look for
   changes in the value of the bit.
   Is this draft proposing that the value 1 means generate
   postcards?
   Is this draft proposing to generate postcards only when the
   value of the bit changes?

4) The "flow ID" and "packet ID" are not clearly defined, and are
   unlikely to work. Flows will need to be identified across
   varying transports with multiple levels of headers. In IOAM
   raw export, the suggestion is to export packet header fields so
   that the collector can identify flows at any granularity it desires.

5) At a high level PBT-I and IOAM immediate export are similar.
   I have concerns with some of the details:

   a) As mentioned in 3 above, I don't know what a "flow ID" is
      and I am skeptical that any proposal would be flexible
      enough and useful enough. It is simpler to just export
      the packet header fields and let the collector deal with it.

   b) We will have a separate discussion on trace vector vs
      template ID.

   c) I do not see PBT-I / immediate export as a total
      replacement for IOAM. There are pros and cons to each,
      and I expect to see deployments of each. With this in
      mind, unless there is a strong reason, I believe that the
      formats should be aligned as much as possible. I don't
      see a strong reason to deviate from the IOAM format in
      this draft.

Mickey

>
> Thanks,
>
> Tianran
>
>
>
> *From:* ippm [mailto:[email-masked]] *On Behalf Of *Tianran Zhou
> *Sent:* Thursday, March 07, 2019 3:12 PM
> *To:* Shwetha Bhandari (shwethab) <[email-masked]>; Greg Mirsky <
> [email-masked]>
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [ippm] [IPPM] Status and plans for
> draft-ietf-ippm-ioam-data
>
>
>
> Hi Shwetha,
>
>
>
> We want to work as a separate document. Can we find a way to complement
> the existing ioam-data draft, while not put everything within one document?
>
>
>
> Best,
>
> Tianran
>
>
>
> *From:* ippm [mailto:[email-masked] <[email-masked]>] *On
> Behalf Of *Shwetha Bhandari (shwethab)
> *Sent:* Thursday, March 07, 2019 12:10 PM
> *To:* Greg Mirsky <[email-masked]>
> *Cc:* [email-masked]; [email-masked]
> *Subject:* Re: [ippm] [IPPM] Status and plans for
> draft-ietf-ippm-ioam-data
>
>
>
> Hi Greg,
>
>
>
> How is it out of scope?
>
>
>
>    - Immediate export of the IOAM data at any node
>
>
>
> This solves a problem in some of the protocols and deployments where IOAM
> is encapsulated and there is not enough space to carry entire trace or
> issues such as ECMP hashing due to L4 header located deeper in the packet
> when IOAM trace is large.
>
> <https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftools.ietf.org%2Fhtml%2Fdraft-song-ippm-postcard-based-telemetry-01.txt&data=02%7C01%7Cgbarak%40mellanox.com%7C8971a1455508481ee4f608d6a26080f5%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636874932692388048&sdata=10a8fra97ltja4KK%2FDni5nVNbwu9hp7yrHQUrlIGW54%3D&reserved=0>
>
> Given IOAM data draft is not finalized this addition helps in improving
> the data draft that becomes the basis for IOAM encapsulation work. It also
> helps in deployment of IOAM with multiple encapsulation and in diverse
> networks.
>
>    -  A flag bit in trace option to indicate "active OAM"  ...
>
> Please refer to -
> https://mailarchive.ietf.org/arch/msg/ippm/I6d96i67zmgj9_kE0E4TewfwNrc on
> how this solves the problem of now being able to detect if the IOAM data is
> collected for a crafted probe or a packet returned due to Loopback flag
>
>
>
> Thanks,
>
> Shwetha
>
>
>
> *From: *Greg Mirsky <[email-masked]>
> *Date: *Thursday, March 7, 2019 at 9:06 AM
> *To: *Shwetha bhandari <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>, "[email-masked]" <
> [email-masked]>, "[email-masked]" <
> [email-masked]>
> *Subject: *Re: [ippm] [IPPM] Status and plans for
> draft-ietf-ippm-ioam-data
>
>
>
> Hi Shwetha,
>
>    - Immediate export of the IOAM data at any node ...
>    -  A flag bit in trace option to indicate "active OAM"  ...
>
> On Wed, Mar 6, 2019 at 5:08 PM Shwetha Bhandari (shwethab) <
> [email-masked]> wrote:
>
> Hi Greg,
>
>
>
> > I encourage the editors to present for discussion by WG before adding
> definitions of new functions to the WG document.
>
> We are proposing these changes as part of addressing comments and hence
> this discussion, we have not published the -05 version yet.
>
>
>
> *From: *Greg Mirsky <[email-masked]>
> *Date: *Thursday, March 7, 2019 at 3:56 AM
> *To: *Shwetha bhandari <[email-masked]>
> *Cc: *"[email-masked]" <[email-masked]>, "[email-masked]" <
> [email-masked]>, "[email-masked]" <
> [email-masked]>
> *Subject: *Re: [ippm] [IPPM] Status and plans for
> draft-ietf-ippm-ioam-data
>
>
>
> Hi Shwetha,
>
> you've said, "In addition to this, based on some of the face to face and
> side meetings at IETF we are introducing the following ..."
>
> I encourage the editors to present for discussion by WG before adding
> definitions of new functions to the WG document. From the brief description
> you've shared, I don't see how these new behaviors fit in the scope of this
> document.
>
>
>
> Regards,
>
> Greg
>
>
>
> On Thu, Feb 28, 2019 at 4:20 AM Shwetha Bhandari (shwethab) <
> [email-masked]> wrote:
>
> Hi Adrian, All,
>
>
>
> Here is a summary of comments received for draft-ietf-ippm-ioam-data-04
> and corresponding github issues the authors are maintaining to address them
> in -05:
>
>
>
>
>
> *Reviewer*
>
> *Brief description of the comment*
>
> *Link to github tickets and discussion*
>
> 1
>
> Tianran Zhou<[email-masked]>
>
> *queue depth and buffer occupancy
> <https://mailarchive.ietf.org/arch/msg/ippm/IpKxrhvQCL9H7Jxl6q0rjx5VJRc>*
>
> Robin <[email-masked]>
>
> *Drop reference to Segment routing examples in the introduction
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *"While, for example, the IOAM node identifier (Node-ID) does not have to
> be unique in a deployment , the combination of Node-ID and Namespace-ID
> will always be unique.  " Why assume that the node id will not be unique?
> It is only to increase complexity.*
>
> Robin <[email-masked]>
>
> *RFC2119  language
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *IOAM-Trace-Type is still being referred to as 16 bit identifier instead
> of 24bit
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *  IOAM-Trace-Type Bit 12-22 must be zero - why not simply ignore it
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *The first element of the node data list (node data list [0]) contains the
> last node of the path while the last node data of the node data list (node
> data list[n]) contains the first node data of the path traced.
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *Since some customized data maybe encapsulated in the IOAM header, maybe
> it is not always aligned with 4-octet or 8-octet. Is it necessary to
> introduce some padding mechanisms?
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *Though it is necessary to save space it is a little strange for 3-octet
> node-id comparing popular 4-octet and 16-octet node id in other protocols.
> The 3-octet node id remove the possibility of reuse and consistency. It may
> introduce the unnecessary confu
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>sion*
>
> Robin <[email-masked]>
>
> *The queue length is not enough since there may be multiple queues for a
> specific outgoing interface.  If there is H-QoS, it will be more complex.
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *units of the buffer occupancy bytes or percentage
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> Robin <[email-masked]>
>
> *Fix IOAM-Trace-type length in the examples
> <https://mailarchive.ietf.org/arch/msg/ippm/v4a75IXTi8GKfDl_Ddm_5WN6gxs>*
>
> John Lemon <[email-masked]>
>
> Tal Mizrahi <[email-masked]>
>
> Haoyu song <[email-masked]>
>
> Haoyu song <[email-masked]>
>
> In addition to this, based on some of the face to face and side meetings
> at IETF we are introducing the following:
>
> Adding immediate export flag: tell the network node to immediately export
> IOAM requested data to the collector. By that, it can help solve problems
> such as insufficient allocated space in the packets' headers or large IOAM
> header leading to inefficient lookup of data e.g. L4 header that follows.
>
> The authors have discussed and added notes/comments in github for each
> issue listed above and welcome the workgroup to comment there, or we can
> continue discussing it here.
>
>
>
> Thanks,
>
> Shwetha
>
>
>
>
>
> "Adrian Farrel" <[email-masked]> Sat, 23 February 2019 10:27 UTC
>
> Hi,
>
>
>
> [email-masked]
> [email-masked]

---

From: Haoyu song <[email-masked]>
Date: Thu, 7 Mar 2019 18:24:55 +0000

In my opinion, PBT can work as an independent mode similar to the E2E and Trace modes in IOAM.

The current IOAM draft is updated to add a single bit to indicate sending a postcard, but it’s not sufficient to cover the case.
The collector would need  flowid and sequence number to correlate the postcards for the same packet. In IOAM, only E2E mode has an optional sequence number. But E2E mode is not supposed to send postcard along the path anyway. In Trace mode, there’s no sequence number. I have suggested in another email to add the sequence number in IOAM header.
But to make things clean, perhaps another mode is the right way to go.

Regards,
Haoyu

From: ippm [mailto:[email-masked]] On Behalf Of Tianran Zhou
Sent: Thursday, March 07, 2019 12:12 AM
To: Tianran Zhou <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Greg Mirsky <[email-masked]>
Cc: [email-masked]
Subject: [ippm] co-design the post card telemetry

A forked thread.

Thanks,
Tianran

To: Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Greg Mirsky <[email-masked]<mailto:[email-masked]>>

Hi Shwetha,

Best,
Tianran

Hi Greg,

How is it out of scope?

Thanks,
Shwetha

Hi Shwetha,

Regards,
Greg

Hi Shwetha,

Regards,
Greg

Hi Adrian, All,

Reviewer

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Thanks,

Shwetha

Hi,

plans for this draft.

Thanks,

Adrian

---

From: Haoyu song <[email-masked]>
Date: Thu, 7 Mar 2019 18:58:46 +0000

Hi Mickey,

Thank you for your detailed review and questions.
Please see my response inline.

Best,
Haoyu

From: ippm [mailto:[email-masked]] On Behalf Of Mickey Spiegel
Sent: Thursday, March 07, 2019 2:15 AM
To: Tianran Zhou <[email-masked]>
Cc: [email-masked]
Subject: Re: [ippm] co-design the post card telemetry

On Thu, Mar 7, 2019 at 12:12 AM Tianran Zhou <[email-masked]<mailto:[email-masked]>> wrote:
A forked thread.
As many folks have interest about this.
Let’s discuss and co-design the post card telemetry. Here is our draft.
https://tools.ietf.org/html/draft-song-ippm-postcard-based-telemetry-01.txt

HS> We didn’t specify the postcard format yet, but a standard format will help the interoperability.
RAWEXPORT is a possible way. We also think it’s possible to use specific IPFIX template for data export,
with the advantage to ease the workload of data collector.

HS> The purpose of PTB-M is simple: avoid augmenting user packets with any new header at all.
The mark is only a trigger for postcard generation. As you described below, each node is totally free
to decide what to export.

It is possible to ease the task of the collector by defining a uniform
export format, from switches to the collector. See
draft-spiegel-ippm-ioam-rawexport-01 for one such proposal..

Other concerns:

HS> Section 2.5 in the draft describes this option exactly.

HS> For the implementation considerations, please see our iFIT solution framework draft.
https://datatracker.ietf.org/doc/draft-song-opsawg-ifit-framework/
In this draft, we only describe the postcard mechanism but don’t cover the implementation issues.

HS> Fair point. Will examine it.

HS> Fair point. Given that’s not standardized yet, we just explore all the possibilities. We have considered
to consolidate the alternate marking scheme with other in band telemetry techniques.

HS> These are used to identify individual packets, as long as it’s not changed once it’s assigned.
FlowID can be assigned by a central controller to guarantee the uniqueness. If it’s automatically generated,
the large size also make the conflict probability very low.
Exporting the header may incur two much bandwidth overhead and processing load, especially when
there are other outer layer tunnel headers.

HS> For the same reason mentioned above. Also, as I mentioned in an earlier email, sequence number is necessary to identify
a unique packet.

HS> We clearly state that PBT is complement to IOAM. See our IFIT framework draft for more details. Our current thinking is
to make PBT a mode of IOAM.

Mickey

Thanks,
Tianran

From: ippm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Tianran Zhou
Sent: Thursday, March 07, 2019 3:12 PM
To: Shwetha Bhandari (shwethab) <[email-masked]<mailto:shwethab@cisco..com>>; Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:ippm@ietf..org>
Subject: Re: [ippm] [IPPM] Status and plans for draft-ietf-ippm-ioam-data

Hi Shwetha,

Best,
Tianran

Hi Greg,

How is it out of scope?

Thanks,
Shwetha

Hi Shwetha,

Regards,
Greg

Hi Shwetha,

Regards,
Greg

Hi Adrian, All,

Reviewer

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Thanks,

Shwetha

Hi,

plans for this draft.

appear to have reached a conclusion, and some never even got an answer..

Thanks,

Adrian

---

From: "Frank Brockners (fbrockne)" <[email-masked]>
Date: Thu, 7 Mar 2019 19:36:22 +0000

Good point. I’ve added this consideration to the discussion on the related pull request https://github.com/inband-oam/ietf/pull/96

Frank

From: ippm <[email-masked]> On Behalf Of Haoyu song
Sent: Donnerstag, 7. März 2019 19:25
To: Tianran Zhou <[email-masked]>; Tianran Zhou <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Greg Mirsky <[email-masked]>
Cc: [email-masked]
Subject: Re: [ippm] co-design the post card telemetry

Regards,
Haoyu

From: ippm [mailto:[email-masked]] On Behalf Of Tianran Zhou
Sent: Thursday, March 07, 2019 12:12 AM
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: [ippm] co-design the post card telemetry

A forked thread.

Thanks,
Tianran

Hi Shwetha,

Best,
Tianran

Hi Greg,

How is it out of scope?

Thanks,
Shwetha

Hi Shwetha,

Regards,
Greg

Hi Shwetha,

Regards,
Greg

Hi Adrian, All,

Reviewer

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Thanks,

Shwetha

Hi,

plans for this draft.

Thanks,

Adrian

---

From: Mickey Spiegel <[email-masked]>
Date: Thu, 7 Mar 2019 13:54:26 -0800

Please see replies inline.

On Thu, Mar 7, 2019 at 10:59 AM Haoyu song <[email-masked]> wrote:

> *From:* ippm [mailto:[email-masked]] *On Behalf Of *Mickey Spiegel
> *Sent:* Thursday, March 07, 2019 2:15 AM
> *To:* Tianran Zhou <[email-masked]>
> *Cc:* [email-masked]
> *Subject:* Re: [ippm] co-design the post card telemetry
>
>
>
>
>

What standard format are you referring to?

I don't see any proposed format in the draft for PBT-M. The
whole point is that there is nothing to format in the data plane.

I see a proposed format for PBT-I. That is something we can
discuss, as I have commented below.

I don't see any proposed export format in the draft.

> to decide what to export.

If data plane packets are not augmented with any new header, and the
trigger is only one of myriad methods to trigger postcard generation,
why does anything need to be specified in a draft or RFC?

> HS> Section 2.5 in the draft describes this option exactly.

IMO this is an obvious method that is not sufficient justification for
a new draft or RFC.

> In this draft, we only describe the postcard mechanism but don’t cover the

I do not believe such "implementation considerations" should be defined
in any draft or RFC. They are not an interoperability concern. Let
different implementations figure out different ways to do things. This is a
good area for research and experimentation.

> FlowID can be assigned by a central controller to guarantee the
> uniqueness. If it’s automatically generated,
>

It is still not clear me to what definition of "flow ID" you are aiming
for. Let me suggest two options and point out the issues with each
in the context of PBT-M:

   1. User assigned. In case of PBT-M, this would require some sort
      of common packet matching technique to be applied at every hop,
      such as an ACL. My main issue with this is the mismatch in
      granularity. An ACL that decides which traffic should be monitored
      for possible postcard generation is likely to be coarser than what
      the collector is interested in such as a 5-tuple. Trying to program
      every 5-tuple at every hop would be a scalability concern.

   2. Algorithmically derived. In case of PBT-M, this would require a
      common algorithm to be defined that would be carried out at
      each hop. Different hardware implementations tend to compute
      hashes in different ways. I believe it would be difficult to arrive at
      a common algorithm that many implementations are willing to
      support.

> HS> For the same reason mentioned above. Also, as I mentioned in an
> earlier email, sequence number is necessary to identify
>
> a unique packet.
>
>
>
> HS> We clearly state that PBT is complement to IOAM. See our IFIT
> framework draft for more details. Our current thinking is
>

I agree that anything that is specified should be in the context of
IOAM. However, as described above, I don't see any need to
define PBT-M in an internet draft or RFC, just PBT-I / immediate
export. Whether this PBT-I "mode" justifies another IOAM type
value, I don't see the rationale at this point but this can be a
subject for further discussion.

Mickey

> *To:* Shwetha Bhandari (shwethab) <[email-masked]
> <shwethab@cisco..com>>; Greg Mirsky <[email-masked]>
> *Cc:* [email-masked]; [email-masked] <ippm@ietf..org>

---

From: Haoyu song <[email-masked]>
Date: Thu, 7 Mar 2019 22:15:50 +0000

Inline.

From: Mickey Spiegel [mailto:[email-masked]]
Sent: Thursday, March 07, 2019 1:54 PM
To: Haoyu song <[email-masked]>
Cc: Tianran Zhou <[email-masked]>; [email-masked]
Subject: Re: [ippm] co-design the post card telemetry

On Thu, Mar 7, 2019 at 10:59 AM Haoyu song <[email-masked]<mailto:[email-masked]>> wrote:
Hi Mickey,

Best,
Haoyu

From: ippm [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Mickey Spiegel
Sent: Thursday, March 07, 2019 2:15 AM
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] co-design the post card telemetry

[HS] It’s not included in this draft and I’m not sure it’s in the scope or should be discussed in another draft. If you think
this is the right place, we can add it in a later revision.

[HS]I think  we need to decide how to trigger the postcard. Also the current draft means to enumerate all
the possibilities. We may end up with choosing one best solution.

Other concerns:

[HS] Again, this is discussed here for completeness. I also mentioned several drawbacks
of this approach. Finally we might drop this option.

[HS] Well, that’s for information only. Not for standard track for sure.

[HS] You summarize the two options correctly. But for user assigned flowid, it’s configured from controller and
only the head node of the domain need to be configured. And the flow can have any granularity.  The algorithmic option
is similar. Only the head node needs to generate a flowid and there should be no requirement for what hashing algorithm
is used.

[HS] The above discussion may explain some of our thoughts better. Let’s see
what other people think about this issue.

Mickey

Mickey

Thanks,
Tianran

Hi Shwetha,

Best,
Tianran

Hi Greg,

How is it out of scope?

Thanks,
Shwetha

Hi Shwetha,

Regards,
Greg

Hi Shwetha,

Regards,
Greg

Hi Adrian, All,

Reviewer

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Thanks,

Shwetha

Hi,

plans for this draft.

Thanks,

Adrian

---

From: Mickey Spiegel <[email-masked]>
Date: Thu, 7 Mar 2019 14:27:30 -0800

Replying to one comment specifically inline.

On Thu, Mar 7, 2019 at 2:15 PM Haoyu song <[email-masked]> wrote:

<snip>

> [HS] You summarize the two options correctly. But for user assigned
> flowid, it’s configured from controller and
>
> only the head node of the domain need to be configured. And the flow can
> have any granularity.  The algorithmic option
>
> is similar. Only the head node needs to generate a flowid and there should
> be no requirement for what hashing algorithm
>
> is used.
>

For PBT-I, you could do such a thing though I wonder about the
value.

For PBT-M, the flow ID value cannot be signaled in the data plane,
so either it needs to be derived at each hop, or it is only exported
from the head node and not from other nodes, which implies that
packet headers must be exported from those other nodes.

There is at least one collector implementation that is capable of
correlating reports from multiple hops, without the need for a
"flow ID" of any sort.

Mickey

---

From: Haoyu song <[email-masked]>
Date: Thu, 7 Mar 2019 23:31:39 +0000

Reply inline

Sent: Thursday, March 07, 2019 2:28 PM

On Thu, Mar 7, 2019 at 2:15 PM Haoyu song <[email-masked]<mailto:[email-masked]>> wrote:

<snip>

[HS] Our current thought is that for PBT-M, there is no flow ID involved because we have no way to carry it and we don’t want to
burden each node to acquire it. That’s why we also list the postcard correlation as  a key challenge for PBT-M and suggest several
way to mitigate it in the draft. But PBT-M still has its merits because it doesn’t change the packet size. Also , for PBT-M, using packet marking is
superior to using ACL as stated in the draft. So we should give each option more thorough considerations.

Note that 5 tuple can only help to correlate the flow, but not enough to identify a specific packet. This is critical postcard correlation.
Also, getting 5 tuple may need extra work in network nodes. Alternatively, you can simply send the first N bytes of the packet as identifier.
The drawback of this option is: either you waste some bandwidth because N is larger than necessary or you miss the header due to the
outer tunnel headers. Also, this creates a heavy post-processing workload for the data collectors.

Mickey

---

From: Haoyu song <[email-masked]>
Date: Fri, 8 Mar 2019 00:01:36 +0000

I hope we can work together to consolidate the cases and make a unified solution.

Haoyu

From: Frank Brockners (fbrockne) [mailto:[email-masked]]
Sent: Thursday, March 07, 2019 11:36 AM
To: Haoyu song <[email-masked]>; Tianran Zhou <[email-masked]>; Tianran Zhou <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Greg Mirsky <[email-masked]>; Barak Gafni <[email-masked]>
Cc: [email-masked]
Subject: RE: co-design the post card telemetry

Frank

From: ippm <[email-masked]<mailto:[email-masked]>> On Behalf Of Haoyu song
Sent: Donnerstag, 7. März 2019 19:25
To: Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Tianran Zhou <[email-masked]<mailto:[email-masked]>>; Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Greg Mirsky <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>
Subject: Re: [ippm] co-design the post card telemetry

Regards,
Haoyu

A forked thread.

Thanks,
Tianran

Hi Shwetha,

Best,
Tianran

Hi Greg,

How is it out of scope?

Thanks,
Shwetha

Hi Shwetha,

Regards,
Greg

Hi Shwetha,

Regards,
Greg

Hi Adrian, All,

Reviewer

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

Thanks,

Shwetha

Hi,

plans for this draft.

Thanks,

Adrian

---

From: Tianran Zhou <[email-masked]>
Date: Fri, 8 Mar 2019 07:50:12 +0000

Hi Mickey,

What do you mean by “Trying to program every 5-tuple at every hop would be a scalability concern.”
Do you mean to export with 5-tuple information? If so maybe only simple IP network works.
For tunnels, MPLS, etc, it’s hard to extract the 5-tuple.
And for different encapsulations, the position of the 5-tuple varies. So you may need to configure/program every device to adapt to the scenario. That would be huge operational work.
However, an Flow-ID only need to be generated and attached at the egress node. And the position is fixed, easy to identify.

Best,
Tianran

Sent: Friday, March 08, 2019 6:28 AM

<snip>

Mickey