Subject: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104

From: Haoyu song <[email-masked]>
Date: Tue, 26 Mar 2019 13:07:18 +0000

Dear all,

Per the side meeting discussion on IOAM last night, we decide to describe and compare the IOAM trace mode header and PBT header (updated to align with IOAM spec) to spur discussions within the WG, so we can decide if we should make PBT-I another mode of IOAM ( in addition to Incremental Trace, Pre-allocated Trace, E2E, and POT modes). If so, the new mode could be named PHP (Per Hop Postcard).

Today we have some further discussion with Barak and below reflects the discussion results. I’ll update our PBT draft as well to reflect these considerations.

Please feel free to provide your thoughts and inputs. Thank you very much!

Best regards,
Haoyu

IOAM Trace Mode (Incremental and Pre-allocated) Header format:
    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |        Namespace-ID           |NodeLen  | Flags | RemainingLen|
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |               IOAM-Trace-Type                 |  Reserved     |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+
   |                                                               |  |
   |                        node data list [0]                     |  |
   |                                                               |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  D
   |                                                               |  a
   |                        node data list [1]                     |  t
   |                                                               |  a
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   ~                             ...                               ~  S
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  p
   |                                                               |  a
   |                        node data list [n-1]                   |  c
   |                                                               |  e
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  |
   |                                                               |  |
   |                        node data list [n]                     |  |
   |                                                               |  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+<-+

PHP Header Format:

    0                   1                   2                   3

    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |        Namespace-ID           | Extended Flags  | Hop Count   |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |               IOAM-Trace-Type                 |  Reserved     |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |                     Flow ID                                   |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   |                     Sequence Number                           |

   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Key Differences between PBT-I and IOAM Trace:

1)      PBT-I: No need for “NodeLen” field; bits saved for flags

2)      PBT-I: No need for “immediate export” flag; mode identified through “mode” field before the header

3)      PBT-I: No need for “remaining length” field, since no IOAM data will be included in user packet

4)      PBT-I: No need for “Node Data List” for obvious reason.

5)      PBT-I: Have a “flow ID” field which helps to identify and correlate a flow

a.      Avoid the need to truncate the original packet header

b.      Remove the flow parsing burden of the data collector and analyzer

6)      PBT-I: Have a “sequence number” field which helps to correlate postcards for a same user packet

a.      Without it,  it’s very difficult to correctly correlate postcards (see detailed discussion about this issue on PBT-M in the PBT draft)

7)      PBT-I: Have an optional “hop count” field which helps to correlate and track the order of the postcards

a.      If unused, set to all “0”

b.      If used, start from 1, and every node increments it by 1.
Common fields

1)      Namespace-ID

2)      Flags

a.      More flag bits are available due to removing the “NodeLen” field

b.      No “immediate export” flag needed

3)      Trace Type

a.      Although we think a template ID is a better way to replace the bitmap-based trace type, ultimately different modes should be made consistent for this.

Barak suggests that Flow ID and Sequence Number can also be made optional by setting some flags, if user prefer to use other means to correlate packets. This subjects to further discussion.

---

From: Barak Gafni <[email-masked]>
Date: Tue, 26 Mar 2019 13:24:52 +0000

Haoyu,

Thank you very much for putting this suggestion, and detailed comparison.
I believe that there are multiple topics we should further discuss in the side meeting:

  1.  Whether this is an additional option of IOAM or is indicated by the I-bit
     *   One reason to have the I-bit is that it enables to move between modes as the packet traverse the network, in case of space limitations, as discussed by the WG
  2.  I do believe we should have full alignment on the header. I haven’t found a reason why to break this so far, although as stated below there is some redundancy when working in the immediate export mode
  3.  As we discussed, there are multiple ways to determine existence of the flowid and serial num, such as flag, length of the header, etc. so these should be optional
  4.  In my mind, we should agree on (1) and (2), then continue from there
Thanks again,
Barak

From: Haoyu song <[email-masked]>
Sent: Tuesday, March 26, 2019 2:07 PM
To: Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners (fbrockne) <[email-masked]>; Ramesh Sivakolundu (sramesh) <[email-masked]>; Tal Mizrahi <[email-masked]>; Barak Gafni <[email-masked]>
Cc: IETF IPPM WG <[email-masked]>
Subject: Compare IOAM Trace Mode Header and PBT Mode Header RE: [ippm] IPPM - IOAM Side Meeting @IETF 104

Dear all,

Best regards,
Haoyu

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

  1.  PBT-I: No need for “NodeLen” field; bits saved for flags
  2.  PBT-I: No need for “immediate export” flag; mode identified through “mode” field before the header
  3.  PBT-I: No need for “remaining length” field, since no IOAM data will be included in user packet
  4.  PBT-I: No need for “Node Data List” for obvious reason.
  5.  PBT-I: Have a “flow ID” field which helps to identify and correlate a flow
     *   Avoid the need to truncate the original packet header
     *   Remove the flow parsing burden of the data collector and analyzer
  6.  PBT-I: Have a “sequence number” field which helps to correlate postcards for a same user packet
     *   Without it,  it’s very difficult to correctly correlate postcards (see detailed discussion about this issue on PBT-M in the PBT draft)
  7.  PBT-I: Have an optional “hop count” field which helps to correlate and track the order of the postcards
     *   If unused, set to all “0”
     *   If used, start from 1, and every node increments it by 1.
Common fields

  1.  Namespace-ID
  2.  Flags
     *   More flag bits are available due to removing the “NodeLen” field
     *   No “immediate export” flag needed
  3.  Trace Type
     *   Although we think a template ID is a better way to replace the bitmap-based trace type, ultimately different modes should be made consistent for this.

---

From: Haoyu song <[email-masked]>
Date: Tue, 26 Mar 2019 13:36:58 +0000

Please see my response inline.

Haoyu

From: Barak Gafni [mailto:[email-masked]]
Sent: Tuesday, March 26, 2019 6:25 AM
To: Haoyu song <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners (fbrockne) <[email-masked]>; Ramesh Sivakolundu (sramesh) <[email-masked]>; Tal Mizrahi <[email-masked]>
Cc: IETF IPPM WG <[email-masked]>
Subject: RE: Compare IOAM Trace Mode Header and PBT Mode Header RE: [ippm] IPPM - IOAM Side Meeting @IETF 104

Haoyu,

  1.  Whether this is an additional option of IOAM or is indicated by the I-bit
     *   One reason to have the I-bit is that it enables to move between modes as the packet traverse the network, in case of space limitations, as discussed by the WG
[HS] I remember from yesterday’s discussion, most people agreed that, in case of space limitation for the trace mode, the node can automatically trigger an export, without needing any flag. This is a pure configuration issue.

  1.  I do believe we should have full alignment on the header. I haven’t found a reason why to break this so far, although as stated below there is some redundancy when working in the immediate export mode
[HS] If this is used as a mode, there’s really no need to make full alignment, although it may be convenient to try to align the common fields.

  1.  As we discussed, there are multiple ways to determine existence of the flowid and serial num, such as flag, length of the header, etc. so these should be optional
  2.  In my mind, we should agree on (1) and (2), then continue from there
Thanks again,
Barak

From: Haoyu song <[email-masked]<mailto:[email-masked]>>
Sent: Tuesday, March 26, 2019 2:07 PM
To: Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Ramesh Sivakolundu (sramesh) <[email-masked]<mailto:[email-masked]>>; Tal Mizrahi <[email-masked]<mailto:[email-masked]>>; Barak Gafni <[email-masked]<mailto:[email-masked]>>
Cc: IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Compare IOAM Trace Mode Header and PBT Mode Header RE: [ippm] IPPM - IOAM Side Meeting @IETF 104

Dear all,

Best regards,
Haoyu

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

1)      Namespace-ID

2)      Flags

3)      Trace Type

---

From: Barak Gafni <[email-masked]>
Date: Tue, 26 Mar 2019 13:48:22 +0000

We don’t have an agreement and need more feedback, so let’s continue the discussion in the meeting

Thanks,
Barak

From: Haoyu song <[email-masked]>
Sent: Tuesday, March 26, 2019 2:37 PM
To: Barak Gafni <[email-masked]>; Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners (fbrockne) <[email-masked]>; Ramesh Sivakolundu (sramesh) <[email-masked]>; Tal Mizrahi <[email-masked]>
Cc: IETF IPPM WG <[email-masked]>
Subject: RE: Compare IOAM Trace Mode Header and PBT Mode Header RE: [ippm] IPPM - IOAM Side Meeting @IETF 104

Haoyu

To: Haoyu song <[email-masked]<mailto:[email-masked]>>; Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Ramesh Sivakolundu (sramesh) <[email-masked]<mailto:[email-masked]>>; Tal Mizrahi <[email-masked]<mailto:[email-masked]>>

Haoyu,

Dear all,

Best regards,
Haoyu

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

  1.  Namespace-ID
  2.  Flags

---

From: Haoyu song <[email-masked]>
Date: Tue, 26 Mar 2019 13:52:39 +0000

Sure. I just mean that most attendees for yesterday’s side meeting hold such opinion for the item (1).

Best,
Haoyu

Sent: Tuesday, March 26, 2019 6:48 AM

Thanks,
Barak

To: Barak Gafni <[email-masked]<mailto:[email-masked]>>; Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Ramesh Sivakolundu (sramesh) <[email-masked]<mailto:[email-masked]>>; Tal Mizrahi <[email-masked]<mailto:[email-masked]>>

Haoyu

Haoyu,

Dear all,

Best regards,
Haoyu

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

1)      Namespace-ID

2)      Flags

3)      Trace Type

---

From: Mickey Spiegel <[email-masked]>
Date: Tue, 26 Mar 2019 15:40:21 -0700

Please see comments inline.

On Tue, Mar 26, 2019 at 6:07 AM Haoyu song <[email-masked]> wrote:

> Dear all,
>
>
>
> Per the side meeting discussion on IOAM last night, we decide to describe
> and compare the IOAM trace mode header and PBT header (updated to align
> with IOAM spec) to spur discussions within the WG, so we can decide if we
> should make PBT-I another mode of IOAM ( in addition to Incremental Trace,
> Pre-allocated Trace, E2E, and POT modes). If so, the new mode could be
> named PHP (Per Hop Postcard).
>

Thanks for putting this together. It makes it much easier to discuss
commonalities and differences, so that we can decide whether this should be
indicated by another IOAM-Type field value or a flag in one of the existing
trace types.

IMO, there is no need for all these additional three letter acronyms and
terminology such as PBT or PHP. I have also heard some objections to the
word "immediate" in "immediate export". If we go with a new IOAM-Type field
value, my suggestion would be "IOAM Trace with Direct Export Option". If we
go with a flag, my suggestion would be just "Direct Export" or "Direct
Reports". Anyway, we can decide on terminology later.

>
> Today we have some further discussion with Barak and below reflects the
> discussion results. I’ll update our PBT draft as well to reflect these
> considerations.
>
>
>
Agree that there is no need for "NodeLen" in this case.

> through “mode” field before the header
We need to decide whether to indicate this case using a new IOAM-Type field
value or a flag.

> will be included in user packet
Agree.

> 4)      PBT-I: No need for “Node Data List” for obvious reason.
Agree.

> b.      Remove the flow parsing burden of the data collector and analyzer
While a "flow ID" may seem like a good idea at first glance, IMO it is not
all that useful. Reiterating what I stated during the meeting:

   - I don't believe in central assignment of "flow ID" because the
   granularity at which you configure what you are interested in monitoring
   (e.g. ACLs) differs from the granularity that the monitoring/analytics
   systems care about (typically 5-tuple). Typically such ACLs will not have
   individual entries for every 5-tuple, they will use masks such as /24 or
   /16.
   - While "flow ID" may be useful for correlation of a particular packet
   as it traverses the network, with metadata being exported separately at
   each hop, it is not that useful for correlating different packets belonging
   to the same flow (e.g. 5-tuple).
      - If you buy the argument in the first bullet, then "flow ID" will be
      assigned algorithmically at the encapsulating node. This has many
      implications, captured in the next few bullets.
      - The "flow ID" does not remove the flow parsing burden because the
      monitoring/analytics systems just see "flow ID" as a number that at best
      assists with correlation. Flow identity, in the sense that flows are
      presented to users of the monitoring/analytics systems, is
determined from
      the packet header fields. Since it is possible for packets containing
      exported metadata to be lost, this means that every packet containing
      exported metadata should contain sufficient packet header fields to
      identify the flow, in addition to the "flow ID".
      - The encapsulating node may change during the lifetime of a flow.
      Trying to define a common "flow ID" assignment algorithm that could work
      across multiple different vendor implementations is futile, even for a
      simple encapsulation. These types of algorithms tend to be hardware
      specific. For more complex encapsulations, it would also be difficult to
      reach consensus on which fields should be looked at by the "flow ID"
      assignment algorithm. This means that the "flow ID" will change whenever
      the the encapsulating node changes, e.g. due to changes in the paths
      determined by routing protocols in the previous network.
      - The monitoring/analytics system can simplify its operation by
      always relying on packet header fields to identify the flow, rather than
      bouncing back and forth between relying on packet header fields
and relying
      on "flow ID". The latter is susceptible to corner cases that will
      significantly increase the complexity of the implementation.

6)      PBT-I: Have a “sequence number” field which helps to correlate
> postcards for a same user packet
>
> a.      Without it,  it’s very difficult to correctly correlate postcards
> (see detailed discussion about this issue on PBT-M in the PBT draft)
>
If the system carrying out the correlation knows the network topology, it
is possible to correlate without presence of a sequence number. We have an
implementation that carries out such correlation based on knowledge of the
network topology, and using the ingress and egress interface IDs present in
the telemetry metadata.

Any implementation relying on "sequence number" will have to deal with the
case where the encapsulating node changes. Either the identity of the
encapsulating node would have to be included in the packet, or some other
approach would need to be taken such as randomizing the initial sequence
number, in order to minimize the possibility of sequence number collisions.

Even if we conclude that "sequence number" is useful, we would still have
to decide whether it is necessary to put the "sequence number" directly in
this "IOAM Trace Option", or whether it is good enough to include a
separate "IOAM Edge-to-Edge Option" as is already defined in
draft-ietf-ippm-ioam-data-05.

In summary, I agree that there are some fields that are not applicable in
this case. However, given the need for 4 byte alignment, defining a new
IOAM-type field value does not save any bytes in the resulting data
packets. Moreover, the use case for the proposed new fields such as "flow
ID" and "sequence number" is not that strong. I do not yet see sufficient
justification for a new IOAM-type field value.

Mickey

7)      PBT-I: Have an optional “hop count” field which helps to correlate
> and track the order of the postcards
>
> a.      Although we think a template ID is a better way to replace the
> bitmap-based trace type, ultimately different modes should be made
> consistent for this.
>
>
>
> Barak suggests that Flow ID and Sequence Number can also be made optional
> by setting some flags, if user prefer to use other means to correlate
> packets. This subjects to further discussion.
>
>
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>

---

From: Haoyu song <[email-masked]>
Date: Wed, 27 Mar 2019 12:13:55 +0000

Hi Mickey,

From: Mickey Spiegel [mailto:[email-masked]]
Sent: Tuesday, March 26, 2019 3:40 PM
To: Haoyu song <[email-masked]>
Cc: Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners (fbrockne) <[email-masked]>; Ramesh Sivakolundu (sramesh) <[email-masked]>; Tal Mizrahi <[email-masked]>; Barak Gafni <[email-masked]>; IETF IPPM WG <[email-masked]>
Subject: Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104

On Tue, Mar 26, 2019 at 6:07 AM Haoyu song <[email-masked]<mailto:[email-masked]>> wrote:
Dear all,

Thanks for putting this together. It makes it much easier to discuss commonalities and differences, so that we can decide whether this should be indicated by another IOAM-Type field value or a flag in one of the existing trace types.

IMO, there is no need for all these additional three letter acronyms and terminology such as PBT or PHP. I have also heard some objections to the word "immediate" in "immediate export". If we go with a new IOAM-Type field value, my suggestion would be "IOAM Trace with Direct Export Option". If we go with a flag, my suggestion would be just "Direct Export" or "Direct Reports". Anyway, we can decide on terminology later.

[HS] We do need a name to describe the header and feature. I don’t see a problem here. I talked with Barack and thought about a case for using “immediate export” in the trace mode: it change the router behavior whenever there’s no room for extra trace data: If this bit is not set, no data will be added and the end result is that you will only get a partial trace (kind of the currently defined behavior); if the bit is set, then a postcard will be generate to export the current trace data so room is made to accommodate new trace data. And PHP is another mode option for which each node must generate a postcard. Now the semantics for each option is clean.

Best regards,
Haoyu

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

2)      PBT-I: No need for “immediate export” flag; mode identified through “mode” field before the header
We need to decide whether to indicate this case using a new IOAM-Type field value or a flag.

b.      Remove the flow parsing burden of the data collector and analyzer
While a "flow ID" may seem like a good idea at first glance, IMO it is not all that useful. Reiterating what I stated during the meeting:

  *   I don't believe in central assignment of "flow ID" because the granularity at which you configure what you are interested in monitoring (e.g. ACLs) differs from the granularity that the monitoring/analytics systems care about (typically 5-tuple). Typically such ACLs will not have individual entries for every 5-tuple, they will use masks such as /24 or /16.
  *   While "flow ID" may be useful for correlation of a particular packet as it traverses the network, with metadata being exported separately at each hop, it is not that useful for correlating different packets belonging to the same flow (e.g. 5-tuple).

     *   If you buy the argument in the first bullet, then "flow ID" will be assigned algorithmically at the encapsulating node. This has many implications, captured in the next few bullets.
     *   The "flow ID" does not remove the flow parsing burden because the monitoring/analytics systems just see "flow ID" as a number that at best assists with correlation. Flow identity, in the sense that flows are presented to users of the monitoring/analytics systems, is determined from the packet header fields. Since it is possible for packets containing exported metadata to be lost, this means that every packet containing exported metadata should contain sufficient packet header fields to identify the flow, in addition to the "flow ID".
     *   The encapsulating node may change during the lifetime of a flow. Trying to define a common "flow ID" assignment algorithm that could work across multiple different vendor implementations is futile, even for a simple encapsulation. These types of algorithms tend to be hardware specific. For more complex encapsulations, it would also be difficult to reach consensus on which fields should be looked at by the "flow ID" assignment algorithm. This means that the "flow ID" will change whenever the the encapsulating node changes, e.g. due to changes in the paths determined by routing protocols in the previous network.
     *   The monitoring/analytics system can simplify its operation by always relying on packet header fields to identify the flow, rather than bouncing back and forth between relying on packet header fields and relying on "flow ID". The latter is susceptible to corner cases that will significantly increase the complexity of the implementation.

a.      Without it,  it’s very difficult to correctly correlate postcards (see detailed discussion about this issue on PBT-M in the PBT draft)
If the system carrying out the correlation knows the network topology, it is possible to correlate without presence of a sequence number. We have an implementation that carries out such correlation based on knowledge of the network topology, and using the ingress and egress interface IDs present in the telemetry metadata.

[HS] The knowledge of network topology is a too strong requirement which doesn’t apply for carrier networks. So sequence number is necessary for the general case. Again, I have talked with Barak about the possibility to make both Flow ID and sequence number optional in case some user really keens to reduce the overhead and can do without them. Bust still, I believe in most cases, these two fields can make life much easier.

Any implementation relying on "sequence number" will have to deal with the case where the encapsulating node changes. Either the identity of the encapsulating node would have to be included in the packet, or some other approach would need to be taken such as randomizing the initial sequence number, in order to minimize the possibility of sequence number collisions.

[HS] I didn’t see the need to change the encapsulation nodes in my use cases. But if there is, the flowID and sequence number can change. I don’t see a big trouble here. Maybe I miss something here?

Even if we conclude that "sequence number" is useful, we would still have to decide whether it is necessary to put the "sequence number" directly in this "IOAM Trace Option", or whether it is good enough to include a separate "IOAM Edge-to-Edge Option" as is already defined in draft-ietf-ippm-ioam-data-05.

[HS] We have discussed about this. It’s improper to include E2E with Trace Option to just get the sequence number. That’s an abuse of the modes and doesn’t bring any benefit. A new mode is the right way to go.

In summary, I agree that there are some fields that are not applicable in this case. However, given the need for 4 byte alignment, defining a new IOAM-type field value does not save any bytes in the resulting data packets. Moreover, the use case for the proposed new fields such as "flow ID" and "sequence number" is not that strong. I do not yet see sufficient justification for a new IOAM-type field value.

Mickey

1)      Namespace-ID

2)      Flags

3)      Trace Type

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: Mickey Spiegel <[email-masked]>
Date: Wed, 27 Mar 2019 10:39:23 -0700

Please see responses inline.

On Wed, Mar 27, 2019 at 5:14 AM Haoyu song <[email-masked]> wrote:

> *From:* Mickey Spiegel [mailto:[email-masked]]
> *Sent:* Tuesday, March 26, 2019 3:40 PM
> *To:* Haoyu song <[email-masked]>
> *Cc:* Shwetha Bhandari (shwethab) <[email-masked]>; Frank Brockners
> (fbrockne) <[email-masked]>; Ramesh Sivakolundu (sramesh) <
> [email-masked]>; Tal Mizrahi <[email-masked]>; Barak Gafni <
> [email-masked]>; IETF IPPM WG <[email-masked]>
> *Subject:* Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header
> RE: IPPM - IOAM Side Meeting @IETF 104
>
>
>
> [HS] We do need a name to describe the header and feature. I don’t see a
> problem here. I talked with Barack and thought about a case for using
> “immediate export” in the trace mode: it change the router behavior
> whenever there’s no room for extra trace data: If this bit is not set, no
> data will be added and the end result is that you will only get a partial
> trace (kind of the currently defined behavior); if the bit is set, then a
> postcard will be generate to export the current trace data so room is made
> to accommodate new trace data. And PHP is another mode option for which
> each node must generate a postcard. Now the semantics for each option is
> clean.
>

[MS2] There is no need for a bit to allow exporting the current stack and
replacing it with a new stack. I don't see any reason to disallow such
behavior. When the MTU limit is reached, give transit nodes a choice
whether they want to export the current stack and start over, or not add
anything to the existing stack and keep progressing it. Exporting the
current stack does not harm downstream nodes in any way. It is a little
more work for the collector, but realistically any good collector
implementation will want to deal with reports of dropped packets, which
requires a similar type of correlation.

The only thing we need to cleanly indicate is when no trace metadata is to
be included in the data plane at all, either using an "immediate export"
flag or a new IOAM-Type field value.

>       presented to users of the monitoring/analytics systems, is determined from
>       bouncing back and forth between relying on packet header fields and relying
> [HS] The knowledge of network topology is a too strong requirement which
> doesn’t apply for carrier networks. So sequence number is necessary for the
> general case. Again, I have talked with Barak about the possibility to make
> both Flow ID and sequence number optional in case some user really keens to
> reduce the overhead and can do without them. Bust still, I believe in most
> cases, these two fields can make life much easier.
>

[MS2] As I argued above, I disagree about flow ID. Any collector
implementation that wants to recognize the case of a change in
encapsulating node will need to implement correlation based on packet
header fields. Once that is implemented, it is simpler to just use that
always, rather than bouncing back and forth between packet header fields
and flow ID.

> [HS] I didn’t see the need to change the encapsulation nodes in my use
> cases. But if there is, the flowID and sequence number can change. I don’t
> see a big trouble here. Maybe I miss something here?
>

[MS2] We need to minimize the chance of sequence number collisions. If two
different packets are identified as the same packet and correlation is
attempted, it will lead to incorrect, misleading results.

> [HS] We have discussed about this. It’s improper to include E2E with Trace
> Option to just get the sequence number. That’s an abuse of the modes and
> doesn’t bring any benefit. A new mode is the right way to go.
>

[MS2] What you are stating is not a fact, it is an opinion. You may have
reservations about the efficiency of this encoding, but calling it "an
abuse of modes" is overstating things by a lot, and is clearly a misreading
of the draft.

>From the beginning when IOAM-Type was defined, we always said that it is
possible to include multiple IOAM-Type Options in the same packet. Look at
the current contents of the E2E Option, which are sequence number and
timestamp. Do you really think anybody would find it useful to just export
sequence number and timestamp without anything else?

Mickey

> 7)      PBT-I: Have an optional “hop count” field which helps to
> correlate and track the order of the postcards
>

---

From: Haoyu song <[email-masked]>
Date: Thu, 28 Mar 2019 08:49:21 +0000

inline
发件人：Mickey Spiegel <[email-masked]>
收件人：Haoyu song <[email-masked]>
抄 送：Shwetha Bhandari (shwethab) <[email-masked]>;Frank Brockners (fbrockne) <[email-masked]>;Ramesh Sivakolundu (sramesh) <[email-masked]>;Tal Mizrahi <[email-masked]>;Barak Gafni <[email-masked]>;IETF IPPM WG <[email-masked]>
时间：2019-03-27 18:39:49
主 题：Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104

On Wed, Mar 27, 2019 at 5:14 AM Haoyu song <[email-masked]<mailto:[email-masked]>> wrote:
Hi Mickey,
Please see my response inline.
From: Mickey Spiegel [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Tuesday, March 26, 2019 3:40 PM
To: Haoyu song <[email-masked]<mailto:[email-masked]>>
Cc: Shwetha Bhandari (shwethab) <[email-masked]<mailto:[email-masked]>>; Frank Brockners (fbrockne) <[email-masked]<mailto:[email-masked]>>; Ramesh Sivakolundu (sramesh) <[email-masked]<mailto:[email-masked]>>; Tal Mizrahi <[email-masked]<mailto:[email-masked]>>; Barak Gafni <[email-masked]<mailto:[email-masked]>>; IETF IPPM WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104
Please see comments inline.
On Tue, Mar 26, 2019 at 6:07 AM Haoyu song <[email-masked]<mailto:[email-masked]>> wrote:
Dear all,
Per the side meeting discussion on IOAM last night, we decide to describe and compare the IOAM trace mode header and PBT header (updated to align with IOAM spec) to spur discussions within the WG, so we can decide if we should make PBT-I another mode of IOAM ( in addition to Incremental Trace, Pre-allocated Trace, E2E, and POT modes). If so, the new mode could be named PHP (Per Hop Postcard).
Thanks for putting this together. It makes it much easier to discuss commonalities and differences, so that we can decide whether this should be indicated by another IOAM-Type field value or a flag in one of the existing trace types.
IMO, there is no need for all these additional three letter acronyms and terminology such as PBT or PHP. I have also heard some objections to the word "immediate" in "immediate export". If we go with a new IOAM-Type field value, my suggestion would be "IOAM Trace with Direct Export Option". If we go with a flag, my suggestion would be just "Direct Export" or "Direct Reports". Anyway, we can decide on terminology later.
[HS] We do need a name to describe the header and feature. I don’t see a problem here. I talked with Barack and thought about a case for using “immediate export” in the trace mode: it change the router behavior whenever there’s no room for extra trace data: If this bit is not set, no data will be added and the end result is that you will only get a partial trace (kind of the currently defined behavior); if the bit is set, then a postcard will be generate to export the current trace data so room is made to accommodate new trace data. And PHP is another mode option for which each node must generate a postcard. Now the semantics for each option is clean.

[MS2] There is no need for a bit to allow exporting the current stack and replacing it with a new stack. I don't see any reason to disallow such behavior. When the MTU limit is reached, give transit nodes a choice whether they want to export the current stack and start over, or not add anything to the existing stack and keep progressing it. Exporting the current stack does not harm downstream nodes in any way. It is a little more work for the collector, but realistically any good collector implementation will want to deal with reports of dropped packets, which requires a similar type of correlation.

The only thing we need to cleanly indicate is when no trace metadata is to be included in the data plane at all, either using an "immediate export" flag or a new IOAM-Type field value.

[HS3] The bisMickeys modify the currently defined default behavior. It can help to support an unambiguous implementation. Using configuration is always possible but that also makes things a little complicated. We can think about some other scenarios where such indicator is useful in addition to the above mentioned case. For example, the packet is needed to be dropped locally. Such a bit works like an "emergency export" indicator.

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

[MS2] As I argued above, I disagree about flow ID. Any collector implementation that wants to recognize the case of a change in encapsulating node will need to implement correlation based on packet header fields. Once that is implemented, it is simpler to just use that always, rather than bouncing back and forth between packet header fields and flow ID.

[HS3] I only argue that a flow ID can help simplify the implementation in some cases. It help reduce parsing load of analyzer and reduce the data export bandwith. Think about the case of SDN, the central controller will assign the flow ID and maintain the mapping table.
Any implementation relying on "sequence number" will have to deal with the case where the encapsulating node changes. Either the identity of the encapsulating node would have to be included in the packet, or some other approach would need to be taken such as randomizing the initial sequence number, in order to minimize the possibility of sequence number collisions.
[HS] I didn’t see the need to change the encapsulation nodes in my use cases. But if there is, the flowID and sequence number can change. I don’t see a big trouble here. Maybe I miss something here?

[MS2] We need to minimize the chance of sequence number collisions. If two different packets are identified as the same packet and correlation is attempted, it will lead to incorrect, misleading results.

[HS3] The situation you mentioned is rare and cab be avoided, so the collision is extremely unlikely. Even if it happens occasionally, i don't think a single error like this can cause disastrous results.
Even if we conclude that "sequence number" is useful, we would still have to decide whether it is necessary to put the "sequence number" directly in this "IOAM Trace Option", or whether it is good enough to include a separate "IOAM Edge-to-Edge Option" as is already defined in draft-ietf-ippm-ioam-data-05.
[HS] We have discussed about this. It’s improper to include E2E with Trace Option to just get the sequence number. That’s an abuse of the modes and doesn’t bring any benefit. A new mode is the right way to go.

[MS2] What you are stating is not a fact, it is an opinion. You may have reservations about the efficiency of this encoding, but calling it "an abuse of modes" is overstating things by a lot, and is clearly a misreading of the draft.

From the beginning when IOAM-Type was defined, we always said that it is possible to include multiple IOAM-Type Options in the same packet. Look at the current contents of the E2E Option, which are sequence number and timestamp. Do you really think anybody would find it useful to just export sequence number and timestamp without anything else?

[HS3]We need to obey the definition and intended usage of each mode. We should avoid hacking on mode. And stacking multiple modes together increases overheard, and parsing and processing cost. Note that in some encapsulation schemes, it's possible that the two modes are separately encapsulated in different places and E2E only exposes to the end nodes, so the hacking won't work at all.

Mickey

1)      Namespace-ID

2)      Flags

3)      Trace Type

---

From: Haoyu song <[email-masked]>
Date: Thu, 28 Mar 2019 08:58:51 +0000

Sorry, some autofill typos are found in my previous email. Resent.
发件人：Haoyu song <[email-masked]>
收件人：Mickey Spiegel <[email-masked]>
抄 送：IETF IPPM WG <[email-masked]>
时间：2019-03-28 09:50:10
主 题：Re: [ippm] Compare IOAM Trace Mode Header and PBT Mode Header RE: IPPM - IOAM Side Meeting @IETF 104

inline

[HS3] The bit is proposed to be used to modify the currently defined default behavior. It can help to support an unambiguous implementation. Using configuration is always possible but that also makes things a little complicated. We can think about some other scenarios where such indicator is useful in addition to the above mentioned case. For example, the packet is needed to be dropped locally. Such a bit works like an "emergency export" indicator.

PHP Header Format:

    0                   1                   2                   3

   |                     Flow ID                                   |

   |                     Sequence Number                           |

[HS3] We need to obey the definition and intended usage of each mode. We should avoid hacking on well defined modes. And stacking multiple modes together increases overhead, and parsing and processing cost. Note that in some encapsulation schemes, it's possible that the two modes are separately encapsulated in different places and E2E only exposes to the end nodes, so the hacking won't work at all.

Mickey

1)      Namespace-ID

2)      Flags

3)      Trace Type