Export of Delay Performance Metrics in IP Flow Information eXport (IPFIX)
Thomas Graf
Swisscom
Binzring 17
Zurich
8045
Switzerland
thomas.graf@swisscom.com
Benoit Claise
Huawei
benoit.claise@huawei.com
Alex Huang Feng
INSA-Lyon
Lyon
France
alex.huang-feng@insa-lyon.fr
21 July 2024
This document specifies new IP Flow Information Export (IPFIX) Information Elements to export the On-Path Telemetry measured delay on the OAM transit and decapsulation nodes.

Introduction
Network operators usually gather and maintain some forms of statistical delay view of their networks (or segments of their networks). That view is meant to help understanding where in the network, for which customer traffic or services, how much, and why abnormal delay is being accumulated. To that aim, delay-related data need to be reported from devices covering both data and control planes. In order to understand which customer traffic is affected, delay-related data need to be reported into customer data-plane context. That enables network operators to quickly identify when the control-plane updates the current path with a different set of intermediate hops (that is, a change of the forwarding path) and interfaces, how the path delay changes for which customer traffic.
With On-Path Telemetry, described in the Network Telemetry Framework and applied in In Situ Operations, Administration, and Maintenance (IOAM) Deployment and Alternate Marking Deployment Framework, the path delay between two endpoints is measured by inserting a timestamp in the packet.
At least two modes of On-Path Telemetry can be distinguished. Passport mode, where only the last hop in the forwarding path of the On-Path Telemetry domain exposes all the metrics, and postcard mode, where the metrics are also exposed in transit nodes. In both modes the forwarding path exposes performance metrics allowing to determine how much delay has been accumulated on which hop. The proposal in this document makes more sense for the postcard mode.
In order to export the delay-related metrics via IPIFX [RFC7011], this document defines four new IPFIX Information Elements (IEs), exposing the On-Path delay on OAM transit and decapsulation nodes, following the postcard mode principles. Since these IPFIX IEs are performance metrics [RFC8911], they must be registered in the "IANA Performance Metric Registry.
Following the guidelines for Registered Performance Metric Requesters and Reviewers [RFC8911], the different characteristics of the performance metrics (Identifier, Name, URI, Status, Requester, Revision, Revision Date, Description, etc.) must be clearly specified in the "IANA Performance Metric Registry in order for the measurement results using the Performance Metrics to be comparable even if they are performed using different implementations and in different networks. The first performance metric characteristic is the selection of a meaningful name, following the "MetricType_Method_SubTypeMethod_... Spec_Units_Output" naming convention (See Section 7.1.2 of [RFC8911]).

+------------------------------------+-------------------------------+
|      Performance Metric            |  IPFIX Information Element    |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelayMeanDeltaMicroseconds |
|P_RFC[RFC-to-be]_Seconds_Mean (TBD1)|(TBD5)                         |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelayMinDeltaMicroseconds  |
|P_RFC[RFC-to-be]_Seconds_Min (TBD2) |(TBD6)                         |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelayMaxDeltaMicroseconds  |
|P_RFC[RFC-to-be]_Seconds_Max (TBD3) |(TBD7)                         |
+------------------------------------+-------------------------------+
|OWDelay_HybridType1_Passive_I       |pathDelaySumDeltaMicroseconds  |
|P_RFC[RFC-to-be]_Seconds_Sum (TBD4) |(TBD8)                         |
+------------------------------------+-------------------------------+

Table 1: Mapping Between IPFIX IEs and Performance Metrics

Assuming time synchronization on devices, the delay is measured by calculating the difference between the timestamp imposed with On-Path Telemetry in the packet at the OAM encapsulation node and the timestamp exported in the IPFIX flow record from the OAM transit and decapsulation nodes. The lowest, highest, mean, and/or the sum of measured path delay can be exported, thanks to the different IPFIX IE specifications.

Delay use case. Packets flow from host 1 to host 2.

                       On-Path Telemetry Domain
              .........................................
              .                                       .
              .    D1                                 .
              . x------>                              .
              .                                       .
              .          D2                           .
              . x-------------------->                .
              .                                       .
              .                  D3                   .
              . x-----------------------------------> .
              .                                       .
(H1) ------ (R1) ------- (R2) ------- (R3) -------- (R4) ------ (H2)
Host 1  Encapsulation   Transit      Transit   Decapsulation  Host 2
            Node         Node 1       Node 2        Node
              .                                       .
              .                                       .
              .........................................

In the usecase shown in Figure 1 using On-path Telemetry to export the delay metrics, the node R2 exports the delay D1, the node R3 exports the delay D2 and the decapsulation node R4 exports the total delay D3 using IPFIX.
The advantages of this solution is that the delay metrics (min, max, and mean) can be computed on the router, and aggregated directly within the Flow Record, saving export bandwidth and computation on the Collector. For the computation of the min, max, and mean delay metric, to be computed locally on the router, the exporter Metering Process require some local caching/processing computation (for each new packets in the flow), specifically the mean value. A less computational heavy solution for the router is the export of the delay sum instead of the delay mean; on the Collector, the delay mean can easily be computed by a single division operation (using the packet count). The alternative, w ith any delay monitoring on the router, requires the export of every single packet as a separate Flow Record, including the timestamps information, for the Collector to compute delay metrics (min, max, and mean) and the recompute the aggregated Flow Record.

Terminology
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
This document makes use of the terms defined in [RFC7011] and [RFC9378].
The following terms are used as defined in [RFC7011]:
- IPFIX
- IPFIX Information Elements (IEs)
- Flow
- Flow Record
- Exporter
The following terms are used as defined in [RFC8911]:
- Performance Metric
- Registered Performance Metric
- Performance Metrics Registry
The following terms are used as defined in Section 5 of [I-D.ietf-opsawg-oam-characterization]:
- Encapsulation node
- Transit node
- Decapsulation node

Performance Metrics
This section defines the new performance metrics following the template defined in Section 11 of [RFC8911].
IANA Note (to be removed): RFC 8192 Section 4 was taken a guiding example.

IP One-Way Delay Hybrid Type I Passive Performance Metrics
This section specifies four performance metrics for the Hybrid Type I Passive assessment of IP One-Way Delay, to be registered in the "IANA Performance Metric Registry.
All column entries besides the Identifier, Name, URI, Description, Output, and Reference Method categories are the same; thus, this section defines four closely related performance metrics. As a result, IANA has assigned corresponding URIs to each of the four registered performance metrics.

Summary
This category includes multiple indexes of the registered performance metrics: the element Identifier and Metric Name.

ID (Identifier)
IANA has allocated the numeric Identifiers TBD1, TBD2, TBD3, and TBD4 for the four Named Metric Entries in the following section.
RFC EDITOR NOTE: please replace TBD1, TBD2, TBD3, and TBD4.

Name
TBD1: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean
TBD2: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min
TBD3: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max
TBD4: OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum
RFC EDITOR NOTE: please replace [RFC-to-be].

URI
URI: https://www.iana.org/assignments/performance-metrics/OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean
URI: https://www.iana.org/assignments/performance-metrics/OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min
URI: https://www.iana.org/assignments/performance-metrics/OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max
URI: https://www.iana.org/assignments/performance-metrics/OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum
RFC EDITOR NOTE: please replace TBD1, TBD2, TBD3, and TBD4.

Description
- OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean: This metric assesses the mean of one-way delays of all successfully forwarded IP packets constituting a single Flow. We consider the measurement of one-way delay based on a single Observation Point (OP) [RFC7011] somewhere in the network.
- OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min: This metric assesses the minimum of one-way delays of all successfully forwarded IP packets constituting a single Flow. We consider the measurement of one-way delay based on a single Observation Point (OP) [RFC7011] somewhere in the network.
- OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max: This metric assesses the maximum of one-way delays of all successfully forwarded IP packets constituting a single Flow. We consider the measurement of one-way delay based on a single Observation Point (OP) [RFC7011] somewhere in the network.
- OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum: This metric assesses the sum of one-way delays of all successfully forwarded IP packets constituting a single Flow. We consider the measurement of one-way delay based on a single Observation Point (OP) [RFC7011] somewhere in the network.

Change Controller
IETF

Version of Registry Format
1.0

Metric Definition
This category includes columns to prompt the entry of all necessary details related to the metric definition, including the immutable document reference and values of input factors, called "Fixed Parameters".

Reference Definition
Almes, G., Kalidindi, S., Zekauskas, M., and A. Morton, Ed., "A One-Way Delay Metric for IP Performance Metrics (IPPM)", STD 81, RFC 7679, DOI 10.17487/RFC7679, January 2016, <https://www.rfc-editor.org/info/rfc7679>. [RFC7679]
Morton, A. and E. Stephan, "Spatial Composition of Metrics", RFC 6049, DOI 10.17487/RFC6049, January 2011, <https://www.rfc-editor.org/info/rfc6049>. [RFC6049]
Section 3.4 of [RFC7679] provides the reference definition of the singleton (single value) one-way delay metric. Section 4.4 of [RFC7679] provides the reference definition expanded to cover a multi-value sample. Note that terms such as "singleton" and "sample" are defined in Section 2 of [RFC2330].
With the OP [RFC7011] typically located between the hosts participating in the IP Flow, the one-way delay metric requires one individual measurement between the OP and sourcing host, such that the Spatial Composition [RFC6049] of the measurements yields a one-way delay singleton.
This document specifies how to export the performance metric using IPFIX.

Fixed Parameters
None

Method of Measurement
This category includes columns for references to relevant sections of the RFC(s) and any supplemental information needed to ensure an unambiguous method for implementations.

Reference Methods
The foundational methodology for this metric is defined in Section 4 of [RFC7323] using the Timestamps option with modifications that allow application at a mid-path OP [RFC7011].

Packet Stream Generation
The timestamp when the packet is being received at OAM encapsulation node. Format depends on On-Path Telemetry implementation. For IOAM, Section 4.4.1 of [RFC9197] describes what kind of timestamps are supported. Section 4.4.2.3 and 4.4.2.4 describe where the timestamp is being inserted. For the Enhanced Alternate Marking Method, Section 2 of [I-D.zhou-ippm-enhanced-alternate-marking] describes timestamp encoding and granularity.

Traffic Filtering (Observation) Details
Runtime Parameters (in the following sections) may be used for Traffic Filtering.

Sampling Distribution
This metric requires a partial sample of all packets that qualify according to the Traffic Filter criteria.

Runtime Parameters and Data Format
Runtime Parameters are input factors that must be determined, configured into a measurement system, and reported with the results for the context to be complete.
The hybrid type I metering parameters must be reported to provide the complete measurement context. As an example, if the IPFIX Metering Process is used, then the IPFIX Metering Process parameters (IPFIX Template Record, potential traffic filters, and potential sampling method and parameters) that generate the Flow Records must be reported to provide the complete measurement context. At a minimum, the following fields are required:
Src: The IP address of the host in the host A Role (format ipv4 address-no-zone value for IPv4 or ipv6-address-no-zone value for IPv6; see Section 4 of [RFC6991]).
Dst: The IP address of the host in the host B Role (format ipv4 address-no-zone value for IPv4 or ipv6-address-no-zone value for IPv6; see Section 4 of [RFC6991]).
T0: T time, the start of a measurement interval (format "date/time" as specified in Section 5.6 of [RFC3339]; see also "date-and-time" in Section 3 of [RFC6991]). The UTC Time Zone is required by Section 6.1 of [RFC2330]. When T0 is "all-zeros", a start time is unspecified and Tf is to be interpreted as the duration of the measurement interval. The start time is controlled through other means.
Tf: A time, the end of a measurement interval (format "date/time" as specified in Section 5.6 of [RFC3339]; see also "date-and-time" in Section 3 of [RFC6991])). The UTC Time Zone is required by Section 6.1 of [RFC2330]. When T0 is "all-zeros", an ending time and date is ignored and Tf is interpreted as the duration of the measurement interval.

Roles
host A: Launches an IP packet to start the Flow.
host B: Receives the IP packet to start the Flow.
Encapsulation Node: Receives the IP Flow packets and encapsulates the timestamp into the packet.
Transit Node: Receives the IP Flow packets and measures the delay between the timestamp in the packet and the timestamp when the packet was received.
Decapsulation Node: Receives the IP Flow packets and computes the delay between the timestamp in the packet and the timestamp when the packet was received and removes the OAM header from the packet.

Output
This category specifies all details of the output of measurements using the metric.

Type
OWDelay Types are discussed in the subsections below.

Reference Definition
For all output types:
OWDelay_HybridType1_Passive_IP: The one-way delay of one IP packet is a Singleton
For each <statistic> Singleton one of the following subsections applies.

OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean
Similar to Section 7.4.2.2 of [RFC8912], the mean SHALL be calculated using the conditional distribution of all packets with a finite value of one-way delay (undefined delays are excluded) -- a single value, as follows:
See Section 4.1 of [RFC3393] for details on the conditional distribution to exclude undefined values of delay, and see Section 5 of [RFC6703] for background on this analysis choice.
See Section 4.2.2 of [RFC6049] for details on calculating this statistic; see also Section 4.2.3 of [RFC6049].
Mean: The time value of the result is expressed in units of seconds, as a positive value of type decimal64 with fraction digits = 9 (similar to the decimal64 in YANG, Section 9.3 of [RFC6020]) with a resolution of 0.000000001 seconds (1.0 ns), and with lossless conversion to/from the 64-bit NTP timestamp as per Section 6 of [RFC5905].

OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min
Similar to Section 7.4.2.3 of [RFC8912], the minimum SHALL be calculated using the conditional distribution of all packets with a finite value of one-way delay (undefined delays are excluded) -- a single value, as follows:
See Section 4.1 of [RFC3393] for details on the conditional distribution to exclude undefined values of delay, and see Section 5 of [RFC6703] for background on this analysis choice.
See Section 4.3.2 of [RFC6049] for details on calculating this statistic; see also Section 4.3.3 of [RFC6049].
Min: The time value of the result is expressed in units of seconds, as a positive value of type decimal64 with fraction digits = 9 (similar to the decimal64 in YANG, Section 9.3 of [RFC6020]) with a resolution of 0.000000001 seconds (1.0 ns), and with lossless conversion to/from the 64-bit NTP timestamp as per Section 6 of [RFC5905].

OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max
Similar to Section 7.4.2.4 of [RFC8912], the maximum SHALL be calculated using the conditional distribution of all packets with a finite value of one-way delay (undefined delays are excluded) -- a single value, as follows:
See Section 4.1 of [RFC3393] for details on the conditional distribution to exclude undefined values of delay, and see Section 5 of [RFC6703] for background on this analysis choice.
See Section 4.3.2 of [RFC6049] for a closely related method for calculating this statistic; see also Section 4.3.3 of [RFC6049]. The formula is as follows:

 Max = (FiniteDelay[j])
 such that for some index, j, where 1 <= j <= N
 FiniteDelay[j] >= FiniteDelay[n] for all n

where all packets n = 1 through N have finite singleton delays.
Max: The time value of the result is expressed in units of seconds, as a positive value of type decimal64 with fraction digits = 9 (similar to the decimal64 in YANG, Section 9.3 of [RFC6020]) with a resolution of 0.000000001 seconds (1.0 ns), and with lossless conversion to/from the 64-bit NTP timestamp as per Section 6 of [RFC5905].

OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum
The sum SHALL be calculated using the conditional distribution of all packets with a finite value of one-way delay (undefined delays are excluded) -- a single value, as follows:
See Section 4.1 of [RFC3393] for details on the conditional distribution to exclude undefined values of delay, and see Section 5 of [RFC6703] for background on this analysis choice.
See Section 4.3.5 of [RFC6049] for details on calculating this statistic. However in this case FiniteDelay or MaxDelay MAY be used.
Sum: The time value of the result is expressed in units of seconds, as a positive value of type decimal64 with fraction digits = 9 (similar to the decimal64 in YANG, Section 9.3 of [RFC6020]) with a resolution of 0.000000001 seconds (1.0 ns), and with lossless conversion to/from the 64-bit NTP timestamp as per Section 6 of [RFC5905].

Metric Units
- Mean
- Min
- Max
- Sum
The one-way delay of the IP Flow singleton is expressed in seconds.

Calibration
Passive Measurements at an OP could be calibrated against an Active Measurement at host A where the Active Measurement represents the ground truth.

Administrative Items

Status
Current

Requester
This RFC
RFC EDITOR NOTE: please replace This RFC text by the RFC issued from this document

Revision
1.0

Revision Date
RFC Date

Comments and Remarks
none

IPFIX Information Elements
This section specifies the follwing new IPFIX IEs:
pathDelayMeanDeltaMicroseconds:
32-bit unsigned integer that identifies the mean path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node).
pathDelayMinDeltaMicroseconds:
32-bit unsigned integer that identifies the lowest path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node).
pathDelayMaxDeltaMicroseconds:
32-bit unsigned integer that identifies the highest path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node).
pathDelaySumDeltaMicroseconds:
64-bit unsigned integer that identifies the sum of the path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node).

Use Cases
The measured On-Path delay can be aggregated with Flow Aggregation as defined in [RFC7015] to the following device and control-plane dimensions to determine:
- With node id and egressInterface(14), on which node which logical egress interfaces have been contributing to how much delay.
- With node id and egressPhysicalInterface(253), on which node which physical egress interfaces have been contributing to how much delay.
- With ipNextHopIPv4Address(15) or ipNextHopIPv6Address(62), the forwarding path to which next-hop IP contributed to how much delay.
- With mplsTopLabelIPv4Address(47) or destinationIPv6Address and srhActiveSegmentIPv6(495), the forwarding path to which MPLS top label IPv4 address or IPv6 destination address and SRv6 active segment contributed to how much delay.
- BGP communities [RFC1997] are often used for setting a path priority or service selection. With bgpDestinationExtendedCommunityList(488) or bgpDestinationCommunityList(485) or bgpDestinationLargeCommunityList(491) which group of prefixes accumulated at which node how much delay.
- With destinationIPv4Address(13), destinationTransportPort(11), protocolIdentifier (4) and sourceIPv4Address(8), or equivalent IPFIX IEs for IPv6, the forwarding path delay on each node from each IPv4 source address to a specific application in the network.
Let us consider the example depicted in Figure 1 from Section 1 as topology example. Below example table shows the aggregated delay per each node, ingressInterface,(10) egressInterface(14), destinationIPv6Address(28) and srhActiveSegmentIPv6(495).

     +-----------+-----------+------+-------------+-------------+------------+
     | ingress   | egress    | Node | destination | srhActive   | Path Delay |
     | Interface | Interface |      | IPv6Address | SegmentIPv6 |            |
     +-----------+-----------+------+-------------+-------------+------------+
     |    271    |    276    |  R1  | 2001:db8::2 | 2001:db8::4 |    0 us    |
     +-----------+-----------+------+-------------+-------------+------------+
     |    301    |    312    |  R2  | 2001:db8::3 | 2001:db8::4 |   22 us    |
     +-----------+-----------+------+-------------+-------------+------------+
     |    22     |     27    |  R3  | 2001:db8::4 | 2001:db8::4 |   42 us    |
     +-----------+-----------+------+-------------+-------------+------------+
     |    852    |    854    |  R4  | 2001:db8::4 | 2001:db8::4 |  122 us    |
     +-----------+-----------+------+-------------+-------------+------------+

           Table 2: Example table of measured delay. Ascending by delay.

IANA Considerations

Performance Metrics
This document requests IANA to add four new performance metrics under the "Performance Metrics" registry [RFC8911] with the four templates defined in Section 3.

IPFIX Entities
This document requests IANA to register new IPFIX IEs (see table 3) under the "IPFIX Information Elements" registry [RFC7012] available at "IANA IP Flow Information Export (IPFIX) Entities Registry and assign the following initial code points.

     +-------+--------------------------------+
     |Element|              Name              |
     |   ID  |                                |
     +-------+--------------------------------+
     | TBD5  | pathDelayMeanDeltaMicroseconds |
     |       |                                |
     +-------+--------------------------------+
     | TBD6  | pathDelayMinDeltaMicroseconds  |
     |       |                                |
     +-------+--------------------------------+
     | TBD7  | pathDelayMaxDeltaMicroseconds  |
     |       |                                |
     +-------+--------------------------------+
     | TBD8  | pathDelaySumDeltaMicroseconds  |
     |       |                                |
     +-------+--------------------------------+
  Table 3: New IPFIX IEs in the "IPFIX Information Elements" Registry

Note to the RFC-Editor:
- Please replace TBD5 - TBD8 with the values allocated by IANA
- Please replace all instances of [RFC-to-be] in this section with the RFC number assigned to this document

pathDelayMeanDeltaMicroseconds
Name:
pathDelayMeanDeltaMicroseconds
ElementID:
TBD5
Description:
This Information Element identifies the mean path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node), according to OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean in the IANA Performance Metric Registry
Abstract Data Type:
unsigned32
Data Type Semantics:
deltaCounter
Reference:
[RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Mean in the IANA Performance Metric Registry.

pathDelayMinDeltaMicroseconds
Name:
pathDelayMinDeltaMicroseconds
ElementID:
TBD6
Description:
This Information Element identifies the lowest path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node), according to the OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min in the IANA Performance Metric Registry.
Abstract Data Type:
unsigned32
Data Type Semantics:
deltaCounter
Reference:
[RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Min in the IANA Performance Metric Registry.

pathDelayMaxDeltaMicroseconds
Name:
pathDelayMaxDeltaMicroseconds
ElementID:
TBD7
Description:
This Information Element identifies the highest path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node), according to OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max in the IANA Performance Metric Registry.
Abstract Data Type:
unsigned32
Data Type Semantics:
deltaCounter
Reference:
[RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Max in the IANA Performance Metric Registry.

pathDelaySumDeltaMicroseconds
Name:
pathDelaySumDeltaMicroseconds
ElementID:
TBD8
Description:
This Information Element identifies the sum of the path delay of all packets in the Flow, in microseconds, between the OAM encapsulation node and the local node with the OAM domain (either an OAM transit node or an OAM decapsulation node), according to OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum in the IANA Performance Metric Registry.
Abstract Data Type:
unsigned64
Data Type Semantics:
deltaCounter
Reference:
[RFC-to-be], OWDelay_HybridType1_Passive_IP_RFC[RFC-to-be]_Seconds_Sum in the IANA Performance Metric Registry.

Operational Considerations

Time Accuracy
The same recommendation as defined in Section 4.5 of [RFC5153] for IPFIX applies in terms of clock precision to this document as well.

Mean Delay
The mean (average) path delay can be calculated by dividing the pathDelaySumDeltaMicroseconds(TBD8) by the packetDeltaCount(2) at the IPFIX data collection in order to offload the IPFIX Exporter from calculating the mean for every Flow at export time.

Reduced-size encoding
Unsigned64 has been chosen as type for pathDelaySumDeltaMicroseconds to support cases with large delay numbers and where many packets are being accounted. As an example, a specific Flow Record with path delay of 100 milliseconds cannot observe more than 42949 packets without overflowing the unsigned32 counter. The procedure described in Section 6.2 of [RFC7011] may be applied to reduce network bandwidth between the IPFIX Exporter and Collector if unsigned32 would be large enough without wrapping around.

Measurement Interval
The delay metrics are computed for the Flow Record life time. For long-running Flow, we might miss the temporal distribution of the delay (for example, a longer delay only at the beginning of Flow. If this is an operational problem, the IPFIX Metering Process might be configured with a smaller expiration timeout (see Section 5.1.1. Flow Expiration , [RFC5470].

IOAM Application
This document is applicable in IOAM to the Edge-to-Edge and Direct Exporting Option-Type.
In case of Edge-to-Edge Option-Type, as described in Section 4.6 of [RFC9197], by setting bits 2 and 3, timestamps can be encoded as defined in Section 4.4.2.3 and 4.4.2.4 of [RFC9197].
In case of Direct Exporting Option-Type, as described in Section 2 of [I-D.ahuang-ippm-dex-timestamp-ext], by setting Extension-Flags 2 and 3, timestamps can be encoded as defined in Section 4.4.2.3 and 4.4.2.4 of [RFC9197].
For the Enhanced Alternate Marking Method, Section 2 of [I-D.zhou-ippm-enhanced-alternate-marking] defines that within the metaInfo a nanosecond timestamp can be encoded in the encapsulation node and be read at the intermediate and decapsulation node to calculate the on-path delay. [RFC9343] defines how this can be applied to the IPv6 data-plane and [I-D.fz-spring-srv6-alt-mark] defines how this can be applied to the Segment Routing Header in SRv6.

Security Considerations
The IPFIX Information Element introduced in this doucment do not directly introduce security issues. Rather, it defines a set of performance metrics that may, for privacy or business issues, be considered sensitive information.
For example, exporting delay metrics may make attacks possible for the receiver of this information; this would otherwise only be possible for direct observers of the reported Flows along the data path.
The underlying protocol used to exchange the information described here must therefore apply appropriate procedures to guarantee the integrity and confidentiality of the exported information. These protocols are defined in separate documents, specifically the IPFIX protocol document [RFC7011].

Implementation Status
Note to the RFC-Editor: Please remove this section before publishing.

FD.io VPP
INSA Lyon implemented the following IEs as part of a prototype in the FD.io VPP (Vector Packet Processing) platform:
- pathDelayMeanDeltaMicroseconds
- pathDelayMaxDeltaMicroseconds
- pathDelayMinDeltaMicroseconds
- pathDelaySumDeltaMicroseconds
The open source code can be obtained here: [INSA-Lyon-VPP] and was validated at the IETF 116 hackathon.

Huawei VRP
Huawei implemented the following IEs as part of a a production implementation in the VRP platform:
- pathDelayMeanDeltaMicroseconds
- pathDelayMaxDeltaMicroseconds
- pathDelayMinDeltaMicroseconds
- pathDelaySumDeltaMicroseconds
The implementation was validated at the IETF 116 hackathon.

Fluvia
NTT Com implemented the following IEs in the Fluvia Exporter:
- pathDelayMeanDeltaMicroseconds
- pathDelayMaxDeltaMicroseconds
- pathDelayMinDeltaMicroseconds
- pathDelaySumDeltaMicroseconds
The open source code can be obtained here: [NTT-Fluvia] and was validated at the IETF 118 hackathon.

Pmacct Data Collection
Paolo Lucente implemented the IE pathDelayMeanDeltaMicroseconds by dividing IE pathDelaySumDeltaMicroseconds by IE packetDeltaCount in the open source Network Telemetry data collection project pmacct.
The source code can be obtained here: [Paolo-Lucente-Pmacct] and was validated at the IETF 116 hackathon.

Acknowledgements
The authors would like to thank Al Morton (Rest in Peace Al), and Giuseppe Fioccola for their review and valuable comments. Special thanks to Paul Aikten (as IPFIX Designated Expert), Greg Mirsky (as IP Performance Metrics Designated Expeort), and to Med Boucadair for his very detailed feedback.

Normative References

Informative References
[IANA-PERF-METRIC]

IANA Performance Metric Registry
[IANA-IPFIX]

IANA IP Flow Information Export (IPFIX) Entities Registry
[INSA-Lyon-VPP]

INSA Lyon, FD.io VPP implementation
[NTT-Fluvia]

NTT Com, Fluvia Exporter
[Paolo-Lucente-Pmacct]

Paolo Lucente, Pmacct open source Network Telemetry Data Collection
[RFC7011]
[RFC8911]
[RFC2119]
[RFC8174]
[RFC9378]
[I-D.ietf-opsawg-oam-characterization]
[RFC7679]
[RFC6049]
[RFC2330]
[RFC7323]
[RFC9197]
[I-D.zhou-ippm-enhanced-alternate-marking]
[RFC6991]
[RFC3339]
[RFC8912]
[RFC3393]
[RFC6703]
[RFC6020]
[RFC5905]
[RFC7015]
[RFC1997]
[RFC7012]
[RFC5153]
[RFC5470]
[I-D.ahuang-ippm-dex-timestamp-ext]
[RFC9343]
[I-D.fz-spring-srv6-alt-mark]

IPFIX Encoding Examples
This appendix represents two different encodings for the newly introduced IEs. Taking Figure 1 from Section 1 as topology example. Below example Table 4 shows the aggregated delay with ingressInterface, egressInterface, destinationIPv6Address and srhActiveSegmentIPv6.

 +------ +------+-----------+-----------+------+---------+---------+---------+---------+
 |ingress|egress|destination|srhActive  |packet|pathDelay|pathDelay|pathDelay|pathDelay|
 |Inter  |Inter |IPv6Address|SegmentIPv6|Delta |MeanDelta|MinDelta |MaxDelta |SumDelta |
 |face   |face  |           |           |Count |Micro..  |Micro..  |Micro..  |Micro..  |
 +-------+------+-----------+-----------+------+---------+---------+---------+---------+
 |  271  |  276 |2001:db8::4|2001:db8::2|  5   |  36 us  |  22 us  |  74 us  | 180 us  |
 +-------+------+-----------+-----------+------+---------+---------+---------+---------+

  Table 4: Aggregated delay with egressInterface and srhActiveSegmentIPv6

Aggregated On-Path Dealay Examples

Template Record and Data Set with Mean Delta
With encoding in Figure 2, the mean (average) path delay is calculated on the exporting node.
- Ingress interface => ingressInterface
- Egress interface => egressInterface
- IPv6 destination address => destinationIPv6Address
- Active SRv6 Segment => srhIPv6ActiveSegment
- Packet Delta Count => packetDeltaCount
- Minimum One-Way Delay => pathDelayMinDeltaMicroseconds (TBD6)
- Maximum One-Way Delay => pathDelayMaxDeltaMicroseconds (TBD7)
- Mean One-Way Delay => pathDelayMeanDeltaMicroseconds (TBD5)

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |          SET ID = 2           |       Length = 40             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      Template ID = 256        |      Field Count = 8          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0|     ingressInterface = 10   |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0|     egressInterface = 14    |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| destinationIPv6Address = 28 |      Field Length = 16        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| srhIPv6ActiveSegment = 495  |      Field Length = 16        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| packetDeltaCount = 5        |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| pathDelayMeanDelta.. = TBD5 |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| pathDelayMinDelta.. = TBD6  |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |0| pathDelayMaxDelta.. = TBD7  |      Field Length = 4         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Figure 2: Template Record for pathDelayMeanDeltaMicroseconds

The data set is represented as follows:

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |         SET ID = 256          |           Length = 60         |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           ingressInterface =  271                             |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           egressInterface =  276                              |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           destinationIPv6Address =                            |
      |                          ...                                  |
      |                          ...                                  |
      |                          2001:db8::2                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           srhIPv6ActiveSegment = ...                          |
      |                          ...                                  |
      |                          ...                                  |
      |                          2001:db8::4                          |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           packetDeltaCount = 5                                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           pathDelayMeanDeltaMicroseconds =  36                |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           pathDelayMinDeltaMicroseconds =  22                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |           pathDelayMaxDeltaMicroseconds =  74                 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 3: Data Set Encoding for pathDelayMeanDeltaMicroseconds

Template Record and Data Set with Sum Delta
With encoding in Figure 4, the mean (average) path delay is calculated on the IPFIX data collection.
- Ingress interface => ingressInterface
- Egress interface => egressInterface
- IPv6 destination address => destinationIPv6Address
- Active SRv6 Segment => srhIPv6ActiveSegment
- Packet Delta Count => packetDeltaCount
- Minimum One-Way Delay => pathDelayMinDeltaMicroseconds (TBD6)
- Maximum One-Way Delay => pathDelayMaxDeltaMicroseconds (TBD7)
- Sum of One-Way Delay => pathDelaySumDeltaMicroseconds (TBD8)

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |          SET ID = 2           |       Length = 40             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |      Template ID = 257        |      Field Count = 8          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|     ingressInterface = 10   |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0|     egressInterface = 14    |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| destinationIPv6Address = 28 |      Field Length = 16        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| srhIPv6ActiveSegment = 495  |      Field Length = 16        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| packetDeltaCount = 5        |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| pathDelayMinDelta.. = TBD6  |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| pathDelayMaxDelta.. = TBD7  |      Field Length = 4         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |0| pathDelaySumDelta.. = TBD8  |      Field Length = 8         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

     Figure 4: Template Record for pathDelaySumDeltaMicroseconds

The data set is represented as follows:

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |         SET ID = 257          |           Length = 60         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           ingressInterface =  271                             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           egressInterface =  276                              |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           destinationIPv6Address =                            |
   |                          ...                                  |
   |                          ...                                  |
   |                          2001:db8::2                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           srhIPv6ActiveSegment = ...                          |
   |                          ...                                  |
   |                          ...                                  |
   |                          2001:db8::4                          |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           packetDeltaCount = 5                                |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           pathDelayMinDeltaMicroseconds =  22                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           pathDelayMaxDeltaMicroseconds =  74                 |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           pathDelaySumDeltaMicroseconds =  180                |
   |                          ...                                  |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

    Figure 5: Data Set Encoding for pathDelaySumDeltaMicroseconds