Subject: [ippm] New Liaison Statement, "LS - Harmonization of IP Capacity and Latency Parameters: Revision of Draft Rec. Y.1540 on IP packet transfer performance parameters and New Annex A with Lab Evaluation Plan"

From: Liaison Statement Management Tool <[email-masked]>
Date: Wed, 13 Mar 2019 10:23:20 -0700

Title: LS - Harmonization of IP Capacity and Latency Parameters: Revision of Draft Rec. Y.1540 on IP packet transfer performance parameters and New Annex A with Lab Evaluation Plan
Submission Date: 2019-03-13
URL of the IETF Web page: https://datatracker.ietf.org/liaison/1632/
Please reply by 2019-05-01
From: Judit Kiss <[email-masked]>
To: Brian Trammell <[email-masked]>,Bill Cerveny <[email-masked]>,Tommy Pauly <[email-masked]>
Cc: Scott Mansfield <[email-masked]>,Bill Cerveny <[email-masked]>,Brian Trammell <[email-masked]>,IP Performance Measurement Discussion List <[email-masked]>,Mirja Kühlewind <[email-masked]>,Spencer Dawkins <[email-masked]>,[email-masked],Tommy Pauly <[email-masked]>
Response Contacts: A. C. Morton <[email-masked]>
Technical Contacts: 
Purpose: For action

Body: As an update to our communications from the Q17/12 Interim meeting (Darmstadt, 16-17 October 2018) and the Geneva, 27 November - 6 December 2018 meeting with the full SG12, we now share further results from the Q17/12 Interim meeting (Berlin, 5-7 March 2019). 

We have completed nearly all planned tests using the Phase 1 Laboratory test bed that supports development our revised Rec. Y.1540 and new Annex A/Y.1540 specifications of IP Capacity and Latency methods of measurement. We have compared several existing measurement methods based on TCP and UDP transport. The Phase 2 Network tests are just starting now.

We continue to invite interested parties to join the testing efforts, and to coordinate on the development of a new generation of harmonized specifications of IP Capacity performance metrics and methods of measurement, and other key performance parameters.

Our key results and current findings from the Berlin meeting are:

•	The text of the IP Capacity definition and Methods of measurement, and the Phase 1 and Phase 2 sections of the test and evaluation plan were edited and agreed during the meeting. In the agreed text, the Phase 2 study will be carried-out in production networks, and lab networks if they are made available in a timely fashion. This material constitutes revisions to Y.1540 sections and the new ANNEX A/Y.1540. The meeting agreed to seek Consent on this material in May, 2018.
•	The meeting agreed to prepare the Preliminary Test Summary (presented for the first time at this meeting) as a new informative APPENDIX to Y.1540, to provide the supporting test data for the conclusions represented as requirements in the body and Annex A.
•	Even without correction factors, the UDP measurements are considerably closer to the calibrated shaper rate than TCP, despite using the most favourable circumstances for TCP (no added delay or background traffic). With correction for headers that are viewed by the traffic shaper, UDP measurements using iPerf 2 are within 200ppm of the configured shaper rate. UDP-based measurements are the benchmark for capacity, accurately assessing the “ground truth” of the traffic shaper rate under all tested conditions.
•	TCP measurements using iPerf2 underestimate the shaper rate with or without correction factors. Typical round-trip delay and the presence of competing/background traffic tend to make TCP-based estimates of available capacity appreciably worse.
•	Another contribution updated the overall academic Survey, adding material to one section on WiFi performance and one new section Encrypted stream Network QoS parameters and their relationship to QoE. Both topics introduce a set of KPIs/metrics prioritized through machine learning, and models of QoE.  The relevance to active testing includes the possible additional metrics collected, or input for stream design.
•	It is clear from the surveys presented that Internet subscriber use of TCP protocol is declining. Video and browser traffic has been shifting to UDP and higher-layer reliability mechanisms for years, with the most significant growth in 2018 due to CDN adoption. 
•	There was a demonstration of a new prototype measurement tool that makes UDP-based Capacity, Latency, and Loss measurements while searching for the peak Capacity that can be supported on the tested path. The meeting provided useful feedback for the developer.

Q17/12 and collaborating SDOs have covered many new areas of investigation, and are approaching the point where development of coordinated specifications can proceed. 
All group members continue to seek additional SDO and individual support, participation and/or constructive review, so that together, we can proceed toward the next generation of IP performance metrics.

Attachments:
–	SG12-TD775 with revised Y.1540 clauses and new Y.1540 Annex A;
–	SG12-TD776 with new Y.1540 Appendix, Phase 1 Test Summary.
Attachments:

    TD775
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-03-13-itu-t-sg-12-ippm-ls-harmonization-of-ip-capacity-and-latency-parameters-revision-of-draft-rec-y1540-on-ip-packet-transfer-perfo-attachment-1.docx

    TD776
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-03-13-itu-t-sg-12-ippm-ls-harmonization-of-ip-capacity-and-latency-parameters-revision-of-draft-rec-y1540-on-ip-packet-transfer-perfo-attachment-2.docx

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Wed, 20 Mar 2019 19:52:09 +0000

IPPM,

This Liaison details a topic I would like to discuss further
during IETF-104, but the IPPM agenda is quite full, and
a side meeting could provide the opportunity.

The Liaison and attachments describe a fairly extensive 
Lab comparison of test methods for IP Capacity and Latency 
measurement. There is also much to appreciate about how
Internet usage is changing, and how measurements could 
change to continue to be relevant.

Please contact me if you are interested. We'll set up a time
for discussion of the material and next steps.

regards,
Al

> -----Original Message-----
> From: Liaison Statement Management Tool [mailto:[email-masked]]
> Sent: Wednesday, March 13, 2019 1:23 PM
> To: Bill Cerveny <[email-masked]>; Brian Trammell <[email-masked]>;
> Tommy Pauly <[email-masked]>
> Cc: Mirja Kühlewind <[email-masked]>; Scott Mansfield
> <[email-masked]>; Bill Cerveny <[email-masked]>; MORTON,
> ALFRED C (AL) <[email-masked]>; Spencer Dawkins
> <[email-masked]>; IP Performance Measurement Discussion
> List <[email-masked]>; Brian Trammell <[email-masked]>; itu-t-
> [email-masked]; Tommy Pauly <[email-masked]>
> Subject: New Liaison Statement, "LS - Harmonization of IP Capacity and
> Latency Parameters: Revision of Draft Rec. Y.1540 on IP packet transfer
> performance parameters and New Annex A with Lab Evaluation Plan"
> 
> Title: LS - Harmonization of IP Capacity and Latency Parameters: Revision
> of Draft Rec. Y.1540 on IP packet transfer performance parameters and New
> Annex A with Lab Evaluation Plan
> URL of the IETF Web page:
> https://urldefense.proofpoint.com/v2/url?u=https-
> 3A__datatracker.ietf.org_liaison_1632_&d=DwIDaQ&c=LFYZ-
> o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=uLz_oPVMGlxwhg8pC1vN9Gw4ypYFZ
> k506Axq2RLOyo0&s=ynxvwzaXF2tlJ0iSuBqmqYiJfPueJhlEBFtiyfxfNcg&e=
> To: Brian Trammell <[email-masked]>,Bill Cerveny
> <[email-masked]>,Tommy Pauly <[email-masked]>
> Cc: Scott Mansfield <[email-masked]>,Bill Cerveny
> <[email-masked]>,Brian Trammell <[email-masked]>,IP Performance
> Measurement Discussion List <[email-masked]>,Mirja Kühlewind
> <[email-masked]>,Spencer Dawkins <[email-masked]>,itu-
> [email-masked],Tommy Pauly <[email-masked]>
> Body: As an update to our communications from the Q17/12 Interim meeting
> (Darmstadt, 16-17 October 2018) and the Geneva, 27 November - 6 December
> 2018 meeting with the full SG12, we now share further results from the
> Q17/12 Interim meeting (Berlin, 5-7 March 2019).
> 
> We have completed nearly all planned tests using the Phase 1 Laboratory
> test bed that supports development our revised Rec. Y.1540 and new Annex
> A/Y.1540 specifications of IP Capacity and Latency methods of measurement.
> We have compared several existing measurement methods based on TCP and UDP
> transport. The Phase 2 Network tests are just starting now.
> 
> We continue to invite interested parties to join the testing efforts, and
> to coordinate on the development of a new generation of harmonized
> specifications of IP Capacity performance metrics and methods of
> measurement, and other key performance parameters.
> 
> •	The text of the IP Capacity definition and Methods of measurement,
> and the Phase 1 and Phase 2 sections of the test and evaluation plan were
> edited and agreed during the meeting. In the agreed text, the Phase 2
> study will be carried-out in production networks, and lab networks if they
> are made available in a timely fashion. This material constitutes
> revisions to Y.1540 sections and the new ANNEX A/Y.1540. The meeting
> agreed to seek Consent on this material in May, 2018.
> •	The meeting agreed to prepare the Preliminary Test Summary
> (presented for the first time at this meeting) as a new informative
> APPENDIX to Y.1540, to provide the supporting test data for the
> conclusions represented as requirements in the body and Annex A.
> •	Even without correction factors, the UDP measurements are
> considerably closer to the calibrated shaper rate than TCP, despite using
> the most favourable circumstances for TCP (no added delay or background
> traffic). With correction for headers that are viewed by the traffic
> shaper, UDP measurements using iPerf 2 are within 200ppm of the configured
> shaper rate. UDP-based measurements are the benchmark for capacity,
> accurately assessing the “ground truth” of the traffic shaper rate under
> all tested conditions.
> •	TCP measurements using iPerf2 underestimate the shaper rate with or
> without correction factors. Typical round-trip delay and the presence of
> competing/background traffic tend to make TCP-based estimates of available
> capacity appreciably worse.
> •	Another contribution updated the overall academic Survey, adding
> material to one section on WiFi performance and one new section Encrypted
> stream Network QoS parameters and their relationship to QoE. Both topics
> introduce a set of KPIs/metrics prioritized through machine learning, and
> models of QoE.  The relevance to active testing includes the possible
> additional metrics collected, or input for stream design.
> •	It is clear from the surveys presented that Internet subscriber use
> of TCP protocol is declining. Video and browser traffic has been shifting
> to UDP and higher-layer reliability mechanisms for years, with the most
> significant growth in 2018 due to CDN adoption.
> •	There was a demonstration of a new prototype measurement tool that
> makes UDP-based Capacity, Latency, and Loss measurements while searching
> for the peak Capacity that can be supported on the tested path. The
> meeting provided useful feedback for the developer.
> 
> Q17/12 and collaborating SDOs have covered many new areas of
> investigation, and are approaching the point where development of
> coordinated specifications can proceed.
> All group members continue to seek additional SDO and individual support,
> participation and/or constructive review, so that together, we can proceed
> toward the next generation of IP performance metrics.
> 
> Attachments:
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2019-2D03-2D13-2Ditu-2Dt-2Dsg-2D12-2Dippm-2Dls-2Dharmonization-2Dof-2Dip-2Dcapacity-2Dand-2Dlatency-2Dparameters-2Drevision-2Dof-2Ddraft-2Drec-2Dy1540-2Don-2Dip-2Dpacket-2Dtransfer-2Dperfo-2Dattachment-2D1.docx&d=DwIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=uLz_oPVMGlxwhg8pC1vN9Gw4ypYFZk506Axq2RLOyo0&s=dAR1A1k2C0NFX677nghi88SGItQwZxXCUB3dnMtgshw&e

>     TD776
https://urldefense.proofpoint.com/v2/url?u=https-3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2019-2D03-2D13-2Ditu-2Dt-2Dsg-2D12-2Dippm-2Dls-2Dharmonization-2Dof-2Dip-2Dcapacity-2Dand-2Dlatency-2Dparameters-2Drevision-2Dof-2Ddraft-2Drec-2Dy1540-2Don-2Dip-2Dpacket-2Dtransfer-2Dperfo-2Dattachment-2D2.docx&d=DwIDaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=OfsSu8kTIltVyD1oL72cBw&m=uLz_oPVMGlxwhg8pC1vN9Gw4ypYFZk506Axq2RLOyo0&s=S0LEXw6CpSLFV3ME16BU0wA4iUoUEjs5gLFLs6ElumA&e

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 21 Mar 2019 10:16:29 +0100

Hi Al,
please add me to the list. Indeed exciting and very much relevant topic.

Regards,
Greg

On Wed, Mar 20, 2019 at 8:53 PM MORTON, ALFRED C (AL) <[email-masked]>
wrote:

> > To: Bill Cerveny <[email-masked]>; Brian Trammell <[email-masked]
> >;
> > <[email-masked]>,Spencer Dawkins <[email-masked]
> >,itu-
> _______________________________________________
> ippm mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ippm
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 25 Mar 2019 10:10:22 +0000

IPPM Folks interested in discussion of this Liaison,
and the potential reply:

Based on the quick assembly post-ippm session today,
the best time to meet face2face is 

Break   -   12:20 to 13:00   

We'll meet in the Code Lounge, on the Mezzanine level,
in room Chez Louis.

We'll prepare a draft Liaison reply and circulate on the list.

Al

> From: MORTON, ALFRED C (AL)
> Sent: Wednesday, March 20, 2019 3:52 PM
> To: [email-masked]; Bill Cerveny <[email-masked]>; Brian Trammell
> <[email-masked]>; Tommy Pauly <[email-masked]>
> <[email-masked]>; Spencer Dawkins
> <[email-masked]>
> Subject: RE: New Liaison Statement, "LS - Harmonization of IP Capacity and
> 3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2019-2D03-2D13-2Ditu-
> 2Dt-2Dsg-2D12-2Dippm-2Dls-2Dharmonization-2Dof-2Dip-2Dcapacity-2Dand-
> 2Dlatency-2Dparameters-2Drevision-2Dof-2Ddraft-2Drec-2Dy1540-2Don-2Dip-
> 2Dpacket-2Dtransfer-2Dperfo-2Dattachment-2D1.docx&d=DwIDaQ&c=LFYZ-
> k506Axq2RLOyo0&s=dAR1A1k2C0NFX677nghi88SGItQwZxXCUB3dnMtgshw&e
> 
> >
> >     TD776
> >
> 3A__www.ietf.org_lib_dt_documents_LIAISON_liaison-2D2019-2D03-2D13-2Ditu-
> 2Dt-2Dsg-2D12-2Dippm-2Dls-2Dharmonization-2Dof-2Dip-2Dcapacity-2Dand-
> 2Dlatency-2Dparameters-2Drevision-2Dof-2Ddraft-2Drec-2Dy1540-2Don-2Dip-
> 2Dpacket-2Dtransfer-2Dperfo-2Dattachment-2D2.docx&d=DwIDaQ&c=LFYZ-
> k506Axq2RLOyo0&s=S0LEXw6CpSLFV3ME16BU0wA4iUoUEjs5gLFLs6ElumA&e
>

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Mon, 25 Mar 2019 11:03:22 +0000

Re-sending ...  This meeting is   T O D A Y !

> Sent: Monday, March 25, 2019 6:10 AM
> To: [email-masked]; 'Bill Cerveny' <[email-masked]>; 'Brian Trammell'
> <[email-masked]>; 'Tommy Pauly' <[email-masked]>; 'Greg Mirsky'
> <[email-masked]>; [email-masked]; [email-masked]'
> <[email-masked]>; 'John Lemon' <[email-masked]>
> Cc: 'Mirja Kühlewind' <[email-masked]>; 'Scott Mansfield'
> <[email-masked]>; 'Spencer Dawkins'

---

From: "MORTON, ALFRED C (AL)" <[email-masked]>
Date: Thu, 28 Mar 2019 15:42:40 +0000

IPPM,

Following a side-meeting discussion of the Subject Liaison,
and several individuals indicating continuing interest
in this work,  I would like to propose the brief text of
a Liaison reply. 

The WG will need to reach consensus on the text of
the reply as a next step.

Thanks to all who have expressed interest in this work!
regards,
Al

Proposed Body of Liaison Reply (to all recipients) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

To: ITU-T SG12, ITU-T SG11, BBF ETSI TC STQ, ETSI TC INT

IETF IPPM working group thanks you for your liaison describing results from
your evaluation of IP Capacity and Latency Metrics and  methods of measurement. 
The harmonization of these measurements is welcome in our industry.

We have the following comments on the preliminary results, and
the evaluation plan:

After noting the efficacy of UDP measurements in your results,
we can state that UDP transport is the basis for most IPPM measurement
implementations, and IPPM's measurement protocols.

When assessing IP Capacity and Latency, there should be 
no attempt to determine the technology involved in the path, 
such as AQM. The method should measure the path as a black box. 

The presence of packet-marking-sensitive technologies, such as
Diff-serv queues add complexity to IP Capacity and Latency.

There is not yet a metric that characterizes the BW-Delay product
in the IPPM-literature. 

Live network testing may be affected by Network operator policy,
it could change from the lab measurements - the test traffic is
part of the background traffic. OTOH, the operator could be 
prioritizing test traffic, especially their own authorized testing.

We also suggest to include QUIC-based measurements in your Lab 
evaluation, if possible.

Where a model or post-processing needs to be applied to the measurements,
this aspect must also be specified, especially for transport with multiple 
connections (averaging the connection performance).

Some of the IPPM WG participants who indicated their interest in
November 2019 have begun to share their comments informally, 
as planned.

Please keep the IPPM WG informed of your progress, especially if there are
ways in which IPPM WG participants assist further.

<signed by chairs of IPPM WG>

> k506Axq2RLOyo0&s=dAR1A1k2C0NFX677nghi88SGItQwZxXCUB3dnMtgshw&e=
> 
>     TD776
> k506Axq2RLOyo0&s=S0LEXw6CpSLFV3ME16BU0wA4iUoUEjs5gLFLs6ElumA&e=

---

From: Greg Mirsky <[email-masked]>
Date: Thu, 28 Mar 2019 16:52:21 +0100

Hi Al, and All,
I agree with the response to the liaison.

Regards,
Greg