Subject: [ippm] New Liaison Statement, "LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG12-LS59)"

From: Liaison Statement Management Tool <[email-masked]>
Date: Sat, 23 Mar 2019 08:39:08 -0700

Title: LS/r on current status of the draft Recommendation ITU-T Q.3961 (reply to SG12-LS59)
Submission Date: 2019-03-23
URL of the IETF Web page: https://datatracker.ietf.org/liaison/1634/

From: Arshey Odedra <[email-masked]>
To: Brian Trammell <[email-masked]>,Bill Cerveny <[email-masked]>,Tommy Pauly <[email-masked]>
Cc: Scott Mansfield <[email-masked]>,Bill Cerveny <[email-masked]>,Brian Trammell <[email-masked]>,IP Performance Measurement Discussion List <[email-masked]>,Mirja Kühlewind <[email-masked]>,Spencer Dawkins <[email-masked]>,[email-masked],Tommy Pauly <[email-masked]>
Response Contacts: [email-masked]
Technical Contacts: 
Purpose: For information

Body: This liaison answers SG12-LS59.

The joint meeting of ITU-T SG11 and ETSI TC INT took place in Geneva on 12 March 2019 and they discussed the incoming liaison statements submitted on this subject (SG12-LS73, SG12-LS70, SG12-LS59, SG12-LS58, ETSI TC STQ-STQ(18)059029, ETSI TC STQ-STQ(19)060067r2).
ETSI TC INT and ITU-T SG11 would like to thank ITU-T SG12 for the liaison statement (SG12-LS59), on Draft ITU-T Q.3961 “Testing methodologies of Internet related performance measurements including e2e bit rate within the fixed and mobile operator’s networks” and inform the group about the following:
-	ETSI TC INT informed ITU-T SG11 that any references to ETSI TS 103 222-1 have been deleted from ETSI TS 103 427. ETSI TC INT informed ITU-T SG11 that they decided to endorse ITU-T Q.3960 and close this work item;
-	All revisions as well as the proposed new Annex to Recommendation Y.1540, “Internet protocol data communication service – IP packet transfer and availability performance parameters”, are out of the scope of draft ITU-T Q.3961 because it describes mechanisms of functioning and optimization of TCP/IP. It is not dealing with the end-to-end customer performance. Moreover, the approach defined in Annex to Y.1540 can be used in laboratory or controlled environment;
-	With regard to provide quotes “Coming to the disclaimer of OECD we can say as reported correctly by the regulators page, that The locations and technical features of the broadband connections have been provided by the service users themselves, who are entirely and exclusively liable for their accuracy. The speed resulting from the measurement as the final speed of the connection is also affected by a series of factors, such as the quality of cables, connections and equipment or electromagnetic interference”, this disclaimer is correct and reports components that may heavily affect the user experience with possible impact on churn. For this reason having a set of agreed test specifications that can provide along with GPS coordinates the position of potential lack of customer performance to be analysed.

Following discussion in July 2018 and the TSAG request to organize close collaboration between ITU-T SG11 and ITU-T SG12 on this matter, in March 2019, ITU-T SG11 organized a Workshop on “Benchmarking of emerging technologies and applications. Internet related performance measurements” which was attended by many ITU-T SG11 delegates and it was well received by the audience. ITU-T SG12 experts were invited to attend this event but, unfortunately, they were not able to.
Please note that among the outcomes (SG11-TD813/GEN) of the Workshop on “Benchmarking of emerging technologies and applications. Internet related performance measurements” (during ITU-T SG11 meeting) related to draft ITU-T Q.3961 are:
-	Benchmark is “evaluation of performance value/s of a parameter or set of parameters for the purpose of establishing value/s as the norm against which future performance achievements may be compared or assessed” (ITU-T E.800);
-	Benchmarking is “performance tests of a system based on a suite of standardized performance tests. The main purpose of a performance benchmark is to produce a metric that can be rated and compared with the metric values produced by other systems using the same benchmark” (ITU-T Q.3930);
-	The benchmarking approach can be used for performance assessment of parameters of different systems, networks, services and applications (e.g. call drops, call set-up delay, CPU load, latency, jitter, download transmission speed, upload transmission speed, etc.);
-	The approach highlighted in ITU-T Q.3960 (2016) and the proposed draft ITU-T Q.3961 is compliant with Net Neutrality regulation 2015/2120 from BEREC and OECD 2014 report, underlying that TCP protocol is widely used by customer application.

So far, ITU-T SG11 does not see relevant contributions from ITU-T SG12 experts, which can be integrated into draft ITU-T Q.3961 in order to define testing methodologies of Internet related performance measurements from a customer perspective. Moreover, following the results of the Workshop mentioned above, it was noted that draft ITU-T Q.3961 is in line with BEREC “Net Neutrality Regulatory Assessment Methodology” (2017), presented by the BEREC representative.

Finally, ITU-T SG11 will continue the work on draft ITU-T Q.3961 considering any modifications which may need to be made. The beneficial contributions from ITU-T SG12 experts on supporting the approach specified in draft ITU-T Q.3961 are encouraged by ITU-T SG11.

Attachment: 1
SG11-TD813/GEN: Outcomes of the ITU Workshop benchmarking of emerging technologies and applications. Internet related performance measurements, 11 March 2019, Geneva
Attachments:

    ls80Attach1-11_td813
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-03-23-itu-t-sg-11-ippm-lsr-on-current-status-of-the-draft-recommendation-itu-t-q3961-reply-to-sg12-ls59-attachment-1.docx

    ls80Attach1-11_td813Attach1_Workshop_Benchmarking_of Emerging_Technologies_and_Application
    https://www.ietf.org/lib/dt/documents/LIAISON/liaison-2019-03-23-itu-t-sg-11-ippm-lsr-on-current-status-of-the-draft-recommendation-itu-t-q3961-reply-to-sg12-ls59-attachment-2.pptx