Subject: [xrblock] Adam Roach's No Objection on draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)

From: Adam Roach <[email-masked]>
Date: Mon, 21 May 2018 17:55:49 -0700

Adam Roach has entered the following ballot position for
draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: No Objection

When responding, please keep the subject line intact and reply to all
email addresses included in the To and CC lines. (Feel free to cut this
introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-xrblock-rtcweb-rtcp-xr-metrics/

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

Thanks to the authors and contributors on this document. I have two substantive
comments, and a handful of minor nits.

---------------------------------------------------------------------------

§5.3.1:

>  application can use these metrics to calculate the Mean Opinion Score
>  (MoS) values or Media Delivery Index (MDI) for their services.

"...calculate Estimated Mean Opinion Score (eMOS) values..."

(MOS scores necessarily require human subjective evaluation of audio under
carefully controlled circumstances -- any computer-generated metrics are
inherently estimations, and should not be called MOS scores.)

---------------------------------------------------------------------------

§5.3.1:

>  Error-resilience mechanisms, like RTP retransmission or FEC, are
>  optional in RTCWEB because the overhead of the repair bits adding to
>  the original streams.

This is a little misleading. draft-ietf-rtcweb-rtp-usage says:

   WebRTC endpoints MUST follow the recommendations for FEC use given in
   [I-D.ietf-rtcweb-fec].

And, in turn, draft-ietf-rtcweb-fec says:

   To support the functionality recommended above, implementations MUST
   be able to receive and make use of the relevant FEC formats for their
   supported audio codecs, and MUST indicate this support, as described
   in Section 4.  Use of these formats when sending, as mentioned above,
   is RECOMMENDED.

Rather than trying to reiterate this somewhat nuanced situation in *this*
document, I recommend removing the entire sentence and fixing the rest of the
paragraph. If you choose to keep it, please make sure it more clearly explains
the level of FEC support required of RTCWEB implementations.

===========================================================================

The remainder of my comments are grammatical or editorial nits that should be
fixed before progressing the document.

---------------------------------------------------------------------------

§3:

>  The RTCP Sender Reports (SRs) and Receiver Reports (RRs) [RFC3550]
>  exposes the basic metrics...

"expose"

>  However, these metrics provides...

"provide"

>  For example, it may be useful to distinguish between
>  packets lost and packets discarded due to late arrival, even though
>  they have the same impact on the multimedia quality, it helps in
>  identifying and diagnosing issues.

This is a run-on sentence. Suggest: "...due to late arrival. Even though..."

>  The WebRTC application extracts the statistic from the browser by
>  querying the getStats() API [W3C.WD-webrtc-20161124], but the browser
>  currently only reports the local variables i.e., the statistics
>  related to the outgoing RTP media streams and the incoming RTP media
>  streams.

I don't think the "currently" in this sentence is actually true any longer; and
I expect it to get increasingly less true as time goes on. Consider rephrasing.

>  [I-D.ietf-rtcweb-rtp-usage] does not mandate the use of
>  any RTCP XRs and since their usage is optional.

This is not a sentence. Maybe remove "since"?

---------------------------------------------------------------------------

§4:

>  For example, an application may choose
>  to poll the stack for statistics every 1 second, in this case the
>  underlying stack local will return the current snapshot of the local
>  statistics (for incoming and outgoing media streams).

This is a comma splice. Consider: "...every second. In this case..."

>  However it may
>  return the same remote statistics as before for the remote
>  statistics

Add a comma after "However".

---------------------------------------------------------------------------

§5:

>  Since following metrics are all defined in RTCP XR which is not
>  mandated in WebRTC, all of them are local.

"Since the following..."
       ^^^

"...RTCP XR, which is..."
           ^

>  However, if RTCP XR is
>  supported by negotiation between two browsers, following metrics can
>  also be generated remotely and be sent to local by RTCP XR packets.

"...two browsers, the following..."
                  ^^^

>  Following metrics are classified into 3 categories: network impact

"The following metrics..."
 ^^^

>  viewpoint of application, e.g., bit rate, frames rate or jitter

"...frame rate..."

>  WebRTC
>  application can use these metrics to calculate the Mean Opinion Score

"applications"

---------------------------------------------------------------------------

§5.1.1:

>  lost and duplicated packets.  Lost packets counts are useful for

"Lost packet counts..."

---------------------------------------------------------------------------

§5.1.3:

>  the two run length vectors to identify congestion-related losses,
>  i.e., a router queue became overloaded causing delays and then

Replace "i.e." ("that is") with "e.g." ("for example").

---------------------------------------------------------------------------

§5.2.1:

>  The metric reports the cumulative size of the packets discarded in
>  the interval, it is complementary to number of discarded packets.

"...the interval. It is complementary..."

---------------------------------------------------------------------------

§5.2.2:

>  For
>  audio streams, a single RTP packet may contain one or multiple audio
>  frames, each of which has a fixed length.

I don't think this is generally true. Even in the case of WebRTC, I believe the
default mode of operation for Opus is VBR, which results in frame sizes that
vary from frame to frame.

>  The metrics in this category includes: number of discarded key

"...category include:..."

---------------------------------------------------------------------------

§5.3.1:

>  The un-repaired packets count and repaired loss count defined in

"...unrepaired packet count..."

>  packets to lost packets.  Including this kind of metrics helps the

Choose either "...these kinds of metrics..." or "...this kind of metric..."

---------------------------------------------------------------------------

§5.3.2:

>  related losses, i.e., a router queue became overloaded causing delays

Replace "i.e." with "e.g."

---------------------------------------------------------------------------

§6:

>  identifiers relevant to RTP media in WebRTC.  These group of
>  identifiers are defined on a ReportGroup corresponding to an
>  Synchronization source (SSRC).  In practice the application need to

"This group of identifiers..."
 ^^^^

"...corresponding to a synchronization source..."
                     ^

"...the application needs to..."
                    ^^^^^

>  For XR metrics, it depends on
>  two factors: 1) if it measured at the endpoint, 2) if it reported by
>  the endpoint in an XR report.

"...if it is measured..."
          ^^

"...if it is reported..."
          ^^

---------------------------------------------------------------------------

§11.2:

>  [W3C.WD-webrtc-20161124]
>             Sporny, M. and D. Longley, "WebRTC 1.0: Real-time
>             Communication Between Browsers", World Wide Web Consortium
>             WD WD-webrtc-20161124, November 2016,
>             <https://www.w3.org/TR/2016/WD-webrtc-20161124>.

The authorship of this document is incorrect (neither Sporny nor Longley are
authors on WebRTC).

The URL points to an outdated version of the specification. Please update to
https://www.w3.org/TR/2017/CR-webrtc-20171102/

>  [W3C.WD-webrtc-stats-20161214]
>             Alvestrand, H. and V. Singh, "Identifiers for
>             WebRTC&#039;s Statistics API", World Wide Web Consortium
>             WD WD-webrtc-stats-20161214, December 2016,
>             <https://www.w3.org/TR/2016/WD-webrtc-stats-20161214>.

Please replace "&#039;" with an apostrophe.

Please update the URL to https://www.w3.org/TR/2018/WD-webrtc-stats-20180519/

---

From: Adam Roach <[email-masked]>
Date: Mon, 21 May 2018 19:58:29 -0500

> inherently estimations, and should not be called MOS scores.)

Sorry, that should have said "§5" rather than "§5.3.1."

/a

---

From: "Huangyihong (Rachel)" <[email-masked]>
Date: Tue, 22 May 2018 03:18:23 +0000

Hi Adam,

Thanks for the comments. Please see inline.

BR,
Rachel

> -----Original Message-----
> From: Adam Roach [mailto:[email-masked]]
> Sent: Tuesday, May 22, 2018 8:56 AM
> To: The IESG
> Cc: [email-masked]; Shida Schubert;
> [email-masked]; [email-masked]; [email-masked]
> Subject: Adam Roach's No Objection on
> draft-ietf-xrblock-rtcweb-rtcp-xr-metrics-09: (with COMMENT)
> 

[Rachel]: You're right. Will change to "estimated Mean Opinion Score (MOS)" in the next version.

[Rachel]: Your concern makes sense to me. I'll delete it as following:

OLD

"
   Error-resilience mechanisms, like RTP retransmission or FEC, are
   optional in RTCWEB because the overhead of the repair bits adding to
   the original streams.  But they do help to greatly reduce the impact
   of packet loss and enhance the quality of transmission.  Web
   applications could support certain repair mechanism after negotiation
   between both sides of browsers when needed.  
"

NEW
"
Web applications could support certain RTP error-resilience mechanisms following the recommendations specified in [draft-ietf-rtcweb-rtp-usage].
"

> Please update the URL to
> https://www.w3.org/TR/2018/WD-webrtc-stats-20180519/
> 

[Rachel]: They'll be fixed in the next version. Thanks.

---

From: Adam Roach <[email-masked]>
Date: Mon, 21 May 2018 22:20:33 -0500

Thanks! Your proposed resolutions look good to me.

/a

---

From: Ben Campbell <[email-masked]>
Date: Thu, 24 May 2018 14:26:33 -0500

Hi,

It looks like the discussion on Adam’s comments have reached resolution, and I think that is also mostly true for the other IESG comments. Authors; Please submit a new revision when you are ready.

Thanks!

Ben.

> On May 21, 2018, at 10:20 PM, Adam Roach <[email-masked]> wrote:
>

---

From: "Huangyihong (Rachel)" <[email-masked]>
Date: Fri, 25 May 2018 00:38:57 +0000

Will do it today.

BR,
Rachel

> From: Ben Campbell [mailto:[email-masked]]
> Sent: Friday, May 25, 2018 3:27 AM
> To: [email-masked]; Huangyihong (Rachel)
> <[email-masked]>
> Cc: The IESG <[email-masked]>; [email-masked];
> [email-masked]; [email-masked]
> Subject: Re: Adam Roach's No Objection on
> It looks like the discussion on Adam’s comments have reached resolution, and
> I think that is also mostly true for the other IESG comments. Authors; Please
> submit a new revision when you are ready.
> 
> Thanks!
> 
> Ben.
> 
> >>> Cc: [email-masked]; Shida
> >>> Schubert; [email-masked]; [email-masked];
> >>> [email-masked]
> >>> When responding, please keep the subject line intact and reply to
> >>> all email addresses included in the To and CC lines. (Feel free to
> >>> cut this introductory paragraph, however.)
> >>>
> >>>
> >>> Please refer to
> >>> https://www.ietf.org/iesg/statement/discuss-criteria.html
> >>> Thanks to the authors and contributors on this document. I have two
> >>> substantive comments, and a handful of minor nits.
> >>>
> >>> (MOS scores necessarily require human subjective evaluation of audio
> >>> under carefully controlled circumstances -- any computer-generated
> >>> metrics are inherently estimations, and should not be called MOS
> >>> scores.)
> >>>
> >>>> optional in RTCWEB because the overhead of the repair bits adding
> >>>> to  the original streams.
> >>> Rather than trying to reiterate this somewhat nuanced situation in
> >>> *this* document, I recommend removing the entire sentence and fixing
> >>> the rest of the paragraph. If you choose to keep it, please make
> >>> sure it more clearly explains the level of FEC support required of
> >>> RTCWEB implementations.
> >>>
> >> Web applications could support certain RTP error-resilience mechanisms
> following the recommendations specified in [draft-ietf-rtcweb-rtp-usage].
> >> "
> >>
> >>>
> >>> The remainder of my comments are grammatical or editorial nits that
> >>> should be fixed before progressing the document.
> >>>
> >>>>  For example, it may be useful to distinguish between  packets lost
> >>>> and packets discarded due to late arrival, even though  they have
> >>>> the same impact on the multimedia quality, it helps in  identifying
> >>>> and diagnosing issues.
> >>>> querying the getStats() API [W3C.WD-webrtc-20161124], but the
> >>>> browser  currently only reports the local variables i.e., the
> >>>> statistics  related to the outgoing RTP media streams and the
> >>>> incoming RTP media  streams.
> >>> I don't think the "currently" in this sentence is actually true any
> >>> longer; and I expect it to get increasingly less true as time goes on.
> Consider rephrasing.
> >>>
> >>>>  [I-D.ietf-rtcweb-rtp-usage] does not mandate the use of  any RTCP
> >>>> XRs and since their usage is optional.
> >>>>  For example, an application may choose  to poll the stack for
> >>>> statistics every 1 second, in this case the  underlying stack local
> >>>> will return the current snapshot of the local  statistics (for
> >>>> incoming and outgoing media streams).
> >>>>  supported by negotiation between two browsers, following metrics
> >>>> can  also be generated remotely and be sent to local by RTCP XR
> packets.
> >>>>  audio streams, a single RTP packet may contain one or multiple
> >>>> audio  frames, each of which has a fixed length.
> >>> I don't think this is generally true. Even in the case of WebRTC, I
> >>> believe the default mode of operation for Opus is VBR, which results
> >>> in frame sizes that vary from frame to frame.
> >>>
> >>>>  two factors: 1) if it measured at the endpoint, 2) if it reported
> >>>> by  the endpoint in an XR report.
> >>> The authorship of this document is incorrect (neither Sporny nor
> >>> Longley are authors on WebRTC).
> >>>
> >>> The URL points to an outdated version of the specification. Please
> >>> update to https://www.w3.org/TR/2017/CR-webrtc-20171102/
> >>>
> >>>