Subject: [xrblock] QoE draft submitted

From: Alan Clark <[email-masked]>
Date: Mon, 14 Nov 2011 21:53:02 -0500

Charles

As you requested - I have resubmitted the former AVT Working Group QoE draft
as an individual draft. It is available in the IETF system as:

draft-clark-xrblock-rtcp-xr-qoe-00.txt

This draft is aligned with ITU-T P.800.1, which defines the MOS terminology
that should be used for subjective, objective and estimated MOS scores for
narrowband and wideband codecs.

There are issues with MOS scaling that are well known in the field of
objective and perceptual quality estimation algorithms.  Simply stated -
giving a MOS-LQ of 4.0 does not actually provide much information about the
quality of a Voice call.

(i) Wideband and Narrowband voice occupy the same MOS range  - a wideband
call with a MOS of 3.8 may sound much better than a narrowband call with a
MOS of 4.0. When ITU-T extends its speech quality work to ultra-wideband
then an ultra-wideband MOS of 3.6 would sound better than a wideband MOS of
3.6 .......

(ii) Different approaches to the calculation of MOS may have different
scalings.  For example G.107 is scaled such that an unimpaired G.711 call
has a MOS of approximately 4.5 however the Japanese national standard for
MOS estimation, which is based on G.107, is scaled such that an unimpaired
G.711 call would have a MOS of 3.7, and a typical subjective test would
result in a MOS of approximately 4.1.

The MOS issues above also apply to video.  Subjective tests are normally
performed using consistent devices / image sizes/ image resolutions and
hence video MOS scores for different resolutions typically occupy the same
range. This can lead to expressions of righteous indignation by service
providers as their HD service appears to have the same MOS as their SD
service, which is counter-intuitive.  The terms Absolute MOS-V and Relative
MOS-V are in widespread use within the industry to address this - Absolute
MOS-V does include factors such as image resolution however Relative MOS-V
does not. My personal view is that ITU will develop terminology similar to
that defined in P.800.1 to address this issue.

The draft also makes it clear that G.107 is a network planning tool, which
is the stated purpose/limitation in ITU-T G.107,  provides the proper
references for objective speech quality measurement (for example P.564), and
introduces the relevant draft standards related to objective video MOS
measurement.

I propose that this draft be used as the basis for the XRBLOCK QoE draft as
it is consistent with current and emerging ITU standards and terminology,
and reflects extensive experience within the industry in the reporting and
use of MOS scores for speech and video.

Best Regards

Alan

---

From: "Schwarz, Albrecht (Albrecht)" <[email-masked]>
Date: Tue, 15 Nov 2011 17:50:08 +0100

Alan,

not deeply familiar with the QoE metrics, readers like me are wondering whether there are potential overlappings with the RFC 3611 BT=7 Call Quality or Transmission Quality Metrics.

Your draft states in 1.1 that the RFC 3611 metrics are augmented by rtcp-xr-qoe.
Does this imply
a) additional metrics?
b) replacement of metrics?
c) enhanced existing metrics?

I guess it's (a) but I think it would be worth to add some additional text in clause 1.1.

Thanks,
Albrecht

________________________________
From: [email-masked] [mailto:[email-masked]] On Behalf Of Alan Clark
Sent: Dienstag, 15. November 2011 03:53
To: Charles Eckel (eckelcu); Shida Schubert
Cc: [email-masked]
Subject: [xrblock] QoE draft submitted

Charles

As you requested - I have resubmitted the former AVT Working Group QoE draft as an individual draft. It is available in the IETF system as:

This draft is aligned with ITU-T P.800.1, which defines the MOS terminology that should be used for subjective, objective and estimated MOS scores for narrowband and wideband codecs.

There are issues with MOS scaling that are well known in the field of objective and perceptual quality estimation algorithms.  Simply stated - giving a MOS-LQ of 4.0 does not actually provide much information about the quality of a Voice call.

(i) Wideband and Narrowband voice occupy the same MOS range  - a wideband call with a MOS of 3.8 may sound much better than a narrowband call with a MOS of 4.0. When ITU-T extends its speech quality work to ultra-wideband then an ultra-wideband MOS of 3.6 would sound better than a wideband MOS of 3.6 .......

(ii) Different approaches to the calculation of MOS may have different scalings.  For example G.107 is scaled such that an unimpaired G.711 call has a MOS of approximately 4.5 however the Japanese national standard for MOS estimation, which is based on G.107, is scaled such that an unimpaired G.711 call would have a MOS of 3.7, and a typical subjective test would result in a MOS of approximately 4.1.

The MOS issues above also apply to video.  Subjective tests are normally performed using consistent devices / image sizes/ image resolutions and hence video MOS scores for different resolutions typically occupy the same range. This can lead to expressions of righteous indignation by service providers as their HD service appears to have the same MOS as their SD service, which is counter-intuitive.  The terms Absolute MOS-V and Relative MOS-V are in widespread use within the industry to address this - Absolute MOS-V does include factors such as image resolution however Relative MOS-V does not. My personal view is that ITU will develop terminology similar to that defined in P.800.1 to address this issue.

The draft also makes it clear that G.107 is a network planning tool, which is the stated purpose/limitation in ITU-T G.107,  provides the proper references for objective speech quality measurement (for example P.564), and introduces the relevant draft standards related to objective video MOS measurement.

I propose that this draft be used as the basis for the XRBLOCK QoE draft as it is consistent with current and emerging ITU standards and terminology, and reflects extensive experience within the industry in the reporting and use of MOS scores for speech and video.

Best Regards

Alan

---

From: Alan Clark <[email-masked]>
Date: Tue, 15 Nov 2011 18:31:58 -0500

Albrecht

I can expand on the draft to make this clear.

The modular XR metrics blocks provide a more flexible solution that can
address VoIP, Videoconferencing, IPTV and other applications, and which
should provide more detailed metrics for cases in which this is needed.
Implementers could use the existing VoIP metrics block and add some of the
new XR blocks to provide additional data, or could construct a set of VoIP
metrics entirely from the new blocks.  My view is that it would make most
sense to focus the new blocks on audio, multi-channel audio,
videoconferencing, IPTV and other multimedia applications.

Also - the original intent of the ³HR² draft was to provide a higher
resolution MOS value - i.e. 16 bits rather than 8.  MOS estimation
algorithms are not that precise and hence in terms of MOS value
representation 8 bits/ 1 decimal place is fine, however there was a
perceived need to see changes in the MOS value that may be smaller than 1
decimal place.

Best Regards

Alan

On 11/15/11 11:50 AM, "Schwarz, Albrecht (Albrecht)"
<[email-masked]> wrote:

> Alan,
>  
> not deeply familiar with the QoE metrics, readers like me are wondering
> whether there are potential overlappings with the RFC 3611 BT=7 Call Quality
> or Transmission Quality Metrics.
>

---

From: Alan Clark <[email-masked]>
Date: Tue, 15 Nov 2011 18:31:58 -0500

Albrecht

Best Regards

Alan

---

From: "Schwarz, Albrecht (Albrecht)" <[email-masked]>
Date: Wed, 16 Nov 2011 00:59:29 +0100

Thanks Alan,
such kind of information would be beneficial!
Regards,
Albrecht

________________________________
From: Alan Clark [mailto:[email-masked]]
Sent: Mittwoch, 16. November 2011 00:32
To: Schwarz, Albrecht (Albrecht)
Cc: [email-masked]
Subject: Re: QoE draft submitted

Albrecht

The modular XR metrics blocks provide a more flexible solution that can address VoIP, Videoconferencing, IPTV and other applications, and which should provide more detailed metrics for cases in which this is needed.  Implementers could use the existing VoIP metrics block and add some of the new XR blocks to provide additional data, or could construct a set of VoIP metrics entirely from the new blocks.  My view is that it would make most sense to focus the new blocks on audio, multi-channel audio, videoconferencing, IPTV and other multimedia applications.

Also - the original intent of the "HR" draft was to provide a higher resolution MOS value - i.e. 16 bits rather than 8.  MOS estimation algorithms are not that precise and hence in terms of MOS value representation 8 bits/ 1 decimal place is fine, however there was a perceived need to see changes in the MOS value that may be smaller than 1 decimal place.

Best Regards

Alan

On 11/15/11 11:50 AM, "Schwarz, Albrecht (Albrecht)" <[email-masked]> wrote:

Alan,

Thanks,
Albrecht

Charles

Best Regards

Alan

---

From: Qin Wu <[email-masked]>
Date: Wed, 16 Nov 2011 03:16:52 +0000

Hi,
Albrecht, you ask a very good question. The VOIP metric report specified in RFC3611 only focus on VOIP application.
However VOIP metric application put all the metric parameters in the same metric block, which make it more VOIP specific and can not cover emerging new applications like IPTV. That's why XRBlock WG adopt QoE metric reporting milestone and try to address this.
However the problem is what kind of granularity do we really need to distinguish different kinds of MOS? I think the coarse granularity is to categorize MOS into Speech Quality MOS, Video Quality MOS and Audio-Video Quality MOS.
For each MOS type, we have already had matured calculation algorithm for Speech Quality MOS, i.e., algorithm defined in G.107 and P.564,TS101-329-5. G107 and TS101-329-5 are also referenced by RFC3611 to calculate MOS defined in VOIP metric block of RFC3611.

As for Video Quality MOS and Audio-Video Quality MOS, P.NAMS and PNBAMS specified in ITU-T SG12 can be referenced to calculate the MOS value.
If we further break down these MOS type into more precise granularity, it looks to me, the algorithms used are quite similar to the algorithms we listed above.

So for the more precise granularity of MOS, e.g., narrowband, wideband, ultra wideband, why not just use signaling or out band mechanism to indicate that?

As for the length of MOS value, I think 8 bits can be used to represent integral part prior to decimal place, but another 8 bits can be used to stand for fraction part after decimal place. So 16bits of MOS value should be okay in my opinion.

Regards!
-Qin

________________________________
发件人: [email-masked] [[email-masked]] 代表 Alan Clark [[email-masked]]
发送时间: 2011年11月16日 7:31
到: Schwarz, Albrecht (Albrecht)
Cc: [email-masked]
主题: Re: [xrblock] QoE draft submitted

Albrecht

Also - the original intent of the “HR” draft was to provide a higher resolution MOS value - i.e. 16 bits rather than 8.  MOS estimation algorithms are not that precise and hence in terms of MOS value representation 8 bits/ 1 decimal place is fine, however there was a perceived need to see changes in the MOS value that may be smaller than 1 decimal place.

Best Regards

Alan

On 11/15/11 11:50 AM, "Schwarz, Albrecht (Albrecht)" <[email-masked]<UrlBlockedError.aspx>> wrote:

Alan,

Thanks,
Albrecht

________________________________
From: [email-masked]<UrlBlockedError.aspx>  [mailto:[email-masked]] On Behalf Of Alan  Clark
Sent: Dienstag, 15. November 2011 03:53
To: Charles  Eckel (eckelcu); Shida Schubert
Cc:  [email-masked]<UrlBlockedError.aspx>
Subject: [xrblock] QoE draft  submitted

Charles

Best Regards

Alan

---

From: Paul Coverdale <[email-masked]>
Date: Wed, 16 Nov 2011 11:44:22 +0800

As a newcomer to XRBLOCK, but with some experience in MOS testing, I was
intrigued by the discussion yesterday about the number of bits required to
represent MOS. I don’t know whether you’ve got better uses for spare bits,
but 16 bits is still more than enough to represent MOS for all practical
purposes.

…Paul

From: [email-masked] [mailto:[email-masked]] On Behalf
Of Qin Wu
Sent: Wednesday, November 16, 2011 11:17 AM
To: Alan Clark; Schwarz, Albrecht (Albrecht)
Cc: [email-masked]
Subject: [xrblock] 答复: QoE draft submitted

As for the length of MOS value, I think 8 bits can be used to represent
integral part prior to decimal place, but another 8 bits can be used to
stand for fraction part after decimal place. So 16bits of MOS value should
be okay in my opinion.

  _____  

发件人: [email-masked] [[email-masked]] 代表 Alan Clark
[[email-masked]]
发送时间: 2011年11月16日 7:31
到: Schwarz, Albrecht (Albrecht)
Cc: [email-masked]
主题: Re: [xrblock] QoE draft submitted

Albrecht

Also - the original intent of the “HR” draft was to provide a higher

Best Regards

Alan

Alan,

Your draft states in 1.1 that the RFC 3611 metrics are augmented by
rtcp-xr-qoe.
Does this imply
a) additional metrics?
b) replacement of metrics?
c) enhanced existing metrics?

I guess it's (a) but I think it would be worth to add some additional text
in clause 1.1.

Thanks,
Albrecht

  _____  

From: [email-masked]  [mailto:[email-masked]] On Behalf
Of Alan  Clark
Sent: Dienstag, 15. November 2011 03:53
To: Charles  Eckel (eckelcu); Shida Schubert
Cc:  [email-masked]
Subject: [xrblock] QoE draft  submitted

Charles

As you requested - I have resubmitted the former AVT  Working Group QoE
draft as an individual draft. It is available in the IETF  system as:

The draft  also makes it clear that G.107 is a network planning tool, which
is the stated  purpose/limitation in ITU-T G.107,  provides the proper
references for  objective speech quality measurement (for example P.564),
and introduces the  relevant draft standards related to objective video MOS
measurement.

Best Regards

Alan

---

From: Alan Clark <[email-masked]>
Date: Tue, 15 Nov 2011 23:15:33 -0500

Hi Paul

I think that you may also agree that MOS estimation algorithms are typically
accurate to at most one decimal place and hence that 8 bits is normally
enough.  The motivation for using 16 bits was that some service providers
wanted to be able to see small degradations in MOS.

Best Regards

Alan

On 11/15/11 10:44 PM, "Paul Coverdale" <[email-masked]> wrote:

> represent MOS. I don’t know whether you’ve got better uses for spare bits, but
> 16 bits is still more than enough to represent MOS for all practical purposes.
>  
> …Paul
>  
> 
> From: [email-masked] [mailto:[email-masked]] On Behalf Of
> Qin Wu
> integral part prior to decimal place, but another 8 bits can be used to stand
> for fraction part after decimal place. So 16bits of MOS value should be okay
> in my opinion.
> 
>  
> 
>  
> 
> 
> address VoIP, Videoconferencing, IPTV and other applications, and which should
> provide more detailed metrics for cases in which this is needed.  Implementers
> could use the existing VoIP metrics block and add some of the new XR blocks to
> provide additional data, or could construct a set of VoIP metrics entirely
> from the new blocks.  My view is that it would make most sense to focus the
> new blocks on audio, multi-channel audio, videoconferencing, IPTV and other
> multimedia applications.
> 
> resolution MOS value - i.e. 16 bits rather than 8.  MOS estimation algorithms
> are not that precise and hence in terms of MOS value representation 8 bits/ 1
> decimal place is fine, however there was a perceived need to see changes in
> the MOS value that may be smaller than 1 decimal place.
> 
> Best Regards
> 
> Alan
> 
> <[email-masked] <UrlBlockedError.aspx> > wrote:
> Alan,
> 
> From: [email-masked] <UrlBlockedError.aspx>
> [mailto:[email-masked]] On Behalf Of Alan  Clark
> Cc:  [email-masked] <UrlBlockedError.aspx>
> MOS of 4.0. When  ITU-T extends its speech quality work to ultra-wideband then
> an ultra-wideband  MOS of 3.6 would sound better than a wideband MOS of 3.6
> .......
> 
> scalings.   For example G.107 is scaled such that an unimpaired G.711 call has
> a MOS  of approximately 4.5 however the Japanese national standard for MOS
> estimation, which is based on G.107, is scaled such that an unimpaired G.711
> call would have a MOS of 3.7, and a typical subjective test would result in a
> MOS of approximately 4.1.
> 
> performed using consistent devices / image  sizes/ image resolutions and hence
> video MOS scores for different resolutions  typically occupy the same range.
> This can lead to expressions of righteous  indignation by service providers as
> their HD service appears to have the same  MOS as their SD service, which is
> counter-intuitive.  The terms Absolute  MOS-V and Relative MOS-V are in
> widespread use within the industry to address  this - Absolute MOS-V does
> include factors such as image resolution however  Relative MOS-V does not. My
> personal view is that ITU will develop terminology  similar to that defined in
> P.800.1 to address this issue.
> 
> The draft  also makes it clear that G.107 is a network planning tool, which is
> the stated  purpose/limitation in ITU-T G.107,  provides the proper references
> for  objective speech quality measurement (for example P.564), and introduces
> the  relevant draft standards related to objective video MOS measurement.
> 
> it  is consistent with current and emerging ITU standards and terminology, and
> reflects extensive experience within the industry in the reporting and use of
> MOS scores for speech and video.
> 
> Best Regards
> 
> Alan
>  
>

---

From: Qin Wu <[email-masked]>
Date: Wed, 16 Nov 2011 05:42:12 +0000

That's correct. Higher precision can be achieved by having 16bits with Numeric format 8:8.

Regards!

-Qin

________________________________
发件人: Alan Clark [[email-masked]]
发送时间: 2011年11月16日 12:15
到: Paul Coverdale; Qin Wu; 'Schwarz, Albrecht (Albrecht)'
Cc: [email-masked]
主题: Re: [xrblock] 答复: QoE draft submitted

Hi Paul

I think that you may also agree that MOS estimation algorithms are typically accurate to at most one decimal place and hence that 8 bits is normally enough.  The motivation for using 16 bits was that some service providers wanted to be able to see small degradations in MOS.

Best Regards

Alan

On 11/15/11 10:44 PM, "Paul Coverdale" <[email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx>> wrote:

As a newcomer to XRBLOCK, but with some experience in MOS testing, I was intrigued by the discussion yesterday about the number of bits required to represent MOS. I don’t know whether you’ve got better uses for spare bits, but 16 bits is still more than enough to represent MOS for all practical purposes.

…Paul

From: [email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx> [mailto:[email-masked]] On Behalf Of Qin Wu
Sent: Wednesday, November 16, 2011 11:17 AM
To: Alan Clark; Schwarz, Albrecht (Albrecht)
Cc: [email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx>
Subject: [xrblock] 答复: QoE draft submitted

发件人: [email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx> [[email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx>] 代表 Alan Clark [[email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx>]
发送时间: 2011年11月16日 7:31
到: Schwarz, Albrecht (Albrecht)
Cc: [email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx>
主题: Re: [xrblock] QoE draft submitted

Albrecht

Best Regards

Alan

On 11/15/11 11:50 AM, "Schwarz, Albrecht (Albrecht)" <[email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx> <UrlBlockedError.aspx> > wrote:
Alan,

Thanks,
Albrecht

From: [email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx> <UrlBlockedError.aspx>   [mailto:[email-masked]] On Behalf Of Alan  Clark
Sent: Dienstag, 15. November 2011 03:53
To: Charles  Eckel (eckelcu); Shida Schubert
Cc:  [email-masked]<https://imail.huawei.com/OWA/UrlBlockedError.aspx> <UrlBlockedError.aspx>
Subject: [xrblock] QoE draft  submitted

Charles

Best Regards

Alan

---

From: Colin Perkins <[email-masked]>
Date: Mon, 28 Nov 2011 17:08:32 +0000

On 16 Nov 2011, at 03:16, Qin Wu wrote:
...
> So for the more precise granularity of MOS, e.g., narrowband, wideband, ultra wideband, why not just use signaling or out band mechanism to indicate that?
...

How much of this can be inferred from the RTP payload type being used in the session, without additional signalling?

-- 
Colin Perkins
http://csperkins.org/

---

From: Qin Wu <[email-masked]>
Date: Wed, 30 Nov 2011 14:53:58 +0800

Hi,
----- Original Message ----- 
From: "Colin Perkins" <[email-masked]>
To: "Qin Wu" <[email-masked]>
Cc: "Alan Clark" <[email-masked]>; "Schwarz, Albrecht (Albrecht)" <[email-masked]>; <[email-masked]>
Sent: Tuesday, November 29, 2011 1:08 AM
Subject: Re: [xrblock] QoE draft submitted

...

[Qin]: AFAIK,
a. the narrowband, wideband and ultrawideband only apply to audio applications or audio codecs.
b. audio codecs tend evolving from narrowband to wideband, or from wideband to ultra wideband.
c. most audio codec only support one mode, e.g.,
G.729 only supports narrowband, G.729.1 only supports wideband and G.729.1 SWB  ony support ultra wideband.
AMR-NB support narrowband, AMR-WB support wideband and AMR-WB+ support ultra wideband.
but this doesn't mean that one RTP payload for audio application is not possible to support both narrowband and wideband, e.g., EVRC-NW, EV-VBR.
In EVRC-NW, Encoding capability identification in the payload header is used to indicate if wideband or narrowband is used.
In EV-VBR, Layer identificiation value in the payload format can be used to indicate if narrowband or wideband is used.

So to my understanding, in most cases, payload type can be used to indicate if narrowband or wideband is used in the audio applications.
However in very few other cases, you may need additional information in the payload format to faciliate your identification.
For all the video applications, we don't need such feature.

-- 
Colin Perkins
http://csperkins.org/