Subject: RE: [AVT] Suggestion for how to handle derived/computed metricindraft-avt-rtcpxr-*

From: "Even, Roni" <[email-masked]>
Date: Thu, 26 Jul 2007 22:29:48 +0300

Alan,
What bothers me is that in my view many people building audio and video
EPs will use codecs that come from the DSP manufacturer or from a open
source library. It will be difficult to them to choose the right
metrics. I am not the biggest expert on quality assessments but I think
that there is a difference between the current voice codecs (3, 7Khz)
and the new audio ones (14 and 20Khz) which are now used in video
conferencing and are coming to VOIP applications in term of the quality
assessment. 

Now how do you expect a non-educated EP developer to know what is the
right way to calculate the quality. How will he calculate quality if his
system may use G.711 in one call and AAC-LD in a second call. It needs
either to be very easy to understand or just ask him for raw data.

Roni 

> -----Original Message-----
> From: Alan Clark [mailto:[email-masked]]
> Sent: Tuesday, July 24, 2007 1:08 AM
> To: 'David R Oran'; 'AVT WG'
> Subject: RE: [AVT] Suggestion for how to handle derived/computed
> metricindraft-avt-rtcpxr-*
> 
> Dave
> 
> I agree with the general tenor of your thoughts.  I also agree that we
> should not mandate that endpoints have to implement any particular set
of
> derived metrics - these should be optional and should be implemented
based
> on their usefulness.
> 
> For cases in which there is a fairly well defined metric class (e.g.
MOS
> for
> speech) then I suggest that we do define a separate draft for that
with a
> registry of algorithms that could be (optionally) used - it is easier
to
> update a registry with information that a new version of ITU-T G.xxx
is
> available than it would be to generate a new draft.
> 
> Regards
> 
> Alan
> 
> 
> -----Original Message-----
> From: David R Oran [mailto:[email-masked]]
> Sent: Monday, July 23, 2007 5:51 PM
> To: AVT WG
> Subject: [AVT] Suggestion for how to handle derived/computed metric
> indraft-avt-rtcpxr-*
> 
> At the Mic during the AVT session I sketched a vague idea for how to
> resolve
> the conflict between the desire on the part of many folks to only
carry
> raw/primitive instrumented measurements in RTCP-XR, and other equally
> vocal,
> who argued for carrying the derived/computed metrics as well.
> 
> I won't repeat the pro/con arguments here. Let's assume we want to
carry
> both.
> 
> The chairs asked me to write up my vague thoughts as the beginnings of
a
> concrete proposal.
> 
> Some particular technical difficulties with the derived metrics that
makes
> them different from the primitive measurements is:
> - there are competing standards for how to compute them
> - the results are not generally aggregatable, comparable, or self-
> describing.
> - the set of interesting derived metrics is not small and seems to be
> growing, so it will be difficult to nail these down for stable RTCP
> reporting.
> 
> This makes it problematic to mix primitive and derived metrics in the
same
> metric blocks in RTCP.
> 
> So, my suggestion is to do the following.
> 
> - A standards body who defines one or more consistent derived metrics
may
> create (by bringing a draft to IETF) a separate metric block for those
> metrics.
> - The block and its contained derived/computed metrics would be
completely
> self-describing in terms of exactly what version of what standard
defines
> the block.
> - The definition of the metric would explicit contain the input
variables,
> in order to ensure that those inputs are separately available as
primitive
> metrics in the RTCP-XR. This increases the probability that any metric
> that
> is computable by the endpoint is also computable by a central element
> receiving and processing RTCP-XR reports.
> - If there are competing/overlapping blocks, an endpoint need not do
all
> of
> them and there would be no confusion about exactly which (e.g."video
MOS")
> calculation was done.
> 
> As an aside, I would rather not see further updates to the existing
drafts
> until we resolve this issue, since this is in fact major surgery to
the
> documents that is best done before polishing the text and individual
> metric
> definitions.
> 
> Dave Oran.
> 
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt
> 
> 
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Alan Clark" <[email-masked]>
Date: Thu, 26 Jul 2007 15:58:44 -0400

Roni

Certainly there is a difference between voice and audio codecs. In the case
of voice codecs (both narrowband and wideband) the methods for objectively
measuring the quality of the call and for taking codec characteristics into
account is well understood - there are admittedly several approaches however
there are existing standards that address means for calculating objective
estimates of MOS and even for conformance testing of algorithms that do
this.  

The approach used for voice quality is fairly straightforward.  There are
several parameters that are defined on a codec specific basis - notably Ie
and Bpl, where Ie identifies the quality degradation due to the encoder and
Bpl the robustness of the PLC algorithm against packet loss/ discard.  The
endpoint incorporates a small table of these values for each voice codec
that it supports.  Based on the codec selected for the call - the
appropriate parameters are used.  Some other parameters, for example signal
level, noise level, echo, delay, are less codec dependant.  A good source of
information on how to do this using G.107 is ITU-T Recommendation G.799.1,
which is a VoIP trunking gateway standard that incorporates a clear
definition of how to compute the R factors and MOS scores required by RTCP
XR.

Methods for measuring audio quality are less heavily standardized at this
time, although there are ITU standards such as G.1070 (for IP
videoconferencing) that incorporate voice, video and multimedia quality
metrics and ITU-R BS-1387 (which is a PESQ-like algorithm for audio).  There
is ongoing work within the industry related to IPTV audio stream measurement
that is occurring in at least three different standards organizations.

I'd be happy to put together an informational document that outlines
existing approaches to objective voice/ audio/ video performance measurement
and reviews some of the current industry activities - if that would be
useful?

Regards

Alan

-----Original Message-----
From: Even, Roni [mailto:[email-masked]] 
Sent: Thursday, July 26, 2007 3:30 PM
To: Alan Clark; David R Oran; AVT WG
Subject: RE: [AVT] Suggestion for how to handle
derived/computedmetricindraft-avt-rtcpxr-*

Alan,
What bothers me is that in my view many people building audio and video EPs
will use codecs that come from the DSP manufacturer or from a open source
library. It will be difficult to them to choose the right metrics. I am not
the biggest expert on quality assessments but I think that there is a
difference between the current voice codecs (3, 7Khz) and the new audio ones
(14 and 20Khz) which are now used in video conferencing and are coming to
VOIP applications in term of the quality assessment. 

Now how do you expect a non-educated EP developer to know what is the right
way to calculate the quality. How will he calculate quality if his system
may use G.711 in one call and AAC-LD in a second call. It needs either to be
very easy to understand or just ask him for raw data.

Roni 

of
based
MOS
with a
to
is
> resolve the conflict between the desire on the part of many folks to 
> only
carry
> vocal, who argued for carrying the derived/computed metrics as well.
> 
carry
a
makes
same
may
completely
defines
variables,
primitive
> that is computable by the endpoint is also computable by a central 
> element receiving and processing RTCP-XR reports.
all
MOS")
drafts
the
> metric definitions.