Subject: Re: [AVT] Suggestion for how to handle derived/computedmetricindraft-avt-rtcpxr-*

From: Randell Jesup <[email-masked]>
Date: Thu, 26 Jul 2007 16:54:14 -0400

"Alan Clark" <[email-masked]> writes:
>The approach used for voice quality is fairly straightforward.  There are
>several parameters that are defined on a codec specific basis - notably Ie
>and Bpl, where Ie identifies the quality degradation due to the encoder and
>Bpl the robustness of the PLC algorithm against packet loss/ discard.  The
>endpoint incorporates a small table of these values for each voice codec
>that it supports.  Based on the codec selected for the call - the
>appropriate parameters are used.  Some other parameters, for example signal
>level, noise level, echo, delay, are less codec dependant.  A good source of
>information on how to do this using G.107 is ITU-T Recommendation G.799.1,
>which is a VoIP trunking gateway standard that incorporates a clear
>definition of how to compute the R factors and MOS scores required by RTCP
>XR.

I read G.799.1, and I saw no reference on how to calculate R or MOS.  Even
if they're elsewhere or in an appendix I missed, those may not (properly)
handle things like adaptive jitter buffers (G.799.1 mentions jitter buffers
briefly, but implies the delay must be chosen according to the goals of
the network - implying a fixed jitter buffer).  Given all the ways an
adaptive jitter buffer can deal with lengthening and shrinking the buffer,
I don't think any objective algorithm is going to do a good job of
measuring the effect of that operation, certainly not from outside the
jitter buffer/codec code.

Roni's point was that UA developers generally get black-box codecs to play
with.  They generally know about packet loss, jitter (going into the jitter
buffer), codec, etc , but PLC may well be hidden from them.  Also, a loss
of a packet when someone is listening (i.e. silence) has far less effect
than a loss at the start of a word (witness the Neil Armstrong moon
landing, sort of), and the UA developer probably doesn't know if the
packets on either side of the loss indicate that there was talking or not.

This is doubly the case for video.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: "Alan Clark" <[email-masked]>
Date: Thu, 26 Jul 2007 18:59:23 -0400

>I read G.799.1, and I saw no reference on how to calculate R or MOS.  Even
if they're elsewhere or in an appendix I missed, 

You should look again - Appendix IV to G.799.1 (2004) is entitled "Mapping
of E Model (G.107) Parameters from RFC3611 Metrics". 

>those may not (properly) handle things like adaptive jitter buffers
(G.799.1 mentions jitter buffers briefly, but implies the delay must be
chosen according to the goals of the network - implying a fixed jitter
buffer).  Given all the ways an adaptive jitter buffer can deal with
lengthening and shrinking the buffer, I don't think any objective algorithm
is going to do a good job of measuring the effect of that operation,
certainly not from outside the jitter buffer/codec code.

Actually - I don't quite agree with you.  A jitter buffer essentially
replaces jitter with additional packet loss and delay.  The combined effects
of network packet loss and packets discarded by the jitter buffer due to
late arrival lead to the same result.  The added delay due to the jitter
buffer increases overall delay as seen by the user, however this effect is
generally perceived over a longer time period due to conversational
interaction problems (unless of course there is a high level of echo on the
call)

with.  They generally know about packet loss, jitter (going into the jitter
buffer), codec, etc , but PLC may well be hidden from them.  

This functionality could of course be implemented by the codec.  In
addition, many commercial codecs have APIs that allow metrics to be
retrieved - in some cases a fairly prolific set of metrics.  This does
typically include information related to both packet processing (i.e. loss,
discard, jitter buffer depth) and signal processing (signal levels, noise
levels, echo levels).

> Also, a loss of a packet when someone is listening (i.e. silence) has far
less effect than a loss at the start of a word (witness the Neil Armstrong
moon landing, sort of), and the UA developer probably doesn't know if the
packets on either side of the loss indicate that there was talking or not.

Several answers to this one!!

(i) Objective quality measurement generally does not take into account
emotional or semantic content, but generally tries to be robust over a wide
range of types of content (e.g. testing using content in multiple
languages).  A good analogy was explained to me by a subjective test expert
from a major US service provider - apparently they used to conduct state by
state phone surveys of subscribers to fulfil FCC reporting requirements.
One survey indicated that the (subjective) quality of service in that state
had dropped significantly, which caused considerable concern to senior
management; they found out however that the unemployment rate in the state
had just increased significantly - you can imagine if someone had just lost
their job and the phone company called to say "how happy are you with our
service" that the response may be less than complementary!!

(ii) You can take into account whether packet loss affects silence or speech
however from a service quality perspective this is not necessarily good.  If
there is an apparent dip in service quality at some point as losses (for
some strange reason) are tending to impact talkspurts more than silence then
the service provider may view this as the metric describing service quality
dropping when the service they are delivering is essentially unchanged.

In some senses, a reported MOS score is not a prediction of what this
specific user would say about this specific call but a measure of the
"service quality" of this call expressed in terms of MOS.  The reason for
doing this is that service providers understand what MOS is, it is providing
a language to describe service quality in terms that relate to the user
experience.  

Regards

Alan