Subject: [Sipping] Perceptual objective media quality reporting

From: "Ravi Raviraj" <[email-masked]>
Date: Thu, 1 Mar 2007 13:46:08 -0000

Gordon,

You said: But isn't MOS really a perceptual score, based on how people
really perceive the quality?

Yes, MOS is the perceptual quality measure of media for both audio and
video.

MOS measure types are broadly categorized into three types.

MOS-subjective: this is where humans listen/watch the media and give
their score. Most accurate, but not practical for in-service prediction.

MOS-objective: this is where computer algorithm objectively predicts the
subjective value. Very practical and accurate in-service perceptual
quality measure for media.

MOS-estimate: used with network planning models. This is not recommended
(ITU-T) for actual customer quality opinion prediction.

Within each one of these categories, there are three further sub types,
listening quality, conversational quality and talker quality.

Thus, we have MOS-LQO (listening quality objective), MOS-CQO
(conversational quality objective) and MOS-TQO (talker quality
objective), etc.

(Other types: MOS-LQS, MOS-LQE, MOS-CQS, MOS-CQE, MOS-TQS, MOS-TQE).

Each one of these in practice could have four further combinations based
of media bandwidth.

Please see ITU-T P.800.1 for further details.

You said:

As such, isn't this the reference for MOS, and when applied to
voice/audio, it would be acoustic...certainly not packet. The
packet-based measurement models have been devised to provide an
approximating low-processing way to obtain a MOS score.

What you are thinking of is reference media used with full-reference
models (models used to predict objective scores), such as PESQ for
speech and J.144 for video.

What I am referring to is reference point (as opposed to reference
media). For example in the ingress direction, MOS could be for ear
(acoustical) reference point, electrical reference point or packet input
reference point. There are other variations too.

Yes, packet based measurement models are low in CPU usage. However,
there are such models out there which can objectively predict MOS pretty
well.

Users select between reference points, methods and models to identify
the problem and then to pin point the location (e.g. end point device
problem as opposed to transport problem). 

Currently, users run a low CPU-power-requirement method most times, and
when problems occur they turn on other techniques at different points to
pin-point the problem/location automatically/dynamically.

This is where number of people wants a framework to handle the
situation. 

All above comments applies equally to video (with variations). For
example with media bandwidth, read as video frame-size/resolution.

Ravi.

  _____  

From: Beith, Gordon [mailto:[email-masked]] 
Sent: 28 February 2007 17:47
To: Ravi Raviraj; [email-masked]
Subject: RE: [Sipping] Perceptual objective media quality
reporting[Scanned]

But isn't MOS really a perceptual score, based on how people really
perceive the quality?

As such, isn't this the reference for MOS, and when applied to
voice/audio, it would be acoustic...certainly not packet. The
packet-based measurement models have been devised to provide an
approximating low-processing way to obtain a MOS score.

If a MOS-scale is applied to video, it would be a reference of optics, I
would think.

Or did I miss what you are saying?

/Gordon

  _____  

From: Ravi Raviraj [mailto:[email-masked]] 
Sent: Wednesday, February 28, 2007 11:47 AM
To: [email-masked]
Subject: [Sipping] Perceptual objective media quality reporting[Scanned]

Robert,

The event package defined in "draft-ietf-sipping-rtcp-summary-01.txt"
and RTCP-XR/HR techniques are mainly for RTP/RTCP based media transport
statistics.

There are non RTP/RTCP scenarios like MPEG-TS over UDP.

An example for clarity requirement is reference point for MOS measure.
Is it acoustical, electrical or packet level?

These are two examples of work required.

Ideally, we need an efficient event package with enough flexibility to
support different possibilities without any ambiguity.

Ravi.

  _____  

From: Robert Sparks [mailto:[email-masked]] 
Sent: 28 February 2007 15:41
To: Ravi Raviraj
Cc: sipping LIST
Subject: Re: [Sipping] Perceptual objective media quality
reporting[Scanned]

(Trimming this to sipping)

How is what you're looking for different from
http://www.ietf.org/internet-drafts/draft-ietf-sipping-rtcp-summary-01.t
xt?

RjS

On Feb 28, 2007, at 9:03 AM, Ravi Raviraj wrote:

Hi all,

Over the last few years, a number of perceptual media quality objective
measures and assessment techniques have been standardized by ITU-T. For
example, MOS-LQO, MOS-CQO, etc. Each one of these parameters could have
number of variations like narrow-band/wide-band, electrical/acoustical,
full reference/no-reference, audio/video, etc. When applied within a
network end-point, each media direction (ingress/egress) could have one
or more of these values associated with it. We need a way to report
these values efficiently without any ambiguity. These MOS values could
be sent/distributed as perceptual objective quality SIP events. In this
way, all the advantages of SIP like mobility, etc. will be pulled in.

I would like to propose a work/discussion item on an event package for
this purpose. Even define/specify a quality reporting framework. Is
there any interest out there?

I presume that, an event package work will be within SIPPING WG, where
should we do the framework work? SIP/AVT?

Thanks.

Ravi.

This email and attachment are confidential. If you have received this
message in error, please notify the sender immediately and delete the
message from your computer. Psytechnics Ltd has taken steps to ensure
that this email is virus free but the recipient should take necessary
steps to make certain. 

_______________________________________________

Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping

This list is for NEW development of the application of SIP

Use [email-masked] for questions on current sip

Use [email-masked] for new developments of core SIP

This email and attachment are confidential. If you have received this
message in error, please notify the sender immediately and delete the
message from your computer. Psytechnics Ltd has taken steps to ensure
that this email is virus free but the recipient should take necessary
steps to make certain.
_______________________________________________
Sipping mailing list  https://www1.ietf.org/mailman/listinfo/sipping
This list is for NEW development of the application of SIP
Use [email-masked] for questions on current sip
Use [email-masked] for new developments of core SIP

---

From: "Hedayat, Kaynam" <[email-masked]>
Date: Thu, 1 Mar 2007 11:12:44 -0500

Generally speaking I believe there is room for improvement in the
current event package. The main shortfall of the current package is
limiting the calculation of MOS by the endpoint only and not a central
system, thereby introducing the existing problem of correlating MOS
scores in networks where different endpoints use different algorithms.
Unless everyone agrees on one algorithm, which is absolutely the wrong
thing to do, there is no other way to correlate scores but to compute
MOS at a central location where the operator has choice on the
appropriate algorithm.
 
Regards,
Kaynam

________________________________

	From: Ravi Raviraj [mailto:[email-masked]] 
	Sent: Thursday, March 01, 2007 8:46 AM
	To: [email-masked]
	Subject: [Sipping] Perceptual objective media quality reporting

	Gordon,

	You said: But isn't MOS really a perceptual score, based on how
people really perceive the quality?

	Yes, MOS is the perceptual quality measure of media for both
audio and video.

	MOS-subjective: this is where humans listen/watch the media and
give their score. Most accurate, but not practical for in-service
prediction.

	MOS-objective: this is where computer algorithm objectively
predicts the subjective value. Very practical and accurate in-service
perceptual quality measure for media.

	MOS-estimate: used with network planning models. This is not
recommended (ITU-T) for actual customer quality opinion prediction.

	Within each one of these categories, there are three further sub
types, listening quality, conversational quality and talker quality.

	(Other types: MOS-LQS, MOS-LQE, MOS-CQS, MOS-CQE, MOS-TQS,
MOS-TQE).

	Each one of these in practice could have four further
combinations based of media bandwidth.

	You said:

	What you are thinking of is reference media used with
full-reference models (models used to predict objective scores), such as
PESQ for speech and J.144 for video.

	What I am referring to is reference point (as opposed to
reference media). For example in the ingress direction, MOS could be for
ear (acoustical) reference point, electrical reference point or packet
input reference point. There are other variations too.

	Yes, packet based measurement models are low in CPU usage.
However, there are such models out there which can objectively predict
MOS pretty well.

	Users select between reference points, methods and models to
identify the problem and then to pin point the location (e.g. end point
device problem as opposed to transport problem). 

	Currently, users run a low CPU-power-requirement method most
times, and when problems occur they turn on other techniques at
different points to pin-point the problem/location
automatically/dynamically.

	All above comments applies equally to video (with variations).
For example with media bandwidth, read as video frame-size/resolution.

	Ravi.

________________________________

	But isn't MOS really a perceptual score, based on how people
really perceive the quality?

	If a MOS-scale is applied to video, it would be a reference of
optics, I would think.

	/Gordon

________________________________

	From: Ravi Raviraj [mailto:[email-masked]] 
	Sent: Wednesday, February 28, 2007 11:47 AM
	To: [email-masked]
	Subject: [Sipping] Perceptual objective media quality
reporting[Scanned]

	Robert,

	The event package defined in
"draft-ietf-sipping-rtcp-summary-01.txt" and RTCP-XR/HR techniques are
mainly for RTP/RTCP based media transport statistics.

	An example for clarity requirement is reference point for MOS
measure. Is it acoustical, electrical or packet level?

	Ideally, we need an efficient event package with enough
flexibility to support different possibilities without any ambiguity.

	Ravi.

________________________________

	RjS

	Hi all,

	Over the last few years, a number of perceptual media quality
objective measures and assessment techniques have been standardized by
ITU-T. For example, MOS-LQO, MOS-CQO, etc. Each one of these parameters
could have number of variations like narrow-band/wide-band,
electrical/acoustical, full reference/no-reference, audio/video, etc.
When applied within a network end-point, each media direction
(ingress/egress) could have one or more of these values associated with
it. We need a way to report these values efficiently without any
ambiguity. These MOS values could be sent/distributed as perceptual
objective quality SIP events. In this way, all the advantages of SIP
like mobility, etc. will be pulled in.

	I would like to propose a work/discussion item on an event
package for this purpose. Even define/specify a quality reporting
framework. Is there any interest out there?

	I presume that, an event package work will be within SIPPING WG,
where should we do the framework work? SIP/AVT?

	Thanks.

	Ravi.

	This email and attachment are confidential. If you have received
this message in error, please notify the sender immediately and delete
the message from your computer. Psytechnics Ltd has taken steps to
ensure that this email is virus free but the recipient should take
necessary steps to make certain. 

	Sipping mailing list
https://www1.ietf.org/mailman/listinfo/sipping

	Use [email-masked] for questions on current
sip

	This email and attachment are confidential. If you have received
this message in error, please notify the sender immediately and delete
the message from your computer. Psytechnics Ltd has taken steps to
ensure that this email is virus free but the recipient should take
necessary steps to make certain.

---

From: "Beith, Gordon" <[email-masked]>
Date: Thu, 1 Mar 2007 11:28:28 -0500

Hi Ravi.

I think we are in agreement then. I think I misunderstood what you were
trying to say around the reference point.

Alan's response is also applicable to this problem, and I will defer
further comment to that one.

Regards,

            Gordon

	Gordon,

	You said:

	Ravi.

	/Gordon

	Robert,

	 

	Ravi.

	RjS

	Hi all,

	Thanks.

	Ravi.

	Sipping mailing list