Subject: Measurement and testing parameter

From: "Henri Setiawan" <[email-masked]>
Date: Thu, 15 Apr 1999 09:29:29 +0700

Dear folks,

May be you have been evaluating several vendors solution on VoIP. Is there
anyone have been testing the equipments ? Can you share about the
measurement methods and testing parameter to classify the performance of the
equipment ? Also, if you know the measurement equipment for testing please
let me know.

Other case, please explain to me how can we measure MOS ?

Thank you

Henri Setiawan
R&D of PT Telkom Indonesia

---------
This message came from the IETF IPTEL Working Group Mailing List.

---

From: Tom Gray <[email-masked]>
Date: Thu, 15 Apr 1999 10:50:21 -0700 (PDT)

--- "Hoefelmeyer, Ralph (MCI)" <[email-masked]>
wrote:

> 
> What does "YM_M4I5S3S_4" stand for? 
> 

YM_M4I5S3S_5 is Mean Opinion Score. It is a metric which denotes the
perceived quality of the voice connection. It has five categories going
from Excellent to Unacceptable. This is an important aspect of
YM_M4I5S3S_6 telephony since it will assess the capability of various
YM_M4I5S3S_7 to meet human expectations. It will also be a measure of
YM_M4I5S3S_8 beyond compliance to the YM_M4I5S3S_9 protocol.

_________________________________________________________
Do You Yahoo!?
Get your free @yahoo.com address at http://mail.yahoo.com

---------

---

From: "Hoefelmeyer, Ralph (MCI)" <[email-masked]>
Date: Thu, 15 Apr 1999 17:34:15 -0000

Henri,
The predominant testing method is to determine interoperability between
differing vendors equipment.  Simply hook up the devices to a LAN, and
across a small WAN, set up the IP addresses correctly, and start calling.
Measure call setup times, time to process an address translation, time to
set up the RTP channel, etc.  One uses LAN analyzers to measure the various
latencies, and to view the protocol exchanges.  Hewlett-Packard has an H.323
analyzer in Beta now.  I do not know of a SIP analyzer yet.
This is for an IP centric network implementation.  For testing between a
circuit switched network and a packet based network, I do not know what one
uses to watch telephony switch message exchanges.
The problem in the SIP world is that there are lots of prototypes and no
product on the market that I can buy today.   If someone knows differently,
please let me know ASAP. 
There are some H.323 devices, but even here, there is not much for
production devices.  Erricson has an H.323 Gatekeeper in use in Norway, but
it is an NT box. There are other H.323 gatekeepers out there, but they have
all of the problems one expects with H.323/RAS etc. (elemedia, RadVision,
Cisco). Some are simply software elements one places on a server, others are
proprietary hardware and software solutions.
You should keep a watch out on the URL
http://www.cs.columbia.edu/~hgs/sip/bakeoff.html   for the results of the
SIP Bake-off, an interoperability event held last week in New York City.

What does "MOS" stand for? 

Cordially,
Ralph

Ralph S. Hoefelmeyer
MCI WorldCom Engineer

> -----Original Message-----
> From:	Henri Setiawan [SMTP:[email-masked]]
> Sent:	Wednesday, April 14, 1999 8:29 PM
> To:	IPTEL IETF Mailing List
> Subject:	Measurement and testing parameter
> 
> Dear folks,
>

---

From: Tom Gray <[email-masked]>
Date: Thu, 15 Apr 1999 12:03:53 -0700 (PDT)

Yahoo seems to have a strange sense of humour. its spell checker
changed my English words to some strange internal representaion. Here
is another attempt.

> What does "MOS" stand for? 

MOS is Mean Opinion Score. It is a metric which denotes the perceived
quality of the voice connection. It has five categories going from
Excellent to Unacceptable. This is an important aspect of IP telephony
since it will assess the capability of various
implementations to meet human expectations. It will also be a measure
of interoperability beyond compliance to the signaling protocol.

> M-O-S is Mean Opinion Score. It is a metric
> which denotes the
>IP telephony since it will assess the
> capability of various
>  to meet human expectations. It will
> also be a measure of

---------

---

From: "DelRegno, Christopher (MCI)" <[email-masked]>
Date: Thu, 15 Apr 1999 20:50:37 -0000

Regarding PSTN signalling test equipment, there are many such devices from
HP, etc.  I have not seen one however that can simultaneously test both
sides of a IP/PSTN call.
 
MOS, as noted below, is Mean Opinion Score.  Notice, Opinion.  MOS is an
objective way of measuring the subjective nature of voice quality and the
individual's experience.  MOS, from a provider's perspective should be
measured end-to-end, since this is what the user experience will be based
on.  In codec development, we rely on MOS testing to determine the relative
quality of that codec as compared to PCM.  Further, in network analysis, MOS
can be used to determine the quality of one network technology versus
another, or one carrier's network vs. another.
 
Traditionally, true MOS testing is very expensive.  Most people use more
limited, predictive testing such as DAM testing.  These types of tests
suggest what type of MOS score would result from a true evaluation, without
the cost.  However, DAM testing and the like doesn't guarantee the MOS
score; it is only a predictor, and usually a pretty good one.
 
While codecs may be able to smooth out network impairments within reason,
they cannot smooth out ALL network impairments.  Networks can vary wildly
depending upon architecture, protocols, QOS, etc.  For example, if latency
increases too much, today's echo cancellors may not have a way to
compensate, thereby increasing the echo.  While every other parameter may be
on target, this one problem will significantly affect the overall quality.
 
I am sorry to say that there is no one device that can measure MOS on the
fly, to my knowledge.  Most testing requires the development of test cases
and test content, running that content through pristine networks and
conditions and then comparing the results with the results from varying
codecs, network impairments, etc.  Not necessarily an easy process.
 
Hope that helps.
 
Nick
Christopher N. "Nick" DelRegno 
MCI WorldCom, Access Technology Development 
400 International Pkwy, 1010/041 
Richardson, TX  75081-6606 
Tel: 972-729-3411 
Fax: 972-907-8784 
Email: [email-masked] 

-----Original Message-----
From: Sam Mokbel [mailto:[email-masked]]
Sent: Thursday, April 15, 1999 12:14 PM
To: Hoefelmeyer, Ralph (MCI)
Cc: 'Tom Gray'; 'Henri Setiawan'; IPTEL IETF Mailing List
Subject: Re: Measurement and testing parameter

Sure, the Codec may be able to smooth out a lot of the jitter and delay.
There are limits however to how long you can buffer and introduce additional
engineered delays. I have seen suggestions that delays in excess of 150ms
are not acceptable because they are noticeable to the human ear, assuming
that echo is dealt with properly. Current data traffic patterns and
networking protocols may introduce delays in excess of 150ms. 

Hoefelmeyer, Ralph (MCI) wrote: 

Sam, 
Shouldn't the codec smooth this out?  I am a network and software person,
not a telephony expert ... yet, working on it. 

I would think there would be a buffering function within the codec to smooth
the delay & delay variations out.  Of course, if the delay is to long
(whatever that means), then there is a noticeable gap in the voice stream. 

Thanks,

-- 
Sam Mokbel 
University of Toronto 
External Networking and Facilities Management 
4 Bancroft Ave., Room 101 
Toronto, Ontario M5S 1C1 
Tel: (416) 978-3328 Fax: (416) 978-6620

---

From: "Tom-PT Taylor" <[email-masked]>
Date: Tue, 20 Apr 1999 04:51:32 -0500

ITU-T Study Group 12 are the experts on this stuff.  For further
information, have a look at
http://www.itu.int/itudoc/itu-t/com12/index.html

> Sent:	Tuesday, April 20, 1999 2:38 AM
> To:	Perry Erick Renaldo; IPtel Forum
> Thanks for your help.
> By the way, is there anyone who have the standard test methodology either
> it
> was release by IETF or other standard body or proprietary by vendor ?
> Please
> share with me.
> 
> Regards,
> 
> Henri S
> 
> ----- Original Message -----
> From: Perry Erick Renaldo <[email-masked]>
> To: IPtel Forum <[email-masked]>
> Sent: Tuesday, April 20, 1999 1401
> > Henry,
> > If you want to test data-net product in order to get a clearer view on
> > its performance you might need a standard test methodology. Robert
> > Mandeville has developed a standard test methodology and approved by the
> > IETF under the spell of RFC 2285. Might be a help.
> >
> >
> >
> > ---------

---------

---

From: Tom Gray <[email-masked]>
Date: Tue, 27 Apr 1999 16:08:02 -0700 (PDT)

MOS is a subjective measurement of speech quality. It will be assessed
for all sorts of network conditions. For the PSTN network, the most
significant impairments found were loss, noise and echo so MOS is set
for them. Other impairments have only an almost negligible effect
compared to the big 3. Jitter is accomodated by delay so I would assume
that it would fall under that parameter. There would probably have to
be measurements to understand the effects of other network and codec
impairments as well. After that is understood, relationships can be
established so that thre can be a repeatable predidctions which will
translate between network conditions and human perceived quality.

--- Henri Setiawan <[email-masked]> wrote:
> Dear Tom,
> 
> Thank you for your respon regarding my question on
> measurement and testing
> parameter, on my understanding tha MOS is not the
> only parameter to test
> VoIP products. So, what do you think about the other
> parameter ? Like delay,
> jitter, Codec or others QoS parameter ? According to
> you if we are only the
> user, what we must aware on this technology ?
> Testing the products one by
> one or just waiting until the technology is mature ?
> 
> By the way what do you think about using VoIP as
> asubstitute for to date
> exchange ? Do you have any concept on this ?
> 
> Thanks
> 
> Henri S
> 
> From: Tom Gray <[email-masked]>
> To: 'Tom Gray' <[email-masked]>; 'Henri
> Setiawan'
> <[email-masked]>; IPTEL IETF Mailing List
> <[email-masked]>
> Sent: Friday, April 16, 1999 2324
> Subject: RE: Measurement and testing parameter
> 
> 
> >
> >
> > It should be remembered that MOS is based on the.
> subjective opinions
> > of real people about the quality of a voice
> connection. It is therefore
> > based on their expectations of the service. With
> experience these
> > expectations tend to rise and the expectations of
> the performance of
> > teh PSTN network have risen over the years making
> MOS more stringent as
> > time goes by.
> >
> > An MOS for IP tlephony needs to be developed to
> coincde with the
> > expecations of the people who use it and the
> decision makers who will
> > buy it. It is likely that these people, given IP
> telephony's other
> > advantages, will not expect or require the same
> level of service from
> > IPTel as the PSTN. Direct use of a PSTN MOS is
> probably not predictive
> > or useful.
> >
> > Tom Gray
> >
> >

---------