Subject: Re: [xrblock] 答复: 答复: General comments on xrblock metrics drafts

From: [email-masked]
Date: Thu, 3 Nov 2011 16:02:50 +0800 (CST)

Agree, thanks for your clarification.

Cheers!

Jing

------------------ 原始邮件 ------------------

发件人：Qin Wu

时　间：2011/11/03 14:54:52 星期四

收件人：Schwarz, Albrecht (Albrecht)

, [email-masked], Alan Clark

抄送人：xrblock

主　题：Re: [xrblock] 答复: 答复: General comments on xrblock metrics drafts

Jing:

Albrecht is right. "DSP" is not specification term but implmentation term.

The metric we are talking about here is QoS/QoE Metric and is referred to each measurement result carried in the same XR Block payload.

we can refer to RFC6390 for QoS/QoE metric explaination. This is also what charter said we should get alignment with PMOL work. The PMOL

work is referred to as RFC6390.

Regards!

-Qin

----- Original Message -----

From:
Schwarz, Albrecht (Albrecht)

To:
[email-masked] ;
Alan Clark ;
Qin Wu

Cc:
xrblock

Sent: Wednesday, November 02, 2011 7:10 PM

Subject: RE: [xrblock] 答复: 答复: General comments on xrblock metrics drafts

Folks,

this is getting more and more confusing.

Jing,

don't mix up implementation with specification. This is firstly a specification issue, looks like that the terminology is not sufficiently precisely defined.

Further: separate measurements from reporting, both are different functions.

All considered (by RTCP XR) metrics could be modelled as time-dependent stochastical variables X(t).

Using such an approach you may describe the observation, measurement, estimation or/and the generation of statistics based on that variable.

If the term 'metric' is already defined, please provide a reference to a correspondent RFC.

You may also use ITU-T E.800 as baseline, in particular the terms parameter, characteristic, measure, metric, etc.

And there would be finally a difference between a sample or an average-based (like over a single interval, or over all intervals (= cumulative) value of a particular performance parameter.

Regards,

Albrecht

From: [email-masked] [mailto:[email-masked]] On Behalf Of [email-masked]
Sent: Mittwoch, 2. November 2011 08:27
To: Alan Clark; Qin Wu
Cc: xrblock
Subject: Re: [xrblock] 答复: 答复: General comments on xrblock metrics drafts

I would rather like to regard PPDV described in RFC3550 as cumulative metric since it rely on all the past observation as input.

Also if the digital signal processors (DSPs) can be used to calculate metrics, both Interval metric and cumulative can be calculated and reported immedieately.

Cheers!

-Jing

------------------ 原始邮件 ------------------

发件人：Alan Clark

时　间：2011/11/02 05:28:30 星期三

收件人：[email-masked], Qin Wu

抄送人：xrblock

主　题：Re: 答复: [xrblock] 答复: General comments on xrblock metrics drafts

Jing

I’m not describing how the metric is calculated but what it refers to.. So “Sample” in this case means the sampled value of a continuously varying metric, such as PPDV or audio signal level.

Best Regards

Alan

On 11/1/11 4:33 AM, "[email-masked]" <[email-masked]> wrote:

I agree more samples, more accurate of the metric.

but my interpretation to what Alan said here is if we have 6 samples in one designated interval(x0,x1,x2,x3,x4,x5), the sampling interval bewteen two adjacent samples is set different, in this case, how to calculate average?

Cheers!

-Jing

------------------ 原始邮件 ------------------

发件人：Qin Wu
时　间：2011/11/01 16:17:07 星期二
收件人：Alan Clark , [email-masked]
抄送人：xrblock
主　题：答复: [xrblock] 答复: General comments on xrblock metrics drafts

Interval averages can be quite misleading if the measurements are strongly time varying, which PDV typically is.

[Qin]: I agree people often confuse some statistics measurement that use sample with average. But Sample is not a new concept and widely used in the RFC 3550(e.g., page 15 for the definition of timestample), for example,

you can sample SSRC, you can sample packet, the sample instant is choosen as reference point. jitter calculation is based on the RTP timestamp which
represents the instant when the first data in the packet was sampled. The sample implementation is allowed in the RFC3550.

Also, in my understanding the sample will be used only when it is impractical to keep track of all the packets.

So it looks to me if you gather enough samples within interval, you will get a more accurate average or other statistics measurment that does not rely on per-packet statistics.

Best Regards

Alan

On 10/31/11 5:23 AM, "[email-masked] <https://imail.huawei.com/OWA/UrlBlockedError.aspx> " <[email-masked] <https://imail.huawei.com/OWA/UrlBlockedError.aspx> > wrote:

Hi, Alan:

Question for clarification.

Isn't the sampled metric same as the Interval metric? Is the sampled metric only applied to per-packet reporting ? what's the difference between sampled metric and non-sampled metric?

what's the difference between sampled metric and Interval metric? Would you like to clarify more?

Cheers!

Jing

------------------ 原始邮件 ------------------

发件人：Alan Clark
时　间：2011/10/31 06:51:03 星期一
收件人：[email-masked] <https://imail.huawei.com/OWA/UrlBlockedError.aspx>
抄送人：
主　题：[xrblock] General comments on xrblock metrics drafts

1. Cumulative, Interval and Sample metrics
The drafts current have one bit assigned for Cumulative/Interval indication.... Some metrics are however sampled and don’t have validity for an interval or a call/session; an example of this is PPDV, which is a sampled value of a running average and only has validity for the last 200-300ms before the value is sampled.
Proposal - increase this to two bits to allow for the indication of a Sampled value.

---

From: Qin Wu <[email-masked]>
Date: Thu, 03 Nov 2011 16:35:21 +0800

Hi, Jing:
You have a good clarification. Let me rephrase a little a bit below.

For some metrics, it may be not pratical to gather all the packets during 
the whole measurement duration, espcially we have a very longer 
measurement Interval, e.g., 
you may gather millions of packets during 10 measurement interval.
 It may be very expensive to calculate each metric. So we may sample 
a subset of packets at the end of each measurement Interval.
also we may sample a subset of packet in the middle of each measurement Interval.
 Comparing with gathering all the packets, Sampling metic may be not accurate as
cumulative metric but saves a lot of computation overhead and in some degree 
present key feature of transport impairment.

E.g.,Let's assume we have 10 measurement Interval, we may take 100 samples 
during each measurement Interval, use  algorithm like RFC3550 PPDV
 to calculate stochastical statistics (J1, J2, ...J10)value based on these 100 
samples in each measurement Interval.

Another example, we may only gather the most recent packets from the recent 
measurement interval to the  previous measurement interval and then use RFC
3550 PPDV to calculate calculate stochastical statistics (J1, J2, ...J10)value. 
This is also a example of sampling.

Regards!
-Qin 
  ----- Original Message ----- 
  From: [email-masked] 
  To: Alan Clark ; Qin Wu 
  Cc: xrblock 
  Sent: Thursday, November 03, 2011 4:00 PM
  Subject: Re: Re: 答复: [xrblock] 答复: General comments on xrblock metrics drafts

  To my understanding, you can also apply RFC3550 PPDV algorithm to cumulative metric. 
  If you want to obtain cumulative value, you should calculate PDV cumulative value J(k) from
   the time interval 1 to the time interval k  based on J(1),J(2)...J(k-1) since J(i)=(15*J(i-1)+D)/16, 
  the measurement Interval include from the time interval 1 to time interval k. J(1) is cumulative 
  value during interval 1, J(2) is the cumulative value between time interval 1 and time interval 2. 

  If you only apply RFC3550 PPDV algorithm to at the end of interval, the samples you get is
   only valid at the end of interval you measured, then you may get sampled metric value.  

  So RFC3550 PPDV algorithm should be independent from whether it is cumulative metric 
  or Interval metric or sample 
  metric. 
   
  BR. 
  Jing 
  ------------------ 原始邮件 ------------------ 

  发件人：Alan Clark 
  时　间：2011/11/03 10:10:07 星期四 
  收件人：[email-masked], Qin Wu 
  抄送人：xrblock 
  主　题：Re: 答复: [xrblock] 答复: General comments on xrblock metrics drafts 

  Jing

  One would generally expect that a cumulative metrics indicates performance from the start of a session to the point in time at which the report is sent (or at least created). This would be meaningful for a parameter such as packet loss, but not PPDV.

  PPDV is a running average of the packet-to-packet delay variation, as in PPDV = (PPDV*15 + X) / 16

  PDV (jitter) is often strongly time varying. If there was a period of high jitter early in the session, the PPDV value would go up during this period however would then drop by 1/16th per packet. If the “high” value of PPDV was “P” then after 100 packets this value would have dropped to 0.002P. So PPDV only relates to the jitter level during the most recent 20-30 packets.

  You “could” measure the average PDV value for an interval or the session using 

  Interval Average PDV = Sum(per packet PDV) / Number of packets

  This would be regarded as an interval or cumulative metric however the RFC3550 PPDV metric is better described as the sampled value of the running average 

  Best Regards

  Alan

  On 11/2/11 3:26 AM, "[email-masked]" <[email-masked]> wrote:

    Cheers! 

    -Jing 

    ------------------ 原始邮件 ------------------ 

    发件人：Alan Clark 

    Jing

    I’m not describing how the metric is calculated but what it refers to... So “Sample” in this case means the sampled value of a continuously varying metric, such as PPDV or audio signal level.

    Best Regards

    Alan

    On 11/1/11 4:33 AM, "[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]> " <[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]> > wrote:

      Cheers! 

      -Jing 

      ------------------ 原始邮件 ------------------ 

      发件人：Qin Wu 
      时　间：2011/11/01 16:17:07 星期二 
      收件人：Alan Clark , [email-masked]..cn <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]> 
      抄送人：xrblock 
      主　题：答复: [xrblock] 答复: General comments on xrblock metrics drafts 

      Best Regards

      Alan

      On 10/31/11 5:23 AM, "[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]> <https://imail.huawei.com/OWA/UrlBlockedError.aspx> " <[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]> <https://imail.huawei.com/OWA/UrlBlockedError.aspx> > wrote:

        Hi, Alan: 

        Cheers! 

        Jing 

        ------------------ 原始邮件 ------------------ 

        发件人：Alan Clark 
        时　间：2011/10/31 06:51:03 星期一 
        收件人：[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/收件人：[email-masked]> <https://imail.huawei.com/OWA/UrlBlockedError.aspx> 
        抄送人： 
        主　题：[xrblock] General comments on xrblock metrics drafts 

        1. Cumulative, Interval and Sample metrics
        The drafts current have one bit assigned for Cumulative/Interval indication..... Some metrics are however sampled and don’t have validity for an interval or a call/session; an example of this is PPDV, which is a sampled value of a running average and only has validity for the last 200-300ms before the value is sampled.
        Proposal - increase this to two bits to allow for the indication of a Sampled value.