Subject: Re: [xrblock] 答复: Re: 答复: General comments on xrblock metrics drafts

From: Qin Wu <[email-masked]>
Date: Fri, 04 Nov 2011 09:26:32 +0800

Re: [xrblock] 答复: Re: 答复: General comments on xrblock metrics draftsHi, Alan:
I tend to agree with you. But currently I don't see the sampled metric is applied to all the metrics except PDV metric, 
So my proposal is to add one bit in PDV metric block header to indicate if sample will be used. 
If there is any future metric that need to support sample, we can add indication bit later.

Regards!
-Qin
  ----- Original Message ----- 
  From: Alan Clark 
  To: Qin Wu ; [email-masked] 
  Cc: xrblock 
  Sent: Thursday, November 03, 2011 8:26 PM
  Subject: Re: [xrblock] 答复: Re: 答复: General comments on xrblock metrics drafts

  Qin

  You cannot derive the value of a sampled metric from two interval metrics.

  A sampled metric is useful for measuring and reporting on parameters that vary in time. Sometimes it is acceptable to report the min/max/average of a time varying parameter and sometimes you may want to report the value at points in time.

  At risk of causing complete confusion - I’m going to use a temperature example.  

  (i) Between 2pm and 3pm the average temperature was 10 degrees C, is an interval metric
  (ii) Between 2pm and 3pm the maximum temperature was 11 degrees C, is an interval metric
  (ii) The temperature at 3pm is 2 degrees C is a sampled measurement

  It is remarkably easy to add one bit to the XRBlock head at this time to allow for current or future metrics that are sampled. Given that there are existing RTP examples of this type of metric then we know that it is already useful to report sampled values. 

  Best Regards

  Alan

  On 11/3/11 3:15 AM, "Qin Wu" <[email-masked]> wrote:

    Hi, Alan:
    Thank for your clarification on the difference between Interval metric, Cumulative metric the existing draft described and sampled metric you proposed.
    It looks Sample metric is established upon the Interval metric and present different features of any transport impairments from Culmulative metric.
    e.g., we have a list of Interval metric
    x1, x2, x3,x4,x5,x6,x7,x8,x9,x10
    The sample metric is as follows:
    S1,S2,S3,S4,S5,S6,S7,S8,S9,S10
    So the sampled metic can be calculated using the following example:
    S1=x1; S2= func(x1,x2); S3=func(x2,x3); S4=func(x3,x4);...
    In this example, Si only rely on the xi and xi-1.
     
    So the question is:
    Does sample metric fix the problem you raised below for Cumulative metric?
     
    Also what's the sampled algorithm used for calculating such metric and presented at the end of each Interval? the algorithm similar to PPDV? or any algorithm we can find?
    Do you think measurement Interval for Interval metric is same as one for Sampled metric?
     
    It looks to me that it is not that simple to just add one more bit in the XRBlock header to indicate Sampled metric is used? You should make sure the measurement interval and specific sampled algorithm?
    But Interval metric and Cumulative metric don't have such issue.

    Regards!
    -Qin

      ----- Original Message ----- 
       
      From:  Alan Clark <mailto:[email-masked]>  
       
      To: Qin Wu <mailto:[email-masked]>  ; [email-masked] 
       
      Cc: xrblock <mailto:[email-masked]>  
       
      Sent: Thursday, November 03, 2011 10:41  AM
       
      Subject: Re: [xrblock] 答复: Re: 答复: General  comments on xrblock metrics drafts

      Qin

      An interval metric is measured over a time  interval.  A cumulative metric is the same as an interval metric if there  is only one interval.

      A metric for an interval should not be affected  by previous intervals.

      A “sampled” metric is the value of a metric at  the instant of sampling.

      (a) Simple case

      If we consider 10  intervals, each with 1000 packets expected.  Intervals 1 and 3 have 100  packets missing.

      Cumulative packet loss rate, reported at the end of  each interval would be

      10%, 5%, 6.7%, 5%, 4%, 3.3%, 2.9%, 2.5%, 2.2%,  2.0%

      Interval packet loss rate would be
      10%, 0%, 10%, 0%, 0%, 0%,  0%, 0%, 0%, 0%  

      Averaging the interval loss rates gives  (10+10)/10 = 2.0%

      calculating the running average loss rate and  reporting sampled values would give something roughly like

      10%, 3%,  10%, 3%, 1%, 0.1%, 0.01%, 0%, 0%, 0%

      (b) Slightly less simple  case

      Assume the same intervals and loss rates however say that some of  the packets from interval 1 were not actually lost, but arrive in interval 2.   So at the time the packet loss rate was calculated for interval 1 it  appeared to be 8% instead of 10%.

      We don’t typically report negative  loss rates, so the interval loss rate would be reported as 

      8%,  0%.....

      If we compute loss rate cumulatively then this problem would  only occur at the end of the cumulative period, so the loss rates reported  would be

      8%, 5%, ...

      In other words the cumulative metric  corrects itself.

      This would mean that if we added the interval packet  loss rates together the sum may not match the cumulative value.

      Best  Regards

      Alan

      On 11/2/11 3:11 AM, "Qin Wu" <[email-masked]>  wrote:

        I  tend to agree with you. But It looks to me Interval metric also can be used  to calculate cumulative charastertistics. The only difference to the  Cumulative metric is one has short interval, one has longer  interval.

        Am I right? 

        Or  should we view  Interval metric as a metric that is used to report instant value at the end  of the interval set for the interval metric? in that case, there will be no  difference between Iinteval metric and Sampled  metric,

        Or should we view Interval metric as a basic  metric that can be directly measured and calculated and does not rely on  other metric.

        which one is correct? 

        Before  answering this question,  I think introducing new term "sampled metric"  is very  confused.

        Regards!

        -Qin

------------------------------------------------------------------------
        发件人: [email-masked] [[email-masked]]
        发送时间:  2011年11月2日 14:59
        到: Alan Clark; Qin Wu
        Cc:  xrblock
        主题: Re: Re: 答复: [xrblock] General comments on xrblock  metrics drafts

        I know you intend to seperate Sampled metric from  Interval Metric. However according to the defintion of Interval metric  definition: 
        " 
        The Interval metric is refered to as the reported  values apply to the most recent measurement 
         interval duration  between successive metrics reports 
        " 
        In your understanding, the  samepled metric is referred to as the measured value is sampled at the end  of interval or apply to the end of interval. 
        Is that what you say?  
        But I am not convinced. It looks to me the sampled metric can be  classified into cumulative metric. 
        Take PPDV as example, the J(i)=  J(i-1)++ (|D(i-1,i)| - J(i-1))/16 
        Isn't J(i) a cumulative metric since  the J(i) is the accumulation period characteristic that combine  J(i-1),J(i-2),..J(0) 
        during cumulative duration? 
        Cheers! 
        Jing  
        ------------------ 原始邮件 ------------------ 

        发件人：Alan Clark  
        时　间：2011/11/02 05:25:01 星期三 
        收件人：[email-masked], Qin Wu  
        抄送人：xrblock 
        主　题：Re: 答复: [xrblock] General comments on xrblock  metrics drafts 

        Jing

        An Interval metric is calculated over  the interval - so packet loss for an interval would be the packets lost  during the interval divided by the number of packets expected. 

        If  the packet loss rate was continuously calculated as a running average  (updated per packet) and the value of this packet loss rate metric was  sampled at the end of the interval, it would be a sampled value of the  metric and not the interval value.

        Hence there is a difference  between an interval metric and a sampled metric, which IMHO means that we  should have a way to state that a metric is Cumulative, Interval or  Sampled.

        Best Regards

        Alan

        On 11/1/11 4:21 AM, "[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>  " <[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>  > wrote:

          Are you saying the sample semantics  has already been incorporate into Interval/Cumulative indication?  

          Cheers! 

          Jing 

          ------------------ 原始邮件  ------------------ 

          发件人：Qin Wu 
          时　间：2011/11/01 15:19:54 星期二  
          收件人：Alan Clark , [email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>   
          抄送人：xrblock 
          主　题：答复: [xrblock] General comments on xrblock  metrics drafts 

          Hi, Alan:

          So the Interval  metric you understand is for per RTP packet monitoring or observation  while the sampled metric is not. 
          In other words,
          a. The Interval  metric is obtained by keeping trace of each RTP packet during a specified  Interval and treat all the sequence number valid.
          b.The sampled metric  is obtained by sampling a sub set of RTP packets during a specified  Interval. 
          Is that what you say?

          However  according to PDV drafts and other similar xrblock metrics drafts,  per-packet statistics doesn't apply to these metrics(See security  consideration section).

          That seems to implicate that both  Interval metrics and cumulative metric are not used for per-packet  accounting but for sampling  use.

          Regards!

          -Qin

----------------------------------------------------------------------

          发件人:  [email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>   [[email-masked]  <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>  ] 代表 Alan Clark [[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>  ]
          发送时间: 2011年10月31日 20:08
          到: [email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>  
          Cc: xrblock
          主题: Re: [xrblock] General comments on  xrblock metrics drafts

          Hi  Jing

          An interval metric should be measured throughout an interval  however a sampled metric is an instantaneous value.

          Using  Jitter/PDV as an example:

          - Interval PDV should be calculated by  measuring the delay variation of each packet and averaging these over the  interval. This requires the implementation to sum the individual  measurements and divide by the number of measurements.

          - PPDV  (RFC3550) is calculated by keeping a running average of the individual  delay variation measurements using a scaling factor of 16 and then  reporting the value of this running average at the end of the interval.  With a scaling factor of 16 the PPDV metric is most strongly affected by  the delay variation of the most recent 10-20 packets and hence is not  representative of the whole interval.

          Interval averages can be  quite misleading if the measurements are strongly time varying, which PDV  typically is.

          Best Regards

          Alan

          On 10/31/11 5:23  AM, "[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>   <https://imail.huawei.com/OWA/UrlBlockedError.aspx>  " <[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>   <https://imail.huawei.com/OWA/UrlBlockedError.aspx>  > wrote:

            Hi, Alan: 

            Question for clarification.  

            Isn't the sampled metric same as the Interval metric? Is the  sampled metric only applied to per-packet reporting ? what's the  difference between sampled metric and non-sampled metric? 

            what's  the difference between sampled metric and Interval metric? Would you  like to clarify more? 

            Cheers! 

            Jing  

            ------------------ 原始邮件 ------------------ 

            发件人：Alan  Clark 
            时　间：2011/10/31 06:51:03 星期一 
            收件人：[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/收件人：[email-masked]>   <https://imail.huawei.com/OWA/UrlBlockedError.aspx>  
            抄送人： 
            主　题：[xrblock] General comments on xrblock metrics drafts  

            1. Cumulative, Interval and Sample metrics
            The drafts  current have one bit assigned for Cumulative/Interval indication....  Some metrics are however sampled and don’t have validity for an interval  or a call/session; an example of this is PPDV, which is a sampled value  of a running average and only has validity for the last 200-300ms before  the value is sampled.
            Proposal - increase this to two bits to allow  for the indication of a Sampled value.  

--------------------------------------------------------------------------

      _______________________________________________
      xrblock mailing  list
      [email-masked]
      https://www.ietf.org/mailman/listinfo/xrblock

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Thu, 10 Nov 2011 10:29:24 -0800

(as xrblock chair)

Forwarding to avtcore list as the comments are in regard to an existing AVTCORE working group draft.
I have cc'd xrblock such that folks on xrblock know to post any further comments on this thread to avtcore.

Cheers,
Charles

-----Original Message-----
From: Qin Wu [mailto:[email-masked]] 
Sent: Wednesday, November 09, 2011 4:37 PM
To: Charles Eckel (eckelcu); Alan Clark; [email-masked]
Cc: xrblock
Subject: Re: [xrblock]答复: Re: 答复: General comments on xrblock metrics drafts

Yes, that's what I think to add one such definition to draft-ietf-avtcore-monarch.
Thank for your reminding.

Regards!
-Qin
----- Original Message ----- 
From: "Charles Eckel (eckelcu)" <[email-masked]>
To: "Qin Wu" <[email-masked]>; "Alan Clark" <[email-masked]>; <[email-masked]>
Cc: "xrblock" <[email-masked]>
Sent: Thursday, November 10, 2011 5:56 AM
Subject: Re: [xrblock]答复: Re: 答复: General comments on xrblock metrics drafts

> (As an individual)
> 
> It looks good to me as well. Furthermore, it seems these comments are applicable to monitoring metrics in general, not just those for QoE. That being the case, is the intention to add these definitions into draft-ietf-avtcore-monarch?
> 
> Cheers,
> Charles
> 
>> -----Original Message-----
>> From: [email-masked] [mailto:[email-masked]] On Behalf Of Qin Wu
>> Sent: Tuesday, November 08, 2011 9:51 PM
>> To: Alan Clark; [email-masked]
>> Cc: xrblock
>> Subject: Re: [xrblock]答复: Re: 答复: General comments on xrblock metrics drafts
>> 
>> Your proposal looks good to me.
>> Thanks!
>> 
>> Regards!
>> -Qin
>> 
>> Sent: Tuesday, November 08, 2011 9:36 PM
>> Hi Qin
>> This is an alternative to interval or sample and hence the three of them need to be enumerated.
>> 
>> Interval Metric flags (I): 2 bits
>> 
>>       This field is used to indicate whether the ????????????
>>       metrics block is an Interval, Cumulative or Sampled report, that is,
>>       whether the reported values apply to the most recent measurement
>>       interval duration between successive metrics reports (I=10) (the
>>       Interval Duration) or to the accumulation period characteristic of
>>       cumulative measurements (I=00) (the Cumulative Duration) or
>>       to the value of a continuously measured or calculated that has been
>>       sampled at end of the interval (I=01) (Sampled Value).
>> 
>> Best Regards
>> 
>> Alan
>> 
>> 
>> On 11/7/11 8:43 PM, "Qin Wu" <[email-masked]> wrote:
>> 
>> 
>> 
>> Hi, Alan:
>> Based on our discussion, would you like to propose the change to the drafts regarding
>> sample metric?
>> e.g., what's your proposed change to the Interval metric flag? or just allocate additional
>> falg to sample?
>> Also it will be great you provide the clear definition of sample metric.
>> My understanding to sample metrics is:
>> "
>> Sample metrics
>> 
>> It is referred to the metrics of which the reported values only
>> apply to the sample instant which is any time of the measurement duration.
>> "
>> If this is not precise, please correct me. Thanks!
>> 
>> Regards!
>> -Qin
>> 
>> 
>> 
>> Sent: Friday, November 04, 2011 10:07 AM
>> 
>> Hi Qin
>> The goal in establishing a framework  should be to make provision for things that
>> you would reasonably expect.   There are quite a few continuously varying parameters that could be
>> reported using a sampled approach - audio signal level, noise level, video  bandwidth to name a few.
>> 
>> The first part of the header comprises
>> - Block type (8 bits)
>> - Interval/Cumulative ( 1 bit)
>> - Tag (3  bits)
>> - Reserved (4 bits)
>> 
>> Expanding the Interval/Cumulative to 2  bits still leaves 3 reserved bits, so this
>> would not break the  budget.
>> 
>> Best Regards
>> 
>> Alan
>> 
>> 
>> 
>> 
>> On 11/3/11 9:26 PM,  "Qin Wu" <[email-masked]>  wrote:
>> 
>> 
>> 
>> 
>> Hi, Alan:
>> I tend to agree with you. But currently  I don't see the sampled metric is
>> applied to all the metrics except PDV  metric,
>> A sampled metric is useful for  measuring and  reporting on parameters
>> that vary in time. Sometimes it is   acceptable to report the min/max/average of a time varying
>> parameter  and  sometimes you may want to report the value at points in  time.
>> 
>> It is remarkably easy to add one bit  to  the XRBlock head at this time
>> to allow for current or future metrics that   are sampled. Given that there are existing RTP examples
>> of this type  of metric  then we know that it is already useful to report sampled  values.
>> 
>> Best  Regards
>> 
>> Alan
>> 
>> Thank for your clarification on the   difference between Interval
>> metric, Cumulative metric the existing  draft  described and sampled metric you proposed.
>> It looks  Sample metric is  established upon the Interval metric
>> and present  different features of any  transport impairments from Culmulative  metric.
>> Also what's the  sampled  algorithm used for calculating such
>> metric and presented at the end   of each Interval? the algorithm similar to PPDV? or any algorithm
>> we can  find?
>> It looks to  me that it is not that  simple to just add one more
>> bit in the  XRBlock header to indicate Sampled  metric is used? You should make  sure the measurement
>> interval and specific  sampled  algorithm?
>> An  interval metric is  measured over a time  interval.  A
>> cumulative  metric  is the same as an interval metric if there  is only one   interval.
>> 
>> If we  consider 10  intervals, each with  1000 packets
>> expected.   Intervals 1 and 3 have 100   packets missing.
>> 
>> calculating the running average  loss rate and   reporting
>> sampled values would give something roughly   like
>> 
>> Assume the  same intervals and loss  rates however say that
>> some of  the  packets from interval 1 were not  actually lost, but arrive in  interval 2.   So at the
>> time the  packet loss rate was  calculated for interval 1 it  appeared to be 8%  instead of  10%.
>> 
>> If we compute loss rate cumulatively then  this  problem
>> would  only occur at the end of the cumulative  period, so the  loss rates reported  would be
>> 
>> 8%, 5%,  ...
>> 
>> This would mean  that if we added the interval  packet  loss
>> rates together the sum may  not match the  cumulative value.
>> 
>> Best    Regards
>> 
>> Alan
>> 
>> 
>> I   tend to agree with you. But It  looks to me
>> Interval metric also  can be used  to  calculate cumulative charastertistics. The only  difference to
>> the  Cumulative metric is one has short interval, one  has  longer  interval.
>> 
>> Am I right?
>> 
>> 
>> 
>> Or    should we view  Interval metric as a metric that
>> is  used to  report instant value at the end  of the interval  set for the  interval metric? in that
>> case, there will be no   difference between  Iinteval metric and Sampled   metric,
>> 
>> 
>> 
>> Or should  we view Interval  metric as a basic  metric
>> that can be directly  measured  and calculated and does not rely on  other   metric.
>> 
>> which one is correct?
>> 
>> 
>> 
>> Before   answering this question,   I think
>> introducing new term  "sampled metric"  is  very    confused.
>> 
>> 
>> 
>> Regards!
>> 
>> -Qin
>> 
>> 
>> 
>> 
>> 
>> ________________________________
>> 
>> I know you  intend to seperate  Sampled metric from
>> Interval Metric. However  according  to the defintion of Interval metric  definition:
>> "
>> The Interval metric is refered to as the reported
>> values apply  to the most recent measurement
>> In your understanding, the   samepled  metric is
>> referred to as the measured value is sampled at  the  end  of interval or apply to the end of
>> interval.
>> Is that   what you say?
>> But I am not convinced. It looks to me  the  sampled
>> metric can be  classified into cumulative  metric.
>> Isn't J(i) a cumulative metric  since  the J(i) is the
>> accumulation period characteristic  that combine   J(i-1),J(i-2),..J(0)
>> An Interval metric is calculated   over  the interval
>> - so packet loss for an interval would  be the  packets lost  during the interval divided by the
>> number of packets  expected.
>> 
>> If  the packet loss  rate was continuously  calculated
>> as a running average   (updated per packet) and the value  of this packet loss  rate metric was
>> sampled at the end of the  interval, it  would be a sampled value of the  metric and not the
>> interval value.
>> 
>> Hence there is a difference   between an  interval
>> metric and a sampled metric, which  IMHO means that we   should have a way to state that a  metric is
>> Cumulative, Interval  or  Sampled.
>> 
>> Best  Regards
>> 
>> Alan
>> 
>> 
>> On 11/1/11  4:21 AM, "[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    "
>> <[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    > wrote:
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 收件人：Alan  Clark , [email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
>> 抄送人：xrblock
>> a. The Interval  metric is  obtained by keeping trace of  each RTP packet during a  specified
>> Interval and treat all the  sequence number  valid.
>> However    according to PDV drafts and other similar xrblock  metrics  drafts,  per-packet
>> statistics doesn't apply to  these metrics(See  security  consideration  section).
>> 
>> 
>> That seems to  implicate that both   Interval metrics and cumulative metric are  not used  for
>> per-packet  accounting but for sampling    use.
>> 
>> 
>> 
>> 
>> 
>> Regards!
>> 
>> 
>> -Qin
>> 
>> 
>> 
>> 
>> 
>> 
>> ________________________________
>> 
>> 
>> 发件人:  [email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
>> [[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    ] 代表
>> Alan Clark [[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    ]
>> 发送时间: 2011年10月31日 20:08
>> 到: [email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
>> Cc: xrblock
>> - Interval PDV should  be calculated by  measuring  the delay variation of each  packet and
>> averaging these over the   interval. This  requires the implementation to sum the individual
>> measurements and divide by the number of  measurements.
>> 
>> -  PPDV  (RFC3550) is calculated by  keeping a running average of the  individual  delay
>> variation measurements using a scaling factor  of 16 and then   reporting the value of this running
>> average at  the end  of the interval.  With a scaling factor of 16 the PPDV   metric is most strongly
>> affected by  the delay  variation of the  most recent 10-20 packets and hence is not   representative
>> of the  whole interval.
>> 
>> On  10/31/11  5:23  AM, "[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
>> <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    " <[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
>> <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    > wrote:
>> 
>> 
>> 
>> 
>> 
>> 
>> Hi, Alan:
>> 
>> Isn't the sampled metric same  as the  Interval metric? Is the  sampled metric only  applied to
>> per-packet reporting ? what's the   difference between sampled  metric and non-sampled  metric?
>> 
>> 收件人：[email-masked]
>> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/收件人：[email-masked]>
>> <https://imail.huawei.com/OWA/UrlBlockedError.aspx>
>> 抄送人：
>> The drafts  current  have one bit assigned for  Cumulative/Interval  indication....  Some
>> metrics are however  sampled and  don’t have validity for an interval  or a   call/session; an example
>> of this is PPDV, which is a  sampled value   of a running average and only has  validity for the last
>> 200-300ms before  the value is  sampled.
>> ________________________________
>> 
>> 
>>