Subject: Re: [xrblock] 答复: Re: 答复: General comments on xrblock metrics drafts

From: Qin Wu <[email-masked]>
Date: Wed, 09 Nov 2011 13:50:36 +0800

Re: [xrblock] 答复: Re: 答复: General comments on xrblock metrics draftsYour proposal looks good to me.
Thanks!

Regards!
-Qin
  ----- Original Message ----- 
  From: Alan Clark 
  To: Qin Wu ; [email-masked] 
  Cc: xrblock 
  Sent: Tuesday, November 08, 2011 9:36 PM
  Subject: Re: [xrblock] 答复: Re: 答复: General comments on xrblock metrics drafts

  Hi Qin

  This is an alternative to interval or sample and hence the three of them need to be enumerated. 

  Interval Metric flags (I): 2 bits

        This field is used to indicate whether the ????????????
        metrics block is an Interval, Cumulative or Sampled report, that is,
        whether the reported values apply to the most recent measurement
        interval duration between successive metrics reports (I=10) (the
        Interval Duration) or to the accumulation period characteristic of
        cumulative measurements (I=00) (the Cumulative Duration) or
        to the value of a continuously measured or calculated that has been
        sampled at end of the interval (I=01) (Sampled Value).

  Best Regards

  Alan

  On 11/7/11 8:43 PM, "Qin Wu" <[email-masked]> wrote:

    Hi, Alan:
    Based on our discussion, would you like to propose the change to the drafts regarding sample metric?
    e.g., what's your proposed change to the Interval metric flag? or just allocate additional falg to sample?
    Also it will be great you provide the clear definition of sample metric.
    My understanding to sample metrics is:
    "
    Sample metrics

    It is referred to the metrics of which the reported values only 
     apply to the sample instant which is any time of the measurement duration. 
    "
    If this is not precise, please correct me. Thanks!
     
    Regards!
    -Qin

      ----- Original Message ----- 
       
      From:  Alan Clark <mailto:[email-masked]>  
       
      To: Qin Wu <mailto:[email-masked]>  ; [email-masked] 
       
      Cc: xrblock <mailto:[email-masked]>  
       
      Sent: Friday, November 04, 2011 10:07 AM
       
      Subject: Re: [xrblock] 答复: Re: 答复: General  comments on xrblock metrics drafts

      Hi Qin

      The goal in establishing a framework  should be to make provision for things that you would reasonably expect.   There are quite a few continuously varying parameters that could be  reported using a sampled approach - audio signal level, noise level, video  bandwidth to name a few. 
       
      The first part of the header comprises  
      - Block type (8 bits)
      - Interval/Cumulative ( 1 bit)
      - Tag (3  bits)
      - Reserved (4 bits)

      Expanding the Interval/Cumulative to 2  bits still leaves 3 reserved bits, so this would not break the  budget.

      Best Regards

      Alan

      On 11/3/11 9:26 PM,  "Qin Wu" <[email-masked]>  wrote:

        Hi, Alan:
        I tend to agree with you. But currently  I don't see the sampled metric is applied to all the metrics except PDV  metric, 
        So my proposal is to add one bit in PDV metric block header to  indicate if sample will be used. 
        If there is any future metric that need  to support sample, we can add indication bit  later.
         
        Regards!
        -Qin

          ----- Original Message -----  
           
          From:  Alan Clark <mailto:[email-masked]>   
           
          To: Qin Wu <mailto:[email-masked]>  ;  [email-masked]  
           
          Cc: xrblock <mailto:[email-masked]>   
           
          Sent: Thursday, November 03, 2011 8:26   PM
           
          Subject: Re: [xrblock] 答复: Re: 答复: General   comments on xrblock metrics drafts

          Qin

          You  cannot derive the value of a sampled  metric from two interval  metrics.

          A sampled metric is useful for  measuring and  reporting on parameters that vary in time. Sometimes it is   acceptable to report the min/max/average of a time varying parameter  and  sometimes you may want to report the value at points in  time.

          At risk  of causing complete confusion - I’m going to  use a temperature example.   

          (i) Between 2pm and 3pm the  average temperature was 10 degrees  C, is an interval metric
          (ii)  Between 2pm and 3pm the maximum temperature  was 11 degrees C, is an  interval metric
          (ii) The temperature at 3pm is 2  degrees C is a  sampled measurement

          It is remarkably easy to add one bit  to  the XRBlock head at this time to allow for current or future metrics that   are sampled. Given that there are existing RTP examples of this type  of metric  then we know that it is already useful to report sampled  values. 

          Best  Regards

          Alan

          On 11/3/11 3:15 AM,  "Qin Wu" <[email-masked]>   wrote:

            Hi, Alan:
            Thank for your clarification on the   difference between Interval metric, Cumulative metric the existing  draft  described and sampled metric you proposed.
            It looks  Sample metric is  established upon the Interval metric and present  different features of any  transport impairments from Culmulative  metric.
            e.g., we have a list of  Interval metric
            x1, x2,  x3,x4,x5,x6,x7,x8,x9,x10
            The sample metric is  as  follows:
            S1,S2,S3,S4,S5,S6,S7,S8,S9,S10
            So the sampled metic can  be  calculated using the following example:
            S1=x1; S2=  func(x1,x2);  S3=func(x2,x3); S4=func(x3,x4);...
            In this  example, Si only rely on the  xi and xi-1.
             
            So the  question is:
            Does sample metric fix the  problem you raised  below for Cumulative metric?
             
            Also what's the  sampled  algorithm used for calculating such metric and presented at the end   of each Interval? the algorithm similar to PPDV? or any algorithm  we can  find?
            Do you think measurement Interval for Interval  metric is same as  one for Sampled metric?
             
            It looks to  me that it is not that  simple to just add one more bit in the  XRBlock header to indicate Sampled  metric is used? You should make  sure the measurement interval and specific  sampled  algorithm?
            But Interval metric and Cumulative metric don't have   such  issue.

            Regards!
            -Qin

              ----- Original Message -----   
               
              From:  Alan Clark <mailto:[email-masked]>    
               
              To: Qin Wu <mailto:[email-masked]>   ;  [email-masked]   
               
              Cc: xrblock <mailto:[email-masked]>    
               
              Sent: Thursday, November 03, 2011  10:41   AM
               
              Subject: Re: [xrblock] 答复:  Re: 答复: General   comments on xrblock metrics  drafts

              Qin

              An  interval metric is  measured over a time  interval.  A cumulative  metric  is the same as an interval metric if there  is only one   interval.

              A metric for an interval should not be affected   by  previous intervals.

              A “sampled” metric is the  value of a metric at   the instant of sampling.

              (a)  Simple case

              If we  consider 10  intervals, each with  1000 packets expected.   Intervals 1 and 3 have 100   packets missing.

              Cumulative  packet loss rate,  reported at the end of  each interval would  be

              10%,  5%, 6.7%, 5%, 4%, 3.3%, 2.9%, 2.5%, 2.2%,    2.0%

              Interval packet loss rate would be
              10%, 0%,  10%, 0%,  0%, 0%,  0%, 0%, 0%, 0%  

              Averaging  the interval loss  rates gives  (10+10)/10 =  2.0%

              calculating the running average  loss rate and   reporting sampled values would give something roughly   like

              10%, 3%,  10%, 3%, 1%, 0.1%, 0.01%, 0%, 0%,  0%

              (b)  Slightly less simple  case

              Assume the  same intervals and loss  rates however say that some of  the  packets from interval 1 were not  actually lost, but arrive in  interval 2.   So at the time the  packet loss rate was  calculated for interval 1 it  appeared to be 8%  instead of  10%.

              We don’t typically report negative  loss  rates,  so the interval loss rate would be reported as 

              8%,    0%.....

              If we compute loss rate cumulatively then  this  problem would  only occur at the end of the cumulative  period, so the  loss rates reported  would be

              8%, 5%,  ...

              In other  words the cumulative metric  corrects  itself.

              This would mean  that if we added the interval  packet  loss rates together the sum may  not match the  cumulative value.

              Best    Regards

              Alan

              On 11/2/11 3:11 AM, "Qin  Wu" <[email-masked]>    wrote:

                I   tend to agree with you. But It  looks to me Interval metric also  can be used  to  calculate cumulative charastertistics. The only  difference to  the  Cumulative metric is one has short interval, one  has  longer  interval.

                Am I right? 

                Or    should we view  Interval metric as a metric that is  used to  report instant value at the end  of the interval  set for the  interval metric? in that case, there will be no   difference between  Iinteval metric and Sampled   metric,

                Or should  we view Interval  metric as a basic  metric that can be directly  measured  and calculated and does not rely on  other   metric.

                which one is correct?  

                Before   answering this question,   I think introducing new term  "sampled metric"  is  very    confused.

                Regards!

                -Qin

----------------------------------------------------------------
                发件人:  [email-masked] [[email-masked]]
                发送时间:    2011年11月2日 14:59
                到: Alan Clark; Qin  Wu
                Cc:   xrblock
                主题: Re: Re: 答复:  [xrblock] General comments on  xrblock  metrics  drafts

                I know you  intend to seperate  Sampled metric from  Interval Metric. However  according  to the defintion of Interval metric  definition: 
                "   
                The Interval metric is refered to as the reported   values apply  to the most recent measurement  
                 interval duration  between  successive metrics  reports 
                " 
                In your understanding, the   samepled  metric is referred to as the measured value is sampled at  the  end  of interval or apply to the end of interval. 
                Is that   what you say?  
                But I am not convinced. It looks to me  the  sampled metric can be  classified into cumulative  metric. 
                Take  PPDV as example, the J(i)=  J(i-1)++  (|D(i-1,i)| - J(i-1))/16  
                Isn't J(i) a cumulative metric  since  the J(i) is the  accumulation period characteristic  that combine   J(i-1),J(i-2),..J(0) 
                during cumulative  duration? 
                Cheers!  
                Jing  
                ------------------  原始邮件 ------------------  

                发件人：Alan Clark   
                时　间：2011/11/02 05:25:01 星期三 
                收件人：[email-masked], Qin  Wu   
                抄送人：xrblock 
                主　题：Re: 答复: [xrblock] General  comments on  xrblock  metrics drafts  

                Jing

                An Interval metric is calculated   over  the interval - so packet loss for an interval would  be the  packets lost  during the interval divided by the  number of packets  expected. 

                If  the packet loss  rate was continuously  calculated as a running average   (updated per packet) and the value  of this packet loss  rate metric was  sampled at the end of the  interval, it  would be a sampled value of the  metric and not the   interval value.

                Hence there is a difference   between an  interval metric and a sampled metric, which  IMHO means that we   should have a way to state that a  metric is Cumulative, Interval  or  Sampled.

                Best  Regards

                Alan

                On 11/1/11  4:21 AM, "[email-masked]  <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    " <[email-masked]  <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    > wrote:

  Are you saying the sample   semantics  has already been incorporate into  Interval/Cumulative  indication?   

  Cheers! 

  Jing   

  ------------------ 原始邮件  ------------------   

  发件人：Qin Wu 
  时　间：2011/11/01 15:19:54 星期二   
  收件人：Alan  Clark , [email-masked]   <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>     
  抄送人：xrblock 
  主　题：答复: [xrblock] General  comments on  xrblock  metrics drafts  

  Hi, Alan:

  So the  Interval   metric you understand is for per RTP packet  monitoring or  observation  while the sampled metric is  not. 
  In other  words,
  a. The Interval  metric is  obtained by keeping trace of  each RTP packet during a  specified  Interval and treat all the  sequence number  valid.
  b.The sampled metric  is obtained by  sampling  a sub set of RTP packets during a specified  Interval.   
  Is that what you say?

  However    according to PDV drafts and other similar xrblock  metrics  drafts,  per-packet statistics doesn't apply to  these metrics(See  security  consideration  section).

  That seems to  implicate that both   Interval metrics and cumulative metric are  not used  for per-packet  accounting but for sampling    use.

  Regards!

  -Qin

------------------------------------------------------------------------------

  发件人:  [email-masked]  <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>     [[email-masked]    <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    ] 代表 Alan Clark [[email-masked]  <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    ]
  发送时间: 2011年10月31日 20:08
  到: [email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    
  Cc: xrblock
  主题: Re: [xrblock]  General  comments on  xrblock metrics  drafts

  Hi  Jing

  An interval metric  should  be measured throughout an interval  however a  sampled metric is  an instantaneous value.

  Using   Jitter/PDV as an  example:

  - Interval PDV should  be calculated by  measuring  the delay variation of each  packet and averaging these over the   interval. This  requires the implementation to sum the individual    measurements and divide by the number of  measurements.

  -  PPDV  (RFC3550) is calculated by  keeping a running average of the  individual  delay  variation measurements using a scaling factor  of 16 and then   reporting the value of this running average at  the end  of the interval.  With a scaling factor of 16 the PPDV   metric is most strongly affected by  the delay  variation of the  most recent 10-20 packets and hence is not   representative of the  whole interval.

  Interval  averages can be  quite misleading  if the measurements  are strongly time varying, which PDV   typically  is.

  Best Regards

  Alan

  On  10/31/11  5:23  AM, "[email-masked] <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>     <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    " <[email-masked]   <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>     <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    > wrote:

  Hi, Alan: 

  Question for   clarification.  

  Isn't the sampled metric same  as the  Interval metric? Is the  sampled metric only  applied to  per-packet reporting ? what's the   difference between sampled  metric and non-sampled  metric? 

  what's  the difference  between  sampled metric and Interval metric? Would you  like to   clarify more? 

  Cheers! 

  Jing    

  ------------------ 原始邮件 ------------------   

  发件人：Alan  Clark 
  时　间：2011/10/31 06:51:03  星期一 
  收件人：[email-masked]  <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/收件人：[email-masked]>     <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    
  抄送人： 
  主　题：[xrblock] General comments on  xrblock metrics  drafts  

  1. Cumulative,  Interval and Sample  metrics
  The drafts  current  have one bit assigned for  Cumulative/Interval  indication....  Some metrics are however  sampled and  don’t have validity for an interval  or a   call/session; an example of this is PPDV, which is a  sampled value   of a running average and only has  validity for the last  200-300ms before  the value is  sampled.
  Proposal - increase  this to two bits to allow   for the indication of a Sampled  value.   

------------------------------------------------------------------

              _______________________________________________
              xrblock  mailing   list
              [email-masked]
              https://www.ietf.org/mailman/listinfo/xrblock

---

From: "Charles Eckel (eckelcu)" <[email-masked]>
Date: Wed, 9 Nov 2011 13:56:14 -0800

(As an individual)

It looks good to me as well. Furthermore, it seems these comments are applicable to monitoring metrics in general, not just those for QoE. That being the case, is the intention to add these definitions into draft-ietf-avtcore-monarch?

Cheers,
Charles

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of Qin Wu
> Sent: Tuesday, November 08, 2011 9:51 PM
> To: Alan Clark; [email-masked]
> Cc: xrblock
> Subject: Re: [xrblock]答复: Re: 答复: General comments on xrblock metrics drafts
> 
> Your proposal looks good to me.
> Thanks!
> 
> Regards!
> -Qin
> 
> 			The goal in establishing a framework  should be to make provision for things that
> you would reasonably expect.   There are quite a few continuously varying parameters that could be
> reported using a sampled approach - audio signal level, noise level, video  bandwidth to name a few.
> 
> 				I tend to agree with you. But currently  I don't see the sampled metric is
> applied to all the metrics except PDV  metric,
> 					A sampled metric is useful for  measuring and  reporting on parameters
> that vary in time. Sometimes it is   acceptable to report the min/max/average of a time varying
> parameter  and  sometimes you may want to report the value at points in  time.
> 
> 					It is remarkably easy to add one bit  to  the XRBlock head at this time
> to allow for current or future metrics that   are sampled. Given that there are existing RTP examples
> of this type  of metric  then we know that it is already useful to report sampled  values.
> 
> 					Best  Regards
> 
> 					Alan
> 
> 						Thank for your clarification on the   difference between Interval
> metric, Cumulative metric the existing  draft  described and sampled metric you proposed.
> 						It looks  Sample metric is  established upon the Interval metric
> and present  different features of any  transport impairments from Culmulative  metric.
> 						Also what's the  sampled  algorithm used for calculating such
> metric and presented at the end   of each Interval? the algorithm similar to PPDV? or any algorithm
> we can  find?
> 						It looks to  me that it is not that  simple to just add one more
> bit in the  XRBlock header to indicate Sampled  metric is used? You should make  sure the measurement
> interval and specific  sampled  algorithm?
> 							An  interval metric is  measured over a time  interval.  A
> cumulative  metric  is the same as an interval metric if there  is only one   interval.
> 
> 							If we  consider 10  intervals, each with  1000 packets
> expected.   Intervals 1 and 3 have 100   packets missing.
> 
> 							calculating the running average  loss rate and   reporting
> sampled values would give something roughly   like
> 
> 							Assume the  same intervals and loss  rates however say that
> some of  the  packets from interval 1 were not  actually lost, but arrive in  interval 2.   So at the
> time the  packet loss rate was  calculated for interval 1 it  appeared to be 8%  instead of  10%.
> 
> 							If we compute loss rate cumulatively then  this  problem
> would  only occur at the end of the cumulative  period, so the  loss rates reported  would be
> 
> 							8%, 5%,  ...
> 
> 							This would mean  that if we added the interval  packet  loss
> rates together the sum may  not match the  cumulative value.
> 
> 							Best    Regards
> 
> 							Alan
> 
> 
> 								I   tend to agree with you. But It  looks to me
> Interval metric also  can be used  to  calculate cumulative charastertistics. The only  difference to
> the  Cumulative metric is one has short interval, one  has  longer  interval.
> 
> 								Am I right?
> 
> 
> 
> 								Or    should we view  Interval metric as a metric that
> is  used to  report instant value at the end  of the interval  set for the  interval metric? in that
> case, there will be no   difference between  Iinteval metric and Sampled   metric,
> 
> 
> 
> 								Or should  we view Interval  metric as a basic  metric
> that can be directly  measured  and calculated and does not rely on  other   metric.
> 
> 								which one is correct?
> 
> 
> 
> 								Before   answering this question,   I think
> introducing new term  "sampled metric"  is  very    confused.
> 
> 
> 
> 								Regards!
> 
> 								-Qin
> 
> 
> 
> 
> 
> ________________________________
> 
> 								I know you  intend to seperate  Sampled metric from
> Interval Metric. However  according  to the defintion of Interval metric  definition:
> 								"
> 								The Interval metric is refered to as the reported
> values apply  to the most recent measurement
> 								In your understanding, the   samepled  metric is
> referred to as the measured value is sampled at  the  end  of interval or apply to the end of
> interval.
> 								Is that   what you say?
> 								But I am not convinced. It looks to me  the  sampled
> metric can be  classified into cumulative  metric.
> 								Isn't J(i) a cumulative metric  since  the J(i) is the
> accumulation period characteristic  that combine   J(i-1),J(i-2),..J(0)
> 								An Interval metric is calculated   over  the interval
> - so packet loss for an interval would  be the  packets lost  during the interval divided by the
> number of packets  expected.
> 
> 								If  the packet loss  rate was continuously  calculated
> as a running average   (updated per packet) and the value  of this packet loss  rate metric was
> sampled at the end of the  interval, it  would be a sampled value of the  metric and not the
> interval value.
> 
> 								Hence there is a difference   between an  interval
> metric and a sampled metric, which  IMHO means that we   should have a way to state that a  metric is
> Cumulative, Interval  or  Sampled.
> 
> 								Best  Regards
> 
> 								Alan
> 
> 
> 								On 11/1/11  4:21 AM, "[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    "
> <[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    > wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 	收件人：Alan  Clark , [email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
> 	抄送人：xrblock
> 	a. The Interval  metric is  obtained by keeping trace of  each RTP packet during a  specified
> Interval and treat all the  sequence number  valid.
> 	However    according to PDV drafts and other similar xrblock  metrics  drafts,  per-packet
> statistics doesn't apply to  these metrics(See  security  consideration  section).
> 
> 
> 	That seems to  implicate that both   Interval metrics and cumulative metric are  not used  for
> per-packet  accounting but for sampling    use.
> 
> 
> 
> 
> 
> 	Regards!
> 
> 
> 	-Qin
> 
> 
> 
> 
> 
> 
> ________________________________
> 
> 
> 	发件人:  [email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
> [[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    ] 代表
> Alan Clark [[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>    ]
> 	发送时间: 2011年10月31日 20:08
> 	到: [email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
> 	Cc: xrblock
> 	- Interval PDV should  be calculated by  measuring  the delay variation of each  packet and
> averaging these over the   interval. This  requires the implementation to sum the individual
> measurements and divide by the number of  measurements.
> 
> 	-  PPDV  (RFC3550) is calculated by  keeping a running average of the  individual  delay
> variation measurements using a scaling factor  of 16 and then   reporting the value of this running
> average at  the end  of the interval.  With a scaling factor of 16 the PPDV   metric is most strongly
> affected by  the delay  variation of the  most recent 10-20 packets and hence is not   representative
> of the  whole interval.
> 
> 	On  10/31/11  5:23  AM, "[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
> <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    " <[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/[email-masked]>
> <https://imail.huawei.com/OWA/UrlBlockedError.aspx>    > wrote:
> 
> 
> 
> 
> 
> 
> 	Hi, Alan:
> 
> 	Isn't the sampled metric same  as the  Interval metric? Is the  sampled metric only  applied to
> per-packet reporting ? what's the   difference between sampled  metric and non-sampled  metric?
> 
> 	收件人：[email-masked]
> <http://corpw6.webmail.21cn.com/webmail/jsp/pentsimple/htmlEditor/收件人：[email-masked]>
> <https://imail.huawei.com/OWA/UrlBlockedError.aspx>
> 	抄送人：
> 	The drafts  current  have one bit assigned for  Cumulative/Interval  indication....  Some
> metrics are however  sampled and  don’t have validity for an interval  or a   call/session; an example
> of this is PPDV, which is a  sampled value   of a running average and only has  validity for the last
> 200-300ms before  the value is  sampled.
> ________________________________
> 
> 
>

---

From: Qin Wu <[email-masked]>
Date: Thu, 10 Nov 2011 08:37:02 +0800

Yes, that's what I think to add one such definition to draft-ietf-avtcore-monarch.
Thank for your reminding.

Regards!
-Qin