Subject: [xrblock] potential extension to Discard draft//Re: General comments on xrblock metrics drafts

From: Qin Wu <[email-masked]>
Date: Tue, 01 Nov 2011 17:08:53 +0800

General comments on xrblock metrics draftsHi,
I separate this comment from the previous thread since it relate to potentional extension to the Discard Draft.
I think it is beneficial to distinguish early discard and lat discard through time windows.
Although I don't have use case in mind on how to use early discard and late discard, but
such expansion to the discard metric may be helpful to calculate other newly defined derived metric.
So I like to second this proposal.

Regards!
-Qin
----- Original Message ----- 
  From: Alan Clark 
  To: [email-masked] 
  Sent: Monday, October 31, 2011 6:51 AM
  Subject: [xrblock] General comments on xrblock metrics drafts

  3. Discard draft

  It would be useful to identify early vs late discards, as this can help to determine what the cause was.  

------------------------------------------------------------------------------

  _______________________________________________
  xrblock mailing list
  [email-masked]
  https://www.ietf.org/mailman/listinfo/xrblock

---

From: Varun Singh <[email-masked]>
Date: Tue, 1 Nov 2011 11:36:47 +0200

Hi Qin,

There is an 'E' bit in
http://tools.ietf.org/html/draft-ott-xrblock-rtcp-xt-discard-metrics-01
(Section 3 and 4) that distinguishes between discards due to early and
late arrivals. We have used these reports for congestion control
(H.264 Video) and for subflow reports in Multipath RTP.

Cheers,
Varun

On Tue, Nov 1, 2011 at 11:08, Qin Wu <[email-masked]> wrote:
> Hi,
> I separate this comment from the previous thread since it relate to
> potentional extension to the Discard Draft.
> ________________________________
>

-- 
http://www.netlab.tkk.fi/~varun/

---

From: Qin Wu <[email-masked]>
Date: Tue, 01 Nov 2011 17:36:02 +0800

General comments on xrblock metrics draftsThanks Alan.
Also I separate these comments from the previous thread since it focuses on specific drafts.
Please see inline below.
----- Original Message ----- 
  From: Alan Clark 
  To: [email-masked] 
  Sent: Monday, October 31, 2011 6:51 AM
  Subject: [xrblock] General comments on xrblock metrics drafts

  2. Bursts and Gaps (burst-gap-discard and burst-gap-loss drafts)
  The original Burst/Gap concept addressed four issues:-

  (a) Packet loss and discard have similar effect on the quality of a decoded voice, audio or video stream.

  (b) Packet loss is often (not always) caused by congestion, which also leads to higher jitter levels and hence packet discard.  This means that loss and discard are often closely related in time.  From a QoE perspective it is important to look at the combined effect as the impact on quality may be quite dependant on the timing of loss and discard relative to each other.

  (c) Prior work had focused on the idea of bursts being periods of consecutive loss - and the duration of such bursts is typically very short. Loss (and discard) does however occur in "sparse" bursts which can be seconds in length.  This was the motivation behind the use of a Markov model with the Gmin parameter being used to discriminate bursts and gaps.  

  (d) IP impairments tend to be transient, hence average packet loss rates are not too helpful in understanding QoE. Given a 1% loss rate there is a major difference between losing one packet in every one hundred, which PLC could handle, and losing a burst of ten packets in every one thousand. There are two approaches to reporting transient problems - either send frequent interval reports OR define metrics that can describe transients. The Burst/Gap metrics defined in RFC3611 provide an efficient way of describing transient bursts of lost and discarded packets.

  Having implemented the approach described in RFC3611 in 2000 and used these metrics for over ten years, we have found that the basic premise was sound and does allow for very efficient reporting and accurate QoE metrics.  The feedback from those using these metrics was that they would also like to be able to correlate bursts with other network events and hence it would be helpful to have some form of "time since last burst" metric.

  [Qin]: What's the use of "time since last burst"meric? Would you like to clarify more?

  With regard to the current burst-gap-discard and burst-gap-loss drafts, my opinion is they should be either combined to a single burst-gap-loss-discard draft OR there should be burst-gap-loss-discard and burst-gap-loss drafts.  A combined loss/discard draft is more useful for QoE assessment however there may be some cases in which reporting only loss may be useful.

  [Qin]: I should admit most of metrics defined in both the current-gap-discard and burst-gap-loss are the same, So I have the same concerns  if we need to merge them. However it looks to me it conflicts with the principle defined in draft-ietf-avtcore-monarch, i.e.,
  "contain a single metric or a small number of metrics relevant to a single parameter ".
  But we may have some trick if we hope to combine loss/discard draft to assess QoE, i.e., we can send both burst/gap loss metric and burst/gap discard metric in the same RTCP XR packet.

  The drafts contain sufficient metrics to calculate standard deviations of bursts however I'm personally not convinced that standard deviation is more useful than mean and max values, together with a time since last burst value.  Max burst length is important as this is the period that impacted the user most.  

  [Qin]: The standard deviation is useful since they are used to calculate other derived metrics like 
  a.The fraction of packets discarded during bursts
  b.The fraction of packets discarded during gaps
  c.burst duration mean
  d.burst duration variance

  As for Max burst length, do you mean Max burst duration? Would you like to clarify how to use Max burst length to estimate the impact that adds to the user?

  Alan

---

From: Qin Wu <[email-masked]>
Date: Tue, 01 Nov 2011 10:34:44 +0000

Hi, Varun:
That's a good example. Thank for bringing your use case.
If you like to explain more on why early discard
and late discard should be distincted? How to use them,
I will be more appreciated.

Regards!
-Qin
________________________________________
发件人: [email-masked] [[email-masked]] 代表 Varun Singh [[email-masked]]
发送时间: 2011年11月1日 17:36
到: Qin Wu
Cc: [email-masked]
主题: Re: [xrblock] potential extension to Discard draft//Re: General comments on xrblock metrics drafts

Hi Qin,

There is an 'E' bit in

Cheers,
Varun

---

From: Varun Singh <[email-masked]>
Date: Tue, 1 Nov 2011 13:17:12 +0200

Hi Qin,

This is a quick summary:

In MPRTP, if each path has a different end-to-end delay (RTT mismatch)
and the sender uses both paths then the receiver will drop early or
late packets based on the playout. The receiver signal this
information using the RLE encoded discard packets to optimize the
scheduling algorithm. There may be other alternatives to implement
this functionality (e.g., per-path TMMBR).

In the case of sender driven congestion control, if the packets
arrived late then the receiver would drop the packet without sending
it to the decoder based on the application specific maximum end-to-end
delay, A/V sync. In this case the receiver signals the 'late' packets
that were dropped after arrival.

Cheers,
Varun

---

From: Alan Clark <[email-masked]>
Date: Tue, 1 Nov 2011 18:07:13 -0400

Qin

(i) Time Since Last Burst

We sometimes have users (service providers) that would like to correlate the
more significant loss/discard events with events within the network, for
example route changes or equipment failures.  This is of course possible
within the time frame of an interval, however if cumulative reporting was
used then it would be helpful to know roughly when the last substantial
loss/discard burst occurred relative to the report time. Interval reporting
can inherently generate large numbers of reports, so it is not desirable to
use very short intervals - so a combination of longer intervals with a
³time/delay since last burst² would be a good compromise.

(ii) Loss/Discard Burst

The key point in my comments was that loss and discard often coincide in
time (as they can both be caused by congestion), and if you want to
understand/compute the impact on QoE then you need the combined metric.  If
you have separate loss burst and discard burst metrics then it is not
meaningful to add them, given that they are often not independent of each
other.  We have found, for example, that if there is a burst of packet loss
with some given length and density then the effect of discard is to make
that burst longer.

You can easily visualize this if you think of a stream of packets with
increasing volume interfering with an RTP stream.  As the interfering stream
increases in volume, the jitter level increases, then discards start to
occur, then eventually buffer overflows occur in switches/routers resulting
in packet loss. 

(iii) Statistics related to bursts

You can calculate average burst and gap loss/discard rates without keeping
and reporting squares/ sums of squares.  My observation was that there
should be a purpose for a metric and hence the question - what is the
usefulness of the standard deviation of burst length?  Also, standard
deviation is most meaningful if the values being analyzed are normally
distributed, and it is more likely that burst length would have some form of
long tailed distribution.

(iv) Max burst length

This represents the ³worst² period from a QoE perspective and it would be
useful to know this.

Best Regards

Alan

On 11/1/11 5:36 AM, "Qin Wu" <[email-masked]> wrote:

> Thanks Alan.
>> From:  Alan Clark <mailto:[email-masked]>
>>  
>> To: [email-masked]
>>  
>> (b) Packet loss is often  (not always) caused by congestion, which also leads
>> to higher jitter levels  and hence packet discard.  This means that loss and
>> discard are often  closely related in time.  From a QoE perspective it is
>> important to look  at the combined effect as the impact on quality may be
>> quite dependant on the  timing of loss and discard relative to each other.
>> 
>> (c) Prior work had  focused on the idea of bursts being periods of
>> consecutive loss - and the  duration of such bursts is typically very short.
>> Loss (and discard) does  however occur in ³sparse² bursts which can be
>> seconds in length.  This  was the motivation behind the use of a Markov model
>> with the Gmin parameter  being used to discriminate bursts and gaps.
>> 
>> (d) IP impairments  tend to be transient, hence average packet loss rates are
>> not too helpful in  understanding QoE. Given a 1% loss rate there is a major
>> difference between  losing one packet in every one hundred, which PLC could
>> handle, and losing a  burst of ten packets in every one thousand. There are
>> two approaches to  reporting transient problems - either send frequent
>> interval reports OR define  metrics that can describe transients. The
>> Burst/Gap metrics defined in RFC3611  provide an efficient way of describing
>> transient bursts of lost and discarded  packets.
>> 
>> Having implemented the approach described in RFC3611 in 2000  and used these
>> metrics for over ten years, we have found that the basic  premise was sound
>> and does allow for very efficient reporting and accurate QoE  metrics.  The
>> feedback from those using these metrics was that they would  also like to be
>> able to correlate bursts with other network events and hence  it would be
>> helpful to have some form of ³time since last burst² metric.
>>  
>>  
>>  
>> With regard to  the current burst-gap-discard and burst-gap-loss drafts, my
>> opinion is they  should be either combined to a single burst-gap-loss-discard
>> draft OR there  should be burst-gap-loss-discard and burst-gap-loss drafts.
>> A combined  loss/discard draft is more useful for QoE assessment however
>> there may be some  cases in which reporting only loss may be useful.
>>  
>>  
>>  
>> [Qin]: I should admit most of metrics  defined in both the
>> current-gap-discard and burst-gap-loss are the same, So I  have the same
>> concerns  if we need to merge them. However it looks to me  it conflicts with
>> the principle defined in draft-ietf-avtcore-monarch,  i.e.,
>>  
>> But we may have some trick if we hope  to combine loss/discard draft to
>> assess QoE, i.e., we can send both burst/gap  loss metric and burst/gap
>> discard metric in the same RTCP XR  packet.
>> 
>> The drafts contain sufficient metrics to calculate  standard deviations of
>> bursts however I¹m personally not convinced that  standard deviation is more
>> useful than mean and max values, together with a  time since last burst
>> value.  Max burst length is important as this is  the period that impacted
>> the user most.  
>>  
>>  
>>  
>> As for Max burst length, do you mean  Max burst duration? Would you like to
>> clarify how to use Max burst length to  estimate the impact that adds to the
>> user?
>>  
>>  
>>  
>>  
>>  
>> Alan
>> 
>> 
>> 
>> 
>> 
>>  
>> 
>>  
>> 
>>  
>>

---

From: "Roni Even" <[email-masked]>
Date: Sun, 20 Nov 2011 14:19:58 +0200

Hi Alan,

I am addressing only this topic now.

I understand that you imply that there are cases when the metrics is neither
Cumulative nor Interval and in the current structure you cannot specify that
the time information in the identity block is not relevant. 

I think that the approach may be to allow for sample metrics to override the
identity block adding the time information in the report block itself. I
think that if the sample information (time) will be part of the imeasurement
information it will work if we have the tag but I am not sure if it will be
efficient in term of message size.

Roni

From: [email-masked] [mailto:[email-masked]] On Behalf
Of Alan Clark
Sent: Monday, October 31, 2011 12:51 AM
To: [email-masked]
Subject: [xrblock] General comments on xrblock metrics drafts

1. Cumulative, Interval and Sample metrics
The drafts current have one bit assigned for Cumulative/Interval indication.
Some metrics are however sampled and don't have validity for an interval or
a call/session; an example of this is PPDV, which is a sampled value of a
running average and only has validity for the last 200-300ms before the
value is sampled.
Proposal - increase this to two bits to allow for the indication of a
Sampled value. 

(a) Packet loss and discard have similar effect on the quality of a decoded
voice, audio or video stream.

Loss (and discard) does however occur in "sparse" bursts which can be

helpful to have some form of "time since last burst" metric.

The drafts contain sufficient metrics to calculate standard deviations of
bursts however I'm personally not convinced that standard deviation is more
useful than mean and max values, together with a time since last burst
value.  Max burst length is important as this is the period that impacted
the user most.  

3. Discard draft

It would be useful to identify early vs late discards, as this can help to
determine what the cause was.  

4. PDV draft

As mentioned above - jitter values could be averaged over an interval or
potentially sampled.  

5. QoE draft

Separate comments will be provided
Alan