Subject: [AVT] Re: Comments on draft-tseng-rtcp-streaming-extns-00.txt

From: Magnus Westerlund <[email-masked]>
Date: Mon, 03 Nov 2003 12:55:09 +0100

Hi Gamze,

Comments inline:

Gamze Seckin wrote:

> 
> 2. Regarding the corruption statistics. For media that uses codec that 
> have prediction any error normally remains for a rather long time. 
> However it is very difficult to determine when the effects has been 
> removed. Therefore this measurement is hard to derive on the receiver 
> side. Combined with the definition it only provides a good, possibly bad 
> indication. Also looking at the reasons for corruption, the only real 
> reason is lost or late packets. A good feeling for the quality can be 
> determined by the server simply by employing packet loss vector 
> reporting and knowing when the buffer is empty.
> 
> [g]packet loss is definitely a way of estimating corruption, but this 
 > gives an algorithm in the decoder (or pre-decoder)a chance to provide
 > better statistics.

Yes, it might be possible for such an algorithm to do this. However the 
measurement will not provide that much information. I think you 
basically can only say this is good enough or not, there will not exist 
any possibility to say how bad things actually are. Otherwise you will 
need to clearly specify how this measurement algorithm shall work.

However one concern here is that the corruption measurement will be 
quite complex and require rather much memory to perform properly. In a 
video decoder it will require the measurement tool to track how the 
error propagates over the screen. For some audio codecs it will be 
impossible to determine when the error is totally removed from the loop. 
To do this you would need to have information about what the correct 
state is.

> 
> 3. The rebuffering measurements makes more sense. However there are at 
> least one client that does not really perform rebuffering, namely 
> quicktime. Instead is assumes that if the transmitted bit-rate is 
> acceptable any short term effects will go away and the client will later 
>   have a buffer fullness given by the initial buffering delay. The 
> interesting thing about this measurement is that in 3GPP they already 
> have a way of letting the server measure when rebuffering takes place, 
> namely the OBSN reporting specified in TS 26.234 REL 6 (only preliminary 
> versions available).
> 
> [g] yes, and I believe those statistics are useful, however they are 
 > tight with rate adaptation (only turned on when rate-adaptation
 > is enabled) and tightly coupled with other rtsp messages.

Yes, they however is another way of measuring this. I think the 
rebuffering measure is possibly more useful then the corruption 
measurement.

> 6. Section 1. Last sentence: "As all the other report blocks
> defined in [1], the streaming report blocks are applicable to
> one-to-one or one-to-many applications that generate
> RTCP SR or RR reports." This sentence is not really correct. The use of 
> the RTCP XR reports is possible in any application sending RTCP.
> 
> [g] himm,i am confused i thought that the first sentence covers everything - what is left out? 
> So [1] needs to be updated too?

What I object against is the distinction around RTCP SR or RR reports 
and the use of generate. I would rather formulate it as:

As all the other report blocks defined in [1], the streaming report 
blocks are applicable to one-to-one or one-to-many applications that 
send RTCP in the desired direction.

> 8. Section 2, Figure 1. This figure shows this report formats potential 
> need to have reporting from all layers on the receiving side. This might 
> be a problem for certain application architectures. I am not saying that 
> it can't be used, its only that some applications may have difficulties 
> to include your reports.
> 
> [g] we believe the statistics reported should be extensible also should be allowed to be ignored. 
> So if an end side cannot support it should put simply a zero - what do you think?

I think one need to consider the meaningfulness of allowing only some of 
the measurement to be given.

> 10. Section 3.3, PAge 5, first bullet after figure 2:
> "- at the AVP module: Packet drop and packet errors can sometimes be
> detected by incorrect or missing sequence number or checksum. " Do you 
> really need to consider bit-errors, the only allowed possibility for 
> bit-errors in RTP is if partial checksums are used. Bit-errors surviving 
> to the transport layer is the meaning to be caught by the UDP, etc., 
> checksum.
> 
> [g] there could be cases where bit errors are propagated (e.g. udp lite).

I think that one should be clearer about that this then. However the 
usage of UDP-Lite is still under a lot of discussion within the IETF. 
There are some very though problems for IP to accept bit-errors 
partially. If one are using another transport of RTP then it might be 
more common. I would still like you to be more clear that this might be 
the case if one uses such mechanisms as UDP-Lite and payload formats + 
codecs that accept it.

Cheers

Magnus Westerlund

Multimedia Technologies, Ericsson Research EAB/TVA/A
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: John Lazzaro <[email-masked]>
Date: Mon, 3 Nov 2003 10:42:18 -0800 (PST)

> Magnus Westerlund <[email-masked]> writes:
>

It might be time for AVT to set a provisional policy with regards to
what payload format authors should or should not be doing with respect
to UDP-lite. If one is starting with a blank sheet of paper to
design a payload format, the road one takes to fully leverage UDP-lite
is a different road from the norm.  In the RTP MIDI case, for example,
I would need to add in a few checksums words in key places for the
recovery journal system to work well in a world where delivered packets
may have bit-errors in the payload.  This seemed too complicated to
add to a payload format so close to completion, so I decided not to
go that route.  But maybe a payload author starting design work
today should make a different decision ...

-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu     www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------

---

From: "Gamze Seckin" <[email-masked]>
Date: Wed, 5 Nov 2003 15:24:13 -0800

Hi,
please see comments inline.
 
thanks
g

-----Original Message-----
From: Magnus Westerlund [  <mailto:[email-masked]> mailto:[email-masked]]
Sent: Monday, November 03, 2003 3:55 AM
To: Gamze Seckin
Cc: Alan Tseng; IETF AVT WG
Subject: Re: Comments on draft-tseng-rtcp-streaming-extns-00.txt

Hi Gamze,

Comments inline:

[Gamze Seckin] I agree that measuring corruption is difficult and is a best effort method. 
However it is a useful metric to asses quality of streaming and applications who provide this information should be encouraged to send it. 

 
[Gamze Seckin]  i think we agree on the usefulness of measuring rebuffering.

[Gamze Seckin] thanks - agreed. 

I think one need to consider the meaningfulness of allowing only some of
the measurement to be given.
[Gamze Seckin] some metrics can be still useful without the complete set. I believe this is an application dependent issue. 

[Gamze Seckin] noted.

thanks, 
g 

Cheers

Magnus Westerlund

---

From: Thomas Zeng <[email-masked]>
Date: Tue, 11 Nov 2003 09:09:50 -0800

Hi Alan and Gamze:

My comments on this draft:

General comment: would it be possible to provide the means for server to
signal to client the interval for Quality Metrics reporting? For instance an
optional parameter in RTSP Play response?

1. Like Magnus, I don't feel "AVP Module" (defined in section 2) is a right
choice of name. I suggest "Transport Handler" might be a better fit in
naming the concept (to differentiate with source codec).

2. Please in the introduction provide some insight of your motivations. If
one implements this draft, would one use it for debugging only, or for
billing also? We'd like to see a generic proposal that will, if possible,
satisfy the desire to do QOS based billing, as well as network monitoring /
user experience monitoring/survey.

3. In section 3.3, it appeared that it is assuming that a RTP packet will
contain only one or partial VOP so that the AVP module could determine the
corruption period from the RTP timestamp.  However, in the multi-VOP RTP
packet, then the corruption period need to contain multi-VOPs.  Just a
suggestion to consider the multi-VOP case.

4. Sometimes player freezes not due to "rebuffering" (underflow of player
buffer).  For example, for live streaming, packets not get sent by the
server due to late arrival from author.  If server drop packets
intentionally, then the player will not detect seq number gap in this case.
There could be other reason that player will freeze not due to rebuffering
(but then jump in play time later).  Will this be included in the
rebuffering matrices?  According to the definition, it should.  So
"rebuffering' in this draft does not always refer to "underflow of player
buffer"?  For instance, we've seen freezing due to overflow (caused by bad
server scheduling) and/or player processor overload.

5.Typo on Page 10, "pts_stop" ...  "beginning" should be "end"

6. On 3.4 last sentence, for rebuffering, then there should not reach end of
playback since the player is not playing and the playtime is stalled.  So we
are  not sure if this sentence applies to rebuffering (at least the
rebuffering due to buffer underflow).  It does have some implications if the
player did not receive any more data and abort due to timeout (i.e., staying
in rebuffering and timeout).  So we think these two cases (normal
rebuffering, or rebuffering with timeout) needs to be considered.

> From: Magnus Westerlund [mailto:[email-masked]]
> Sent: Wednesday, November 05, 2003 11:41 PM
> To: Gamze Seckin
> Subject: Re: [AVT] RE: Comments on 
> draft-tseng-rtcp-streaming-extns-00.txt
> 
> 
> Hi Gamze,
> 
> To my knowledge there is no profile dependencies in RFC 3611. Thus it 
> can be used for any of the four profiles defined or under 
> definition. My 
> comment was to point out this fact and that it seems inappropriate to 
> talk about any special profile in this context.
> 
> cheers
> 
> Magnus
> 
> Gamze Seckin wrote:
> 
> > Hi Magnus,
> > 
> > about your comment
> > "7. Section 2, You talk about AVP, but this extension is 
> not dependent on 
> > any special profile and seems to be compatible with any of the four 
> > available or under development profiles that exist."
> > 
> > Do you also mean AVPF? Since I am confused about the 
> relationship between AVPF and RFC3611.
> > 
> > thx
> > g
> > From: Gamze Seckin 
> > Sent: Friday, October 31, 2003 11:08 AM
> > To: Magnus Westerlund; Alan Tseng
> > Cc: IETF AVT WG
> > Subject: [AVT] RE: Comments on 
> draft-tseng-rtcp-streaming-extns-00.txt
> > 
> > 
> > 
> > Hi Magnus, 
> > thanks for the comments.
> > See my response inline.
> > 
> > best regards
> > gamze
> > From: Magnus Westerlund [mailto:[email-masked]]
> > Sent: Wednesday, October 29, 2003 8:17 AM
> > To: Alan Tseng; Gamze Seckin
> > Cc: IETF AVT WG
> > Subject: Comments on draft-tseng-rtcp-streaming-extns-00.txt
> > 
> > 
> > Hi,
> > 
> > Here is my comments on your draft:
> > 
> > 1. First I think the draft isn't very clear on why any metrics are 
> > needed. In the 3GPP document it only states: "The objective of the 
> > streaming service quality metrics is to offer to the 
> service providers 
> > means to evaluate the end user experience.". This doesn't 
> make it very 
> > clear what the real needs are. I think further developing 
> of this is 
> > needed.
> > 
> > [g]you are right, the draft does not talk about that 
> clearly. We should add that.
> > 
> > [g]packet loss is definitely a way of estimating 
> corruption, but this gives an algorithm in the decoder (or 
> pre-decoder)a chance to provide better statistics.
> > 
> > [g] yes, and I believe those statistics are useful, however 
> they are tight with rate adaptation (only turned on when 
> rate-adaptation is enabled) and tightly coupled with other 
> rtsp messages.
> > 
> > 4. As state privately before this drafts need to be checked 
> against the 
> > ID nits: http://www.ietf.org/ID-nits.html
> > 
> > [g] yes, we should definitely do that thanks.
> > 
> > 5. Section 1. The introduction is basically the same as the 
> > introduction. I would recommend you to write a better more 
> informative 
> > introduction.
> > 
> > [g] agreed.
> > 
> > 7. Section 2, You talk about AVP, but this extension is not 
> dependent on 
> > any special profile and seems to be compatible with any of the four 
> > available or under development profiles that exist.
> > 
> > [g] agreed.
> > 
> > 9. Section 3.1, first paragraph: "All timestamps reported by the XR 
> > packet SHALL be RTP timestamps. All durations SHALL be in 
> units of the 
> > RTP timebase." This definition is not clear, because in 
> applications 
> > allowing for switching between payload types the rate can change. I 
> > think that the intention of using the RTP timestamp value 
> for indication 
> > over what interval which the report covers is a good one. 
> However some 
> > more consideration over formulation must be done.
> > 
> > [g] valid point - I guess we need to think more about the 
> corner cases.
> > 
> > 11. Section 6. The streaming metrics should probably have a 
> value to 
> > combine with the parameter to indicate how often the report 
> block should 
> > be included. This seem to be a block that is not needed very often.
> > 
> > [g] valid point, note taken.
> > 
> > 12. Section 8. The security section does not look 
> sufficient. I guess 
> > that the consideration for this is rather similar to the 
> VoIP block. 
> > However I think there should be made some effort to 
> indicate the main 
> > security concerns also in this document.
> > 
> > [g] agreed.
> > 
> > 13. Section 12. The references need to be divided into 
> Normative and 
> > Informative ones.
> > 
> > [g] agreed.
> > 
> > Cheers
> > 
> > Magnus Westerlund
> > 
> --

---

From: [email-masked]
Date: Wed, 12 Nov 2003 18:32:12 +0200

Hi,

some comments on the draft.

1. Overall the draft is interesting.

2. I support most of the comments already given by Magnus and Thomas. In particular, I would like to stress the fact that the concept of corruption is fuzzy. For example a data loss of 1 second does not necessarily map to a subjective quality degradation of 1 second. It might be shorter or longer, depending on how error concealment is done. Especially for video, this is a complicated issue. And yet one aspect where the complexity relies, is the difficulty of extracting the needed information from the decoder, either because of lack of interfaces or because it is computationally intensive to perform this task. 

Regards,

Igor

==========================================
Igor D.D. Curcio
Nokia Corporation
Mobile Software, Architecture & Roadmapping
Multimedia streaming and video telephony
P.O. Box 88
33721 Tampere
Finland
 
Tel. +358 800 8000
Email: [email-masked]
WWW  www.nokia.com

> From: [email-masked] [mailto:[email-masked]]On Behalf Of ext
> Thomas Zeng
> Sent: 11 November, 2003 19:10
> To: 'Magnus Westerlund'; Gamze Seckin
> Subject: RE: [AVT] RE: Comments on