Subject: Re: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

From: KEN KO <[email-masked]>
Date: Thu, 28 Feb 2013 19:05:17 +0000

Thanks for this comment. I agree that VBR encoding is typically used and that is frequently reflected in the short-term average rate of the video stream.
There are also a couple of differences between the model and actual streams that you didn't mention but are probably aware of. First, most protocols used for streaming result in the stream being delivered in bursts rather than as individually spaced packets. Second, many streams are available at multiple encoded rates and support adaptive switching between the rates based on the receiver's determination of variable performance in the network channel. I alluded to some of the above items in the "Additional Topics" section.
With regard to VBR - the variation in encoded rate would cause corresponding variation in the rate at which the dejitter buffer is drained, which would in turn affect the Dejitter-Buffer-Fill sample metric. Whether this change would better reflect actual performance depends on how well a change in the model represents actual VBR behavior. The current proposal uses the long term average (essentially a CBR model) as what seems to be a necessary simplification, since modeling the variation in VBR strikes me as a somewhat daunting task, but if a model enhancement could be shown to be both representative and manageable I would welcome it.
With regard to bursting - I have captured many actual streams and when there the network performance is significantly higher than that required to deliver the video stream, this bursting deviates significantly from the model. However, when the network performance is close to the required performance to support the video, the bursts are spread over time by the network. When network performance equals required performance, the bursts merge into each other and appear as a continuous stream at the destination. So the less margin there is in network performance (meaning, usually, the more interesting the test case), the closer the model approximates actual streaming behavior.
With regard to adaptive streaming, the model could be enhanced to simulate this, but this probably requires discussion with regard to whether it is appropriate. As it is, the metrics generated by the methodology can be described in terms of a small set of parameters including a "streaming rate." Once adaptive streaming is modeled, I think it may become more difficult to express the resulting metrics in intuitive terms.
I'll have a brief presentation in Orlando that will touch on these topics. Will you be there?
Best regards,
Ken
From: Guangqing Deng [mailto:[email-masked]]
Sent: Thursday, February 28, 2013 10:46 AM
To: KEN KO
Cc: ippm
Subject: Re: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

Hi, Ken, I am also interested in the modeling of streaming performance and have you ever done any actual network measurement to verify the accuracy of the streaming performance model in this draft? Usually, to ensure constant image quality, the VBR (Variable Bit Rate) coding method is used and then the streaming rate is always time-varying. Then is it necessary to take the dynamics of streaming rate into consideration for the streaming performance model?

________________________________
Guangqing Deng
cnnic

From: KEN KO<mailto:[email-masked]>
Date: 2013-02-19 09:30
To: [email-masked]<mailto:[email-masked]>
Subject: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt
IPPM,

Here is a new I-D relating to testing for streaming performance. Instead of sending a stream to directly test one condition, it applies a streaming model to metrics from TCP throughput testing to estimate performance for any number of conditions. Comments are welcome.

Regards,
Ken Ko

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Monday, February 18, 2013 1:50 PM
To: KEN KO
Subject: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

A new version of I-D, draft-ko-ippm-streaming-performance-00.txt
has been successfully submitted by Ken Ko and posted to the IETF repository.

Filename:  draft-ko-ippm-streaming-performance
Revision:  00
Title:  Model-Based Estimation of Streaming Performance
Creation date:  2013-02-18
Group:  Individual Submission
Number of pages: 18
URL:             http://www.ietf.org/internet-drafts/draft-ko-ippm-streaming-performance-00.txt
Status:          http://datatracker.ietf.org/doc/draft-ko-ippm-streaming-performance
Htmlized:        http://tools.ietf.org/html/draft-ko-ippm-streaming-performance-00

Abstract:
   This memo defines metrics plus a methodology for post-measurement
   processing of sample metrics to evaluate network path performance
   relative to streaming video traffic. The metrics are based on
   established methodologies for TCP throughput testing. The post-
   processing methodology is based on a model of streaming traffic that
   allows a given sample metric to be evaluated against multiple sets
   of parameters representing different streaming rates, delays and
   buffering values. The results of the post-processing methodology are
   derived metrics that are suitable for statistical analysis.

The IETF Secretariat

_______________________________________________
ippm mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ippm

---

From: KEN KO <[email-masked]>
Date: Fri, 29 Mar 2013 12:46:39 +0000

I think what you suggest (which I agree with) is already reflected in the model behavior. The initial model state is FILL_NOPLAY. In that state, received content is written to the buffer but nothing is played out until the state transition to either FILL_PLAY or MAINTAIN. That initial state (FILL_NOPLAY) represents the period of time you describe, in which content is being downloaded but the video has not yet started to play.
One thing the current draft is missing is figures, which would have made the above description much clearer. I need to learn to draw using ASCII characters!

From: Guangqing Deng [mailto:[email-masked]]
Sent: Thursday, March 28, 2013 10:31 PM
To: KEN KO
Cc: [email-masked]
Subject: Re: RE: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

Hi, Ken, for the model, I have a small suggestion regarding to the Initial Buffer State. In this draft, the Initial Buffer State is set to 0 and that is OK for the model. However, it seems that the Initial Buffer State is usually larger than 0 in actual situations. It is found that for most on-line video providers (at least those in China, such as Youku, Tudou, PPLive, etc.), there is a small period of time (usually several seconds) for commercial advertisement before the media content is played on the side of end users. And in such period of time, the media content is being downloaded from the server to the end user and then the dejitter buffer is being filled. So when the playback really starts (after the advertisement is finished), the dejitter buffer is not empty but has some amount of media content in it. So maybe we can take the scenario of the Initial Buffer State being nonzero into consideration. And it seems interesting to compare the streaming performance in such two different scenarios.

CNNIC

From: KEN KO<mailto:[email-masked]>
Date: 2013-03-01 03:05
To: dengguangqing<mailto:[email-masked]>
CC: ippm<mailto:[email-masked]>
Subject: RE: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt
Thanks for this comment. I agree that VBR encoding is typically used and that is frequently reflected in the short-term average rate of the video stream.
There are also a couple of differences between the model and actual streams that you didn't mention but are probably aware of. First, most protocols used for streaming result in the stream being delivered in bursts rather than as individually spaced packets. Second, many streams are available at multiple encoded rates and support adaptive switching between the rates based on the receiver's determination of variable performance in the network channel. I alluded to some of the above items in the "Additional Topics" section.
With regard to VBR C the variation in encoded rate would cause corresponding variation in the rate at which the dejitter buffer is drained, which would in turn affect the Dejitter-Buffer-Fill sample metric. Whether this change would better reflect actual performance depends on how well a change in the model represents actual VBR behavior. The current proposal uses the long term average (essentially a CBR model) as what seems to be a necessary simplification, since modeling the variation in VBR strikes me as a somewhat daunting task, but if a model enhancement could be shown to be both representative and manageable I would welcome it.
With regard to bursting C I have captured many actual streams and when there the network performance is significantly higher than that required to deliver the video stream, this bursting deviates significantly from the model. However, when the network performance is close to the required performance to support the video, the bursts are spread over time by the network. When network performance equals required performance, the bursts merge into each other and appear as a continuous stream at the destination. So the less margin there is in network performance (meaning, usually, the more interesting the test case), the closer the model approximates actual streaming behavior.
With regard to adaptive streaming, the model could be enhanced to simulate this, but this probably requires discussion with regard to whether it is appropriate. As it is, the metrics generated by the methodology can be described in terms of a small set of parameters including a "streaming rate." Once adaptive streaming is modeled, I think it may become more difficult to express the resulting metrics in intuitive terms.
I'll have a brief presentation in Orlando that will touch on these topics. Will you be there?
Best regards,
Ken
From: Guangqing Deng [mailto:[email-masked]]
Sent: Thursday, February 28, 2013 10:46 AM
To: KEN KO
Cc: ippm
Subject: Re: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

Regards,
Ken Ko

-----Original Message-----
From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]]
Sent: Monday, February 18, 2013 1:50 PM
To: KEN KO
Subject: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

Abstract:

The IETF Secretariat

---

From: "Guangqing Deng" <[email-masked]>
Date: Fri, 29 Mar 2013 11:31:08 +0800

Guangqing Deng
CNNIC 

From: KEN KO
Date: 2013-03-01 03:05
To: dengguangqing
CC: ippm
Subject: RE: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt
Thanks for this comment. I agree that VBR encoding is typically used and that is frequently reflected in the short-term average rate of the video stream. 
There are also a couple of differences between the model and actual streams that you didn’t mention but are probably aware of. First, most protocols used for streaming result in the stream being delivered in bursts rather than as individually spaced packets. Second, many streams are available at multiple encoded rates and support adaptive switching between the rates based on the receiver’s determination of variable performance in the network channel. I alluded to some of the above items in the “Additional Topics” section.
With regard to VBR – the variation in encoded rate would cause corresponding variation in the rate at which the dejitter buffer is drained, which would in turn affect the Dejitter-Buffer-Fill sample metric. Whether this change would better reflect actual performance depends on how well a change in the model represents actual VBR behavior. The current proposal uses the long term average (essentially a CBR model) as what seems to be a necessary simplification, since modeling the variation in VBR strikes me as a somewhat daunting task, but if a model enhancement could be shown to be both representative and manageable I would welcome it. 
With regard to bursting – I have captured many actual streams and when there the network performance is significantly higher than that required to deliver the video stream, this bursting deviates significantly from the model. However, when the network performance is close to the required performance to support the video, the bursts are spread over time by the network. When network performance equals required performance, the bursts merge into each other and appear as a continuous stream at the destination. So the less margin there is in network performance (meaning, usually, the more interesting the test case), the closer the model approximates actual streaming behavior.
With regard to adaptive streaming, the model could be enhanced to simulate this, but this probably requires discussion with regard to whether it is appropriate. As it is, the metrics generated by the methodology can be described in terms of a small set of parameters including a “streaming rate.” Once adaptive streaming is modeled, I think it may become more difficult to express the resulting metrics in intuitive terms.
I’ll have a brief presentation in Orlando that will touch on these topics. Will you be there?
Best regards,
Ken
From: Guangqing Deng [mailto:[email-masked]] 
Sent: Thursday, February 28, 2013 10:46 AM
To: KEN KO
Cc: ippm
Subject: Re: [ippm] FW: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

Guangqing Deng
cnnic 
 
From: KEN KO
Date: 2013-02-19 09:30
To: [email-masked]

Regards,
Ken Ko
 
-----Original Message-----
From: [email-masked] [mailto:[email-masked]] 
Sent: Monday, February 18, 2013 1:50 PM
To: KEN KO
Subject: New Version Notification for draft-ko-ippm-streaming-performance-00.txt

Abstract:

 
The IETF Secretariat
 
[email-masked]