Subject: Re: [MMUSIC] RTSP: Speed header

From: Magnus Westerlund <[email-masked]>
Date: Tue, 23 Oct 2007 14:30:56 +0200

Hi,

See inline:

Magnus Westerlund skrev:
> Hi,
> 
> I while back received an email with some questions on how the Speed
> header works in RTSP. There where also the issue how it interacts with
> the Scale header and with bandwidth adaptation and buffer management.
> 
> Lets do a recap of what the different mechanisms do.
> 
> Scale: Changes the rate at which media is presented to the user. So
> Scale=0.5 would mean slow motion, while Scale=5 would be fast forward in
> the stream.
> 
> Speed: Changes the media delivery rate. Speed=2 would mean that for each
> second a client would receive media that takes it 2 seconds to play back
> to the user. It would also mean that using RTP with a 90kHz timestamp
> clock the RTP streams timestamp values would increase with 180k ticks
> per second.
> 
> Bit-rate adaptation: A way of ensuring that the media traffic does not
> use more bandwidth then what is available. Which means that the server
> would need to change the media bit-rate sent if there isn't enough
> available for the current media stream. This could be done in a number
> of ways but most commonly are done by bit-stream switching. Going from a
> stored/delivered media stream in one quality to another lower quality
> that requires less bit-rate.
> 
> Buffer Management: An attempt to keep the clients buffer in a healthy
> state, aiming at keeping a certain amount of media time present in
> buffer to safeguard against temporary or longer term changes of bit-rate
> capacity available to the streaming session. Thus giving the server time
> to react to the change before an buffer-underrun situation occurs. At
> the same time safe-guard against overflowing buffers. This is for
> example done in 3GPP Packet-based Streaming Service (26.234 Rel6) where
> additional feedback and configuration information is provided for the
> buffer status. Thus allowing a server to change media transmission rate
> on the fly. Which is equivalent to a client using Speed. However it also
> needs to be able to perform bit-rate adaptation so that one can lower
> the quality but increase the speed to refill the buffer.
> 
> The interactions between the different behaviors are interesting but can
> become quite complex. For example the total bit-rate is speed x normal
> bit-rate. It might also be influenced by the Scale used as fast
> forwarding could be accomplished by simple turning up the frame-rate for
> video and rewriting the timestamps compared to scale 1. Slow motion
> could also mean that the bit-rate gets lower. However if trans-coding is
> done to handle Scale then bit-rate may not change at all. If there are a
> limited amount of bit-rate this means that bit-rate adaptation needs to
> step in and do what it can to keep bit-rate within limitations. If that
> can only operate on the media stream or can also contradict speed or
> influence how scale is handled is quite undefined.
> 
> I do have a few question due to this:
> 
> 1. Are people implementing Speed?
> 
> 2. If you are using Speed, how are you doing that, as I have described
> or otherwise?
> 
> 3. Should we remove speed from base specification?
> 
> 4. If not, what do you think needs to be clarified and do you have text
> proposal?

Time to bring up this topic again and get some closure on it. I have
reviewed what people have written and have a number of comments and
considerations around this.

First of all is the general congestion control. To me Speed header shall
not be used for congesting controlling the flow from the server to the
client. To me the usage that seems okay are:

1. In combination with scale to tell how it should fulfill the request.
Here I think the two main cases are Scale=x Speed=x for x!=1 and Scale=x
Speed=1. In other words, realize a playout speedup by increasing the
delivery rate to the client or if the server should rescale the content
to maintain the bit-rate.

2. To indicate faster than normal delivery to fill up the receiver
buffers, i.e. scale=1 speed=x where x>1. Then when buffer reached the
desired level the client requests that speed goes back to 1.0.

Both these has interactions with the actual congestion control used. If
the available bit-rate is limited in both these case the content needs
to be adapted to fulfill.

Are these interesting enough that we actually should write them up so
that it is clear what they mean? The alternative is to remove speed for
now and have the behaviors being undefined in these cases.

Cheers

Magnus Westerlund

IETF Transport Area Director & TSVWG Chair
----------------------------------------------------------------------
Multimedia Technologies, Ericsson Research EAB/TVM/M
----------------------------------------------------------------------
Ericsson AB                | Phone +46 8 4048287
Torshamsgatan 23           | Fax   +46 8 7575550
S-164 80 Stockholm, Sweden | mailto: [email-masked]
----------------------------------------------------------------------

_______________________________________________
mmusic mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/mmusic

---

From: "Jaehwan Kim" <[email-masked]>
Date: Thu, 25 Oct 2007 23:44:32 +0900

Hi Magnus,
To me, scale is also not easy because decoder controls speed of motion
or music in general, not by signaling protocol. What's more, some
decoders do not see RTP timestamp; they use bitstream's one; and
signaling layer should have the same timestamp as bitstream does. Or do
they violate RTP and RTSP?

Why don't we just remove this from base specification and make a
separate one? As you said, in 3GPP, we discuss FastStart, rate
adaptation, and client buffer. I don't mean we need to wait; just we
need to discuss with them and make a good universal solution together.
But if we add this to base, it might be much more difficult.

Cheers,
Jaehwan

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]] 
Sent: Tuesday, October 23, 2007 1:31 PM
To: Magnus Westerlund
Cc: mmusic (E-mail)
Subject: Re: [MMUSIC] RTSP: Speed header

Hi,

See inline:

Cheers

Magnus Westerlund

---

From: "Mike Severa" <[email-masked]>
Date: Thu, 25 Oct 2007 09:05:15 -0700

Hi Jaehwan, 

I agree scale is not easy. And I'm not sure whether its germaine to your
argument, but I don't see how a streaming client could behave properly
without the RTP timestamp. It is surely required for A/V sync, and there
are a number of codecs that don't even have a "built-in" bitstream-level
timestamp, or others for which timestamps only have a kind of
"time-local" meaning. In addition, I do know that there are a number of
players that even ignore bitstream timestamps when they exist in favor
of RTP ones.

That said, I believe the behavior surrounding 3GPP faststart, rate
adaptation, buffer mgmt, etc, have more similarity with the speed header
than with the scale header. I believe the scale header was intended to
provide support for something that these other things being discussed in
3GPP currently do not, at least not directly. 
Even so, while one could argue the speed header might be more similar, I
echo Magnus' opinion below in that I also don't think the intent of that
header was to provide the client a congestion control mechanism, but
that doesn't mean it would not be useful in the context of services that
make use of fast-start/switching, buffer mgmt, etc, even rate adaptation
in certain contexts. It should also be clear that regardless of how much
a 3GPP streamer server might be able to do as defined by those specs,
these headers allow *client control* over aspects of these behaviors for
which I don't believe there is any currently defined alternative
allowing a client this kind of control. 

It has been my experience that in a fast switching, 3GPP streaming-based
environment, the availability of something like a speed header is a
requirement (it is my opinion that current 3GPP docs do not even address
this issue in their concept of "fast switching"). That said, we have not
implemented the speed header per se, but only because it was not yet
well enough defined, or widely enough implemented in our view. But the
server-side behavior is there, we simply use other means for controlling
it.

I also believe that the scale header can play a necessary part in a fast
switching environment as well (Magnus mentions some below), though I
believe it also has other uses.

Personally, I would love to see more explanation of and use cases for
these headers before a decision is made as to whether either can or
should be done away with. However, I admit that this isnt a trivial task
and perhaps that makes it unreasonable..? 

Thanks for bringing this up Magnus, I think it's an important topic.

Mike

-----Original Message-----
From: Jaehwan Kim [mailto:[email-masked]] 
Sent: Thursday, October 25, 2007 7:45 AM
To: Magnus Westerlund
Cc: mmusic (E-mail)
Subject: RE: [MMUSIC] RTSP: Speed header

Hi Magnus,

Cheers,
Jaehwan

-----Original Message-----
From: Magnus Westerlund [mailto:[email-masked]]
Sent: Tuesday, October 23, 2007 1:31 PM
To: Magnus Westerlund
Cc: mmusic (E-mail)
Subject: Re: [MMUSIC] RTSP: Speed header

Hi,

See inline:

Cheers

Magnus Westerlund

---

From: Dave Singer <[email-masked]>
Date: Sun, 28 Oct 2007 05:50:40 +0100

We've had some quite long discussions about these headers in various 
groups.  Most people find them confusing, and there are also problems 
in the model.

First, it does seem that there is a reasonable need for 'VCR' 
controls in a control protocol:  "send me the media at twice rate 
(fast forward), -1 (play reverse) and so on."

Second, the protocol should also accomodate being able to control 
bandwidth;  "send me at twice rate and you also may use twice the 
normal bandwidth".  I (and others) have thought in the past that RTSP 
had this (I can never remember which is scale and which is speed).

It seems fairly clear that in this case the usual link between RTP 
timestamp and transmission time is maintained (modulo traffic 
smoothing).

RTSP has a control to *detach* these two from each other, so you can 
ask for media at say, twice its *playout* rate according to its 
timestamps.  The use cases I have heard tend to center around buffer 
management, but I believe that that is more correctly handled in RTP 
using a protocol such as that defined by 3GPP.

Finally, when playing at non-unitary rates, the concept that the RTP 
receiver is unaware is very hard to maintain.  Double-speed video is 
fairly easily done (if the bandwidth is there) by spacing the RTP 
timestamps closer and therefore playing the frames closer in time. 
This does not work for audio;  a frame there lasts for say 1024 
samples, and it really doesn't achieve any useful effect to get them 
at 512 sample intervals.  Usually, one treats the new stream as being 
at double the sample rate, and you either play it at that rate, or 
down-sample it again, optionally also pitch-shifting.  These 
operations on the  decoded media are obviously well away from the RTP 
stack.

We have seen some interesting experiments in ISMA with dropping 
frames from compressed audio streams, and they work better than you 
might expect for at least the codecs tried, but it can hardly be 
considered a general technique.

So, in summary, should perhaps RTSP have
a) a command to set the playout rate to non-unity; with the default 
that the same bandwidth should be used;
b) a command to set the bandwidth limit for the stream, compared to 
what the normal stream would use at a normal play rate.
c) documentation that the media stack below RTP is necessarily aware 
of non-unity rates and does the appropriate media management;
d) pointers to buffer and bandwidth management systems such as 3GPP 
for fast buffer-fill and other considerations.

?
-- 
David Singer
Apple/QuickTime