Subject: Re: [AVT] H.26L packetization: the two properties of the RTP timestamp

From: [email-masked]
Date: Tue, 12 Mar 2002 11:50:52 +0100

Dear all,

I want to check something before we go on.

I am concerned that being deeply involved in video -which is more than 
understandable ;-)
we may  forget the *key* feature that RTP should give us:
the ability to synchronize with other data streams, and especially audio.

specifically a termimal will receive audio and one audio frame has a time 
stamp,
transported by the RTP TS.

if it receives video the purpose of the RTP TS is to
*assign a time stamp to an image* so that an audio/video
terminal can synchronize audio and video playback.

do we agree that this is the primary purpose here ?

regards,

Philippe Gentric
Software Architect
Philips MP4Net
[email-masked]
http://www.mpeg-4.philips.com

Thomas Stockhammer <[email-masked]>
Sent by: [email-masked]
03/11/2002 14:49

        To:     Stephan Wenger <[email-masked]>
        cc:     Jan vanderMeer/EHV/BE/PHILIPS@EMEA3
"M. Reha Civanlar" <[email-masked]>
[email-masked]
[email-masked]
(bcc: Philippe Gentric/LIM/CE/PHILIPS)
        Subject:        Re: [AVT] H.26L packetization: the two properties of the RTP timestamp
        Classification: 

Dear all,

the timestamp seems to be a real animal! Stephan, what do you think :-)?

> The authors of the JVT/H.26L packetization draft thought about
> using the decoding timestamp, but we dismissed it in favor of
> the sending time.  I don't recall the precise reason, but I believe
> we had some uneasiness with the term itself, as it is (so far) not
> defined in JVT.
>
> I would be willing to change the draft to "decoding timestamp", and
> this will solve the dual-property problem of the RTP timestamp --
> assuming that we mean the timestamp for the decoding of one
> slice, not picture.  For the frequently discussed reasons we don't know
> the decoding timestamp of the whole picture until the final slcie
> is sent.
>
> Whether this approach will facilitate media synchronization is another
> problem.  I don't believe it does.  But I'm willing to listen.

I was actually in favor of using the the RTP as DTS when drafting this
proposal.
But we were not sure if this does not cause any other problems. So we went 
with

the one which seemed to be less ambigious, RFC2250. However, I still 
support
very much the RTP = DTS concept unless anyone tells me a real problem.

About buffers and timestamp, I have a kind of naive idea.

What is this all necessary for. The encoder has to produce something
the decoder is capable to decode under some buffer constraints.

The DTS is the time the slice (!, not frame) is removed from the receiver
buffer,
decoded and the decoded slice (!) is moved to the reference frame buffer!
The video has to be coded such that the receiver buffer can take the sum 
of all
bits
assigned to all slices with DTS>current_time.

In addition, we have a time-stamp which allows to remove a frame from the
reference frame buffer. This is neither PTS nor DTS but something inherent
in H.26L. The encoder should also take care that the decoder can take
all slices in the reference frame buffer (pixel based).

The PTS is again different. A third buffer might be a display buffer which
takes all frames which are no more in the reference frame buffer but where
not display yet because of the PTS. Is this necessary?

The profiles give upper limits on receiver buffer and reference frame 
buffer.

Any problems with this concept?

Best regards

Tom

---

From: Stephen Casner <[email-masked]>
Date: Sun, 17 Mar 2002 19:31:23 -0800 (PST)

In this message I've picked out snippets from several messages in the
H.26L thread and inserted comments inline.  I believe that the RTP
timestamp should be the sampling / presentation timestamp (the two are
roughly equivalent).  I like the design philosphy that has been
espoused, but I don't think the current proposal is heading there.
There is still too much of a tendency to view transmission over IP as
if it were a fixed-bandwidth channel to be managed by the video
encoder filling it with various kinds of data.  IP does flow over
paths with limited bandwidth, but the problem is that the sharing of
bandwidth over that channel needs to consider not just the various
kinds of data that the video encoder knows about, but also other kinds
of traffic from other sources.  The video encoder should separate the
different kinds of data so that they can be managed appropriately with
the other traffic flows.

On Thu, 7 Mar 2002 [email-masked] wrote:
> dont you agree that a "perfect" media codec would be one that
> would provide "packets" that you can simply *fit* into one RTP
> packet as payload ?

Yes, I believe I have said similar words on several occasions.
However, not only does the codec packet need to fit into the RTP
payload, but we need meaningful values to go into the RTP header
fields as well.  Especially the timestamp.

On Tue, 12 Mar 2002 [email-masked] wrote:
> specifically a termimal will receive audio and one audio frame has a
> time stamp, transported by the RTP TS.
>

The primary purpose of the RTP timestamp is intra-media
synchronization, that is, to allow the relative timing of stream
fragments carried in RTP packets to be recreated.

The secondary purpose of the RTP timestamp is inter-media
synchronization, that is, after we have reconstructed the timing of
two streams, to play them out with the right time relationship to each
other.

A third purpose of the RTP timestamp is in the jitter calculation, as
a means to feed back a measure of the quality of the network
transmission.  This purpose is a distant third.  The packet loss rate
is the primary measure of transmission quality; the only reason for
including jitter is that it may provide an indication sooner, but we
should not get too hung up on the jitter measure.

On Thu, 7 Mar 2002, Stephan Wenger wrote:
> The RTP timestamp seems to have two properties (and I almost
> hope that someone corrects me and says that one of the two is
> a mis-interpretation of RFC1889):
> a) To convey media timing information (often sampling instant,
>    sometimes presentation time, both are equivalent wrt. the
>    following discussions)
> b) To implement the jitter buffer management.  Citing RFC1889:
>    Section 6.3.1 (RTCP), paragraph on arrival jitter : "[...]the relative
>    transit time is the difference between a packet's RTP timestamp
>    and the receiver's clock at the time of arrival, measured in the same
>    units.

I believe the RTCP jitter measure, which is inter-arrival delay
variation, should _not_ be used for buffer management.  The right
measure is phase jitter, i.e., the variation in delay between when the
data arrives and when it gets presented.  That's how long it needs to
be buffered in total, which may involve multiple states.  (Storage for
reference pictures may be in addition to that).  The buffer
requirement depends on variation in decode time and transmit order.

> As soon as features such as B-frames come into the picture (where
> the timestamp does not increase monotically with the sequence
> number), an RTP payload spec designer has two choices: stick
> to the "sampling instant" concept for the timestamp and de facto
> require much larger jitter buffers (by allowing higher jitter), or bind
> the timestamp to something more appropriate than the sampling
> time.  The former seems to make media synchronization easier
> but increases delay, the latter may reduce delay but disallows for
> perfect synchronization.

I disagree.  I believe the error in this logic is that a jitter buffer
based on the RTCP jitter measure should be put ahead of all the other
buffers.  When the B frames are transmitted out of order, then
additional delay is imposed, but that delay comes after the decoder.

The total delay stems from the fact that the transmission order is
perturbed and that there are prediction dependencies.  It is entirely
independent of what goes in the RTP timestamp field.

> H.26L is the first simple (non multiplexed in the sense of MPEG)
> media codec I'm aware of that has the capability of de-coupling
> the transmission time from the sampling/presentation time (though
> not decoding time).  For some packets, neither the sampling nor
> the presentation time in the very sense of the words are actually
> defined.  In the example in the draft, the packets of the commercial
> were sampled at an unknown time, and are presented at an unknown
> time (probably never) but they are transmitted in the stream.

So those packets of the commercial should be in a separate stream in
some way.  Or, if they stay in the same stream, they can just have the
same timestamp as the immediately preceding timed frame as if that
frame was just larger.

> When we wrote the draft, we looked at other payload RFCs and found
> the solution of RFC2250 which perfectly solves the jitter buffer
> and jitter calculation problem. Hence, we used it.  However, that means
> that we have to rely for exact timing information to the media stream
> itself ONLY -- though the RTP timestamp can be used to roughly
> synchronize media streams that do not use Enhanced RPS.  Same is
> IMHO the case for RFC2250.

RFC2250 has two forms.  What I assume you mean is the program stream
mode, as opposed to the elementary stream mode.  The program stream
mode is makes marginal use of RTP and is definitely _not_ the best
example to draw from.

If H.26L makes this choice, then what it really means is that you are
looking at RTP as a circuit emulation.  That seems to me to be exactly
the opposite of the philosophy you have espoused that the encoding
would be knowledgable of the transmission.

On Fri, 8 Mar 2002, Stephan Wenger wrote:
> Not network jitter, the difference time[..].
> When you use an IBBP... coding scheme, the transmission order is
> IPBB, but the sampling and presentation order is IBBP.  This is because
> you need the reconstructed P frame to reconstruct either of the B frames.
> At the receiver, you don't know when a packet was sent, and you also
> don't know whether a packet contains something weird like a B picture.
> Hence, when using the sampling/presentation time stamp, your jitter
> buffer size has to be big (timewise), because it has to "store" at
> least the P picture.

Why?  The receiver should hand the packet to the decoder, which will
know what kind of frame it is.  The docoder can choose not to decode a
piece of data if it doesn't yet have a dependency, or if it must shed
load.  That is, the jitter buffer should be in the decoder, not ahead
of it.

>  Also, your RTCP receiver reports will report a
> jitter much higher than the network jitter, namely the number of B
> frames multiplied by the picture clock interval (assuming a fixed
> picture clock interval, of course).

I claim we should not worry about this.

On Mon, 11 Mar 2002 [email-masked] wrote:
> Maybe the way to solve this TS issue is to first cope with the
> buffer size issue, since we seem to always stumble on it.
>
> You are assuming (decoding) buffer sizes may have to be changed
> (increased) during decoding ?
>
> I expect profiles will give upper limits ?
>
> I feel it is a civilized thing to do that the sender indicates buffer
> sizes to the receiver,
>
> BTW for IC design or software implementations on low cost platforms,
> may I remind everybody that  playing around with dynamic buffer allocation
> is still not a cost effective option ;-)

I don't understand why "dynamic" should be a particular problem.  I
can understand that there may be memory size limits, but adjusting up
to that limit should be manageable.  If you want to tolerate the
variance of the Internet and still try to minimize latency, you've got
to be dynamic.
							-- Steve

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 18 Mar 2002 07:19:51 +0100

Hi Steve,

Thanks for your comments.  A few replies inline:

>should not get too hung up on the jitter measure.

Thanks for the clarification.  All you said is true from the design
philosophy point-of-view.

The reason why I asomewhat insistent on the jitter issue is
not so much the RTCP RR.  It is more that receivers tune their
buffers to the appropriate size using a jitter calculation method
very similar to the jitter calculation.  This IS important, because
it directly influences the end-to-end delay, which is the most
critical property of conversational video.

>requirement depends on variation in decode time and transmit order.

I'm not so sure about this one.  But there may be a big misunderstanding
on my side.

Your model (using the phase jitter) assumes that you know a presentation
time for each packet you send.  Ignoring H.26L and its ERPS properties
for now -- this seems to imply that you really want to size the jitter buffer
long enough to include all the RTP packets of a BBP picture video sequence.
There is no technical need for doing this, and it adds very significant delay.
I don't understand, why this is necessary.

>independent of what goes in the RTP timestamp field.

No.  This would be true if you could assume indefinite decoder speed,
e.g. the decoder can decode a B picture at zero delay.  Practically,
this is not the case.

Your (clean) method would lead to more delay than necessary
for a real-world system: First, the RTP buffer would be big enough
to include all packets of a BBP picture sequence.  The P picture
would not be conveyed to the decoder after the second B picture
is available. Second, you decode PBB, and add (at least) one frame
delay for the decoding of the P picture.

All this sounds like a very bad idea in a low delay system.

>same timestamp as the immediately preceding timed frame as if that

No other RTP stream: the packets belonging to the commercial are really
part of the same decoder state.  I hope to be able to make this clearer
during the meeting.

Yes, you can assign them the time stamp of the preceding frame.  We have
considered this.  The problem is, if you do that you loose the intra-stream
synchronization.  You cannot distinguish packets that belong to different
pictures any more by just looking at the RTP timestamp.  I could imagine
a "minus one" timestamp rule: ERPS-Slices have the presentation timestamp
of the current picture minus one.  Maybe this is the solution we are all
looking for.  (Second thought: maybe not.  In H.26L you could, theoretically,
send ONLY information that is never presented, or presented at a time
in the far future you don't know yet.  Difficult to explain.  But I could 
imagine
cases where such a use makes sense, e.g. during the setup of the football
game (transmitting the I pictures of many commercials).  Sound like an off-line
discussion to me.

>load.  That is, the jitter buffer should be in the decoder, not ahead

But what do you do if you have re-ordering or packet losses?  Fact is,
you don;t want to start error concealment before you know for sure that
the missing data is really missing and not only delayed (but within the
jitter buffer size).

See you in Minneapolis, and eagerly awaiting fruitful discussions there,
Stephan