Subject: [AVT] H.26L packetization: the two properties of the RTP timestamp

From: Stephan Wenger <[email-masked]>
Date: Thu, 07 Mar 2002 17:11:04 +0100

Folks,

I have started this new thread in order to discuss one observation
we have made when writing the H.26L packetization draft.  The
issue, however, may also apply to other forthcoming packetization
schemes.

The RTP timestamp seems to have two properties (and I almost
hope that someone corrects me and says that one of the two is
a mis-interpretation of RFC1889):
a) To convey media timing information (often sampling instant,
   sometimes presentation time, both are equivalent wrt. the
   following discussions)
b) To implement the jitter buffer management.  Citing RFC1889:
   Section 6.3.1 (RTCP), paragraph on arrival jitter : "[...]the relative
   transit time is the difference between a packet's RTP timestamp
   and the receiver's clock at the time of arrival, measured in the same
   units.

For continuous media streams, say G.711 audio, such a coupling
of functionalities is not problematic.  As thousands of implementations
show us, it works very well.

As soon as features such as B-frames come into the picture (where
the timestamp does not increase monotically with the sequence
number), an RTP payload spec designer has two choices: stick
to the "sampling instant" concept for the timestamp and de facto
require much larger jitter buffers (by allowing higher jitter), or bind
the timestamp to something more appropriate than the sampling
time.  The former seems to make media synchronization easier
but increases delay, the latter may reduce delay but disallows for
perfect synchronization.

When designing RFC 2429 (H.263+ payload spec), for example,
we decided to use the "presentation time" because of the sync-
problems associated with layered coding, and because no one
with a sane mind would use B-frames in a low-delay environment
anyway (hence the additional delay of a big jitter buffer seemed
acceptable).

H.26L is the first simple (non multiplexed in the sense of MPEG)
media codec I'm aware of that has the capability of de-coupling
the transmission time from the sampling/presentation time (though
not decoding time).  For some packets, neither the sampling nor
the presentation time in the very sense of the words are actually
defined.  In the example in the draft, the packets of the commercial
were sampled at an unknown time, and are presented at an unknown
time (probably never) but they are transmitted in the stream.

When we wrote the draft, we looked at other payload RFCs and found
the solution of RFC2250 which perfectly solves the jitter buffer
and jitter calculation problem. Hence, we used it.  However, that means
that we have to rely for exact timing information to the media stream
itself ONLY -- though the RTP timestamp can be used to roughly
synchronize media streams that do not use Enhanced RPS.  Same is
IMHO the case for RFC2250.

Currently, I do not see a better solution.

Stephan

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: [email-masked]
Date: Fri, 8 Mar 2002 15:21:44 +0100

Stephan,

> time.

Do you mean network jitter or are you referring to the difference in time between transport of the data and the presentation of that data. Could you please also elaborate on why the "sampling instant" approach requires much larger jitter buffers than the
"RFC 2250" approach. I would rather expect the opposite.

> However, that means

Is my assumption correct that at the reciever you would use the RTP time stamp to reconstruct the timing of the H.26L video stream, and that this timing is than used for synchronization (in full analogy with the transport of an MPEG-2 transport stream
using RFC 2250) ?

Best regards,

Jan

Jan van der Meer
Philips Digital Networks - MP4Net
Prof Holstlaan 4
Building WDB-1
P.O. Box 80002
5600 JB Eindhoven
The Netherlands
Phone +31 402 735 774
Fax +31 402 735 545

                    Stephan                                                                                        
                    Wenger                  To:  [email-masked]                                                      
                    <[email-masked]           cc:  [email-masked]                                        
                    berlin.de>               thomas Stockhammer <[email-masked]>                            
                    Sent by:                 (bcc: Jan vanderMeer/EHV/BE/PHILIPS)                                  
                    avt-admin@iet           Subject:  [AVT] H.26L packetization: the two properties of the RTP     
                    f.org                    timestamp                                                             
                                                                                                                   
                                            Classification:                                                        
                    2002-03-07                                                                                     
                    05:11 PM                                                                                       

Folks,

Stephan

---

From: [email-masked]
Date: Mon, 11 Mar 2002 10:20:21 +0100

Reha,

This is an interesting discussion. I guess that it would help to let RTP time stamp indicate the Decoding Time Stamp, but that such approach causes other problems, is that right ?

Best regards,

Jan

                    "M. Reha                                                                                           
                    Civanlar"                   To:  Jan vanderMeer/EHV/BE/PHILIPS@EMEA3                               
                    <civanlar@researc            "Stephan Wenger" <[email-masked]>                              
                    h.att.com>                  cc:  <[email-masked]>                                                    
                                                 <[email-masked]>                                          
                    2002-03-08 07:48             "thomas Stockhammer" <[email-masked]>                          
                    PM                          Subject:  Re: [AVT] H.26L packetization: the two properties of the RTP 
                    Please respond to            timestamp                                                             
                    "M. Reha                                                                                           
                    Civanlar"                   Classification:                                                        

The following is an excerpt from the minutes of the 51st IETF that is
related to this discussion. I just wanted to remind that I am still waiting
for the WG's decision on this issue to make a change in RFC 2250. Taking no
action will be equivalent to deciding that the jitter computation results
reported in RTCP are actually media dependent and media unaware network
elements should be careful about determining network conditions based on
these results (they can still use the jitter computation results for
qualitative comparisons among servers using the same type of media,
however). Any comments?

----------------------------------------------------------------------------
----
"The last functional change requested was related to the RTCP jitter
computation.  An MPEG video frame may need to be carried in more than
one RTP packet.  All such packets carry the same timestamp although
they are sent at different times, and this causes the calculated
jitter to be larger than the true network jitter.  This problem exists
for other payload formats, too, but it is particularly severe when
B-frames are used in MPEG.  The send times of both P- and B-frames are
not related to their timestamps because a future P-frame must be sent
prior to all B-frames that refer to it.  Reha noted that there was a
recent comment on the list stating that the jitter computation based
on the existing definition may actually be useful for the receiver
buffer size determination.  He pointed out that the problem will be
with the media unaware network elements which may try to determine
network state parameters, such as congestion, based on jitter
computations which will not reflect the network jitter in such cases.

Reha suggested that the MPEG payload format should specify a modified
jitter computation that accurately reflects network jitter.  Steve
Casner asked for WG feedback on whether it is reasonable for a payload
format to specify a change in the jitter behavior from what the RTP
spec defines or whether the jitter computation in the RTP spec needs
to be modified in some way.  Steve said a concern is that the
suggested approach requires significant knowledge about the datastream
in order to do the jitter calculation and in many implementations this
may be done in a payload independent portion of the implementation.
Reha stated that the results computed this way will be wrong.
Marshall Eubanks said it doesn't seem to be a good idea to require
people to do something that is wrong.  Steve reiterated his previous
comment about the use of the jitter as a qualitative value to compare
the performance of several servers or a single server over time.  This
is a significant issue that needs to be discussed further on the list."
---------------------------------------------------------------------------
M. Reha Civanlar
[email-masked]

----- Original Message -----
From: "Stephan Wenger" <[email-masked]>
To: <[email-masked]>
Cc: "Stephan Wenger" <[email-masked]>; <[email-masked]>;
<[email-masked]>; <[email-masked]>; "thomas Stockhammer"
<[email-masked]>
Sent: Friday, March 08, 2002 11:50 AM
Subject: Re: [AVT] H.26L packetization: the two properties of the RTP
timestamp

> At 03:21 PM 3/8/2002 +0100, [email-masked] wrote:
> >Stephan,
> >
> >Do you mean network jitter or are you referring to the difference in time
> >between transport of the data and the presentation of that data. Could
you
> >please also elaborate on why the "sampling instant" approach requires
much
> >larger jitter buffers than the
> Not network jitter, the difference time[..].
> When you use an IBBP... coding scheme, the transmission order is
> IPBB, but the sampling and presentation order is IBBP.  This is because
> you need the reconstructed P frame to reconstruct either of the B frames.
> At the receiver, you don't know when a packet was sent, and you also
> don't know whether a packet contains something weird like a B picture.
> Hence, when using the sampling/presentation time stamp, your jitter
> buffer size has to be big (timewise), because it has to "store" at
> least the P picture.  Also, your RTCP receiver reports will report a
> jitter much higher than the network jitter, namely the number of B
> frames multiplied by the picture clock interval (assuming a fixed
> picture clock interval, of course).
>
> If you use the transmission time, things are easy.  The order of the
> timestamp is monotic as is the order of the sequence number, and the
> jitter buffer size is minimal and dependent only on the observed network
> jitter.
>
> > > However, that means
> >Is my assumption correct that at the reciever you would use the RTP time
> >stamp to reconstruct the timing of the H.26L video stream, and that this
> >timing is than used for synchronization (in full analogy with the
> >transport of an MPEG-2 transport stream
> >using RFC 2250) ?
>
> I think so.  This is one of the areas where I'm not really sure what to
do.
> The current H.26L WD contains only the most simple timing information.
> It is my believe that, at the end, H.26L will carry some form of exact
> relative timing information for some applications and NALs.  For others,
> e.g. for low bit rate links, it may be advisable to save the bits and rely
> for synchronization only on the arrival time of the data -- the way most
> H.320 systems work today (ignoring the video TR, and display pictures
> as soon as they are reconstructed).  There will be proposals for this work
> item at the next JVT meeting in Fairfax.  My personal opinion would be
> to use SEI packets for carrying relative (or even absolute) exact timing
> information that is bound to a certain reference buffer -- which can be
> the one that contains the currently decoded picture.  The IP/RDP H.26L
> NAL could furthermore define a SEI message on its own that establishes
> synchronization between the video timing and the transmission time.
> But, as I said,  all these are just an ideas for now, not generally
accepted
> wisdom or WD content.

---

From: [email-masked]
Date: Mon, 11 Mar 2002 10:24:34 +0100

Hello Stephan,

> jitter buffer size is minimal and dependent only on the observed network

But if the IBBP structure of the bitstream remains the same, you still need to store somewhere the I and P pictures to reconstruct the B frames, so in practice there will be no real difference in buffer requirements. Or do I miss something ?

Best regards,

Jan

                    Stephan                                                                                        
                    Wenger                  To:  Jan vanderMeer/EHV/BE/PHILIPS@EMEA3                               
                    <[email-masked]           cc:  Stephan Wenger <[email-masked]>                            
                    berlin.de>               [email-masked]                                                          
                                             [email-masked]                                                    
                    2002-03-08               [email-masked]                                            
                    05:50 PM                 thomas Stockhammer <[email-masked]>                            
                                            Subject:  Re: [AVT] H.26L packetization: the two properties of the RTP 
                                             timestamp                                                             
                                                                                                                   
                                            Classification:                                                        

At 03:21 PM 3/8/2002 +0100, [email-masked] wrote:
>between transport of the data and the presentation of that data. Could you
>please also elaborate on why the "sampling instant" approach requires much

>transport of an MPEG-2 transport stream

I think so.  This is one of the areas where I'm not really sure what to do.
But, as I said,  all these are just an ideas for now, not generally accepted
wisdom or WD content.

Stephan

---

From: Stephan Wenger <[email-masked]>
Date: Mon, 11 Mar 2002 14:27:11 +0100

At 10:24 AM 3/11/2002 +0100, [email-masked] wrote:
>But if the IBBP structure of the bitstream remains the same, you still 
>need to store somewhere the I and P pictures to reconstruct the B frames, 
>so in practice there will be no real difference in buffer requirements. Or 
>do I miss something ?

For classic B frames that may be the case.  However, pease bear in mind that

a) the MPEG decoder knows about B frames, RTP does not.  Hence, the MPEG
decoder could adjust its buffer size as soon as it starts detecting B frames,
whereas you cannot expect this type of media awareness from the RTP 
implementation
(which should react to the network jitter only).  The problem did not occur 
so far,
because few (if any) systems change from a no-B-frame mode to B-frames during a
session.

b) The timing requirements for H.26L's multi-hypothesis frames are much more
relaxed than for B frames.  Using B frames it is implied that you always 
use the
temporally neighboring I/P frames as a reference; but not in H.26L.

c) B-Frames were just an example where the current binding of the sampling 
instant/
presentation time to the network transmission time (for jitter calculation) 
has a
problem.  So far, these problems were managable.  I believe, with H.26L at the
horizon they are no more.

Regards,
Stephan

---

From: [email-masked]
Date: Mon, 11 Mar 2002 14:49:20 +0100

Stephan,

Maybe the way to solve this TS issue is to first cope with the buffer size 
issue,
since we seem to always stumble on it.

You are assuming (decoding) buffer sizes may have to be changed 
(increased) during decoding ?

I expect profiles will give upper limits ?

I feel it is a civilized thing to do that the sender indicates buffer 
sizes to the receiver,

BTW for IC design or software implementations on low cost platforms,
may I remind everybody that  playing around with dynamic buffer allocation
is still not a cost effective option ;-)

Regards,

Philippe Gentric
Software Architect
Philips MP4Net
[email-masked]
http://www.mpeg-4.philips.com

Stephan Wenger <[email-masked]>
Sent by: [email-masked]
03/11/2002 14:27

        To:     Jan vanderMeer/EHV/BE/PHILIPS@EMEA3
        cc:     Stephan Wenger <[email-masked]>
[email-masked]
[email-masked]
thomas Stockhammer <[email-masked]>
(bcc: Philippe Gentric/LIM/CE/PHILIPS)
        Subject:        Re: [AVT] H.26L packetization: the two properties of the RTP   timestamp
        Classification: 

>so in practice there will be no real difference in buffer requirements. 
Or 

For classic B frames that may be the case.  However, pease bear in mind 
that

a) the MPEG decoder knows about B frames, RTP does not.  Hence, the MPEG
decoder could adjust its buffer size as soon as it starts detecting B 
frames,
whereas you cannot expect this type of media awareness from the RTP 
implementation
(which should react to the network jitter only).  The problem did not 
occur 
so far,
because few (if any) systems change from a no-B-frame mode to B-frames 
during a
session.

b) The timing requirements for H.26L's multi-hypothesis frames are much 
more
relaxed than for B frames.  Using B frames it is implied that you always 
use the
temporally neighboring I/P frames as a reference; but not in H.26L.

instant/
presentation time to the network transmission time (for jitter 
calculation) 
has a
problem.  So far, these problems were managable.  I believe, with H.26L at 
the
horizon they are no more.

Regards,
Stephan

---

From: [email-masked]
Date: Mon, 11 Mar 2002 22:28:24 +0200

Dear all,

Stephan wrote:
> > The authors of the JVT/H.26L packetization draft thought about
> > using the decoding timestamp, but we dismissed it in favor of
> > the sending time.  I don't recall the precise reason, but I believe
> > we had some uneasiness with the term itself, as it is (so far) not
> > defined in JVT.
> >
> > I would be willing to change the draft to "decoding timestamp", and
> > this will solve the dual-property problem of the RTP timestamp --
> > assuming that we mean the timestamp for the decoding of one
> > slice, not picture.  For the frequently discussed reasons 
> > we don't know
> > the decoding timestamp of the whole picture until the final slcie
> > is sent.
> >
> > Whether this approach will facilitate media synchronization 
> > is another
> > problem.  I don't believe it does.  But I'm willing to listen.

Tom responded: 
> I was actually in favor of using the the RTP as DTS when drafting this
> proposal.
> But we were not sure if this does not cause any other 
> problems. So we went with
> the one which seemed to be less ambigious, RFC2250. However, 
> I still support
> very much the RTP = DTS concept unless anyone tells me a real problem.

I believe we had three basic reasons against DTS as the RTP timestamp:

1) It was not clear how to define the RTP timestamp of compound packets. Each slice in the compound packet should have its own DTS. While the RTP timestamp can carry the DTS of only one slice of a compound packet, a separate mechanism to carry the DTS for the rest of the slices would have been needed anyway. For consistency, it is probably better to use the same mechanism to carry DTS regardless of the use of single slice mode or compound packets. As Stephan already mentioned in an earlier email, a good mechanism to carry DTS and PTS is the use of Supplemental Enhancement Information. See document JVT-B109 (http://standard.pictel.com/ftp/video-site/0201_Gen/JVT-B109.doc) for further details.

2) In a streaming application, a transmitter may want to transmit intra frames earlier than their natural decoding position in order to allow more time for possible retransmissions of these intra frames (see more discussion below). Use of DTS as RTP timestamp would ruin reliable RTP interarrival jitter calculation of packets whose actual transmission time differs from DTS. 

In such systems that allow retransmission, streaming clients typically have a receiver buffer that is capable of storing a relatively large amount of data.  Initially, when a streaming session is establish, a client does not start playing the stream back immediately, but rather it typically buffers the incoming data for a few seconds.  This buffering helps to maintain pauseless playback, because, in case of occasional increased transmission delays or network throughput drops, the client can decode and play buffered data.  The buffering is also necessary for either automatic or selective retransmission in any protocol level.  If any part of a picture is lost, a retransmission mechanism may be used to resend the lost data.  If the retransmitted data is received before its scheduled decoding or playback time, the loss is perfectly recovered.  When this mode is in use, non-disposable pictures can be sent earlier than their natural coding order indicates, whereas disposable picture!
s, such as conventional B pictures, can be sent later than their natural coding order indicates.  Consequently, any retransmitted parts of non-disposable pictures are more likely to be received before their scheduled decoding or playback time compared to parts of retransmitted disposable pictures.

3) We were (or mainly I was) not sure that DTS is needed or used in practice in some receivers:
- In the JVT codec, the decoding order of pictures is carried in picture numbers/IDs. Thus, decoding timestamp is not needed to resolve the decoding order.
- In conversational applications, a typical receiver maintains a small transmission delay jitter buffer. When a packet is removed from the jitter buffer, it is decoded as fast as possible regardless of its decoding timestamp.

Tom wrote:
> About buffers and timestamp, I have a kind of naive idea.
> 
> What is this all necessary for. The encoder has to produce something
> the decoder is capable to decode under some buffer constraints.
> 
> The DTS is the time the slice (!, not frame) is removed from 
> the receiver
> buffer,
> decoded and the decoded slice (!) is moved to the reference 
> frame buffer!
> The video has to be coded such that the receiver buffer can 
> take the sum of all
> bits
> assigned to all slices with DTS>current_time.
> 
> In addition, we have a time-stamp which allows to remove a 
> frame from the
> reference frame buffer. This is neither PTS nor DTS but 
> something inherent
> in H.26L. The encoder should also take care that the decoder can take
> all slices in the reference frame buffer (pixel based).
> 
> The PTS is again different. A third buffer might be a display 
> buffer which
> takes all frames which are no more in the reference frame 
> buffer but where
> not display yet because of the PTS. Is this necessary?
> 
> The profiles give upper limits on receiver buffer and 
> reference frame buffer.
> 
> Any problems with this concept?

I agree with this roughly. I don't think we need a "third buffer", but rather the reference picture buffer and the display buffer can be combined. My input contributions VCEG-O45r1 and JVT-B089 discussed the issue as follows:

"Decoders may need to buffer multiple uncompressed pictures for two reasons: First, the pictures can be used as a reference for motion compensation. Conventionally, a picture buffer similar to the one in H.263 Annex U has been used. Second, buffering might be required to enable displaying at a correct rate. For example, conventional B pictures are decoded after its anchor pictures but displayed before its latter anchor picture. We propose that the buffer handling for both of these cases is unified. We call the unified buffer as the post-decoder buffer. Notice that some pictures of the post-decoder buffer, such as conventional B pictures, cannot be referred to in the motion compensation process. A picture can be removed from the post-decoder buffer when it is displayed and when it is no longer needed as a reference picture."

Best Regards,
Miska

---

From: "Gary Sullivan" <[email-masked]>
Date: Mon, 11 Mar 2002 15:32:50 -0800

Obviously, the issue of timing itself is a matter needing some
additional work in the JVT.  Since the timing model of the JVT
codec is not yet fully sorted out, it may take some time to
sort out exactly how the packetization should deal with it.

This packetization draft was submitted before being fully coordinated
with the JVT organization, so it is not surprising that it may have
some issues that need reconciliation with the yet-to-be-finalized
JVT design.

The desire expressed by the JVT and its parent bodies has been to
have one defined packetization method for this type of video
for all applications, and for that definition to be coordinated with
the JVT in its development.

I therefore ask the IETF to consider this draft to be only
preliminary input on the needs of this codec.  The JVT and its
parent bodies have expressed a strong interest in the packetization
design for its codec, and will continue to have a strong interest
as work progresses.

Best Regards,

-Gary S.

+> -----Original Message-----
+> From: Stephan Wenger [mailto:[email-masked]] 
+> Sent: Monday, March 11, 2002 5:32 AM
+> To: [email-masked]
+> Cc: M. Reha Civanlar; [email-masked]; 
+> [email-masked]; Stephan Wenger; thomas Stockhammer
+> Subject: Re: [AVT] H.26L packetization: the two properties 
+> of the RTP timestamp
+> 
+> 
+> Hi Jan,
+> 
+> At 10:20 AM 3/11/2002 +0100, [email-masked] wrote:
+> >This is an interesting discussion. I guess that it would 
+> help to let RTP 
+> >time stamp indicate the Decoding Time Stamp, but that such 
+> approach causes 
+> >other problems, is that right ?
+> 
+> The authors of the JVT/H.26L packetization draft thought about
+> using the decoding timestamp, but we dismissed it in favor of
+> the sending time.  I don't recall the precise reason, but I believe
+> we had some uneasiness with the term itself, as it is (so far) not
+> defined in JVT.
+> 
+> I would be willing to change the draft to "decoding timestamp", and
+> this will solve the dual-property problem of the RTP timestamp --
+> assuming that we mean the timestamp for the decoding of one
+> slice, not picture.  For the frequently discussed reasons we 
+> don't know
+> the decoding timestamp of the whole picture until the final slcie
+> is sent.
+> 
+> Whether this approach will facilitate media synchronization 
+> is another
+> problem.  I don't believe it does.  But I'm willing to listen.
+> 
+> Regards,
+> Stephan
+> 
+> 
+> 
+> _______________________________________________
+> Audio/Video Transport Working Group
+> [email-masked]
+> https://www1.ietf.org/mailman/listinfo/avt
+>