Subject: [AVT] RTP: explanation of "sampling instant"

From: Stephen Casner <[email-masked]>
Date: Tue, 7 Jan 2003 15:08:51 -0800 (PST)

Here is the second edit to the RTP spec.  In Section 5.1, I've added
text at the end of the specification of the "timestamp" field in the
RTP header to explain the notion of "sampling instant" better.  Do you
think this helps or hinders?  Should I cut the last paragraph?  I
included it to give an idea how sampling relates to prerecorded media.

                                                        -- Steve

OLD:
        timestamp: 32 bits
             The timestamp reflects the sampling instant of the first
             octet in the RTP data packet.  The sampling instant MUST be
             derived from a clock that increments monotonically and
             linearly in time to allow synchronization and jitter
             calculations (see Section 6.4.1).  The resolution of the
             clock MUST be sufficient for the desired synchronization
             accuracy and for measuring packet arrival jitter (one tick
             per video frame is typically not sufficient).  The clock
             frequency is dependent on the format of data carried as
             payload and is specified statically in the profile or
             payload format specification that defines the format, or
             MAY be specified dynamically for payload formats defined
             through non-RTP means.  If RTP packets are generated
             periodically, the nominal sampling instant as determined
             from the sampling clock is to be used, not a reading of the
             system clock.  As an example, for fixed-rate audio the
             timestamp clock would likely increment by one for each
             sampling period.  If an audio application reads blocks
             covering 160 sampling periods from the input device, the
             timestamp would be increased by 160 for each such block,
             regardless of whether the block is transmitted in a packet
             or dropped as silent.

             The initial value of the timestamp SHOULD be random, as for
             the sequence number.  Several consecutive RTP packets will
             have equal timestamps if they are (logically) generated at
             once, e.g., belong to the same video frame.  Consecutive
             RTP packets MAY contain timestamps that are not monotonic
             if the data is not transmitted in the order it was sampled,
             as in the case of MPEG interpolated video frames.  (The
             sequence numbers of the packets as transmitted will still
             be monotonic.)

NEW ADDITION:
             The sampling instant is chosen as the point of reference
             for the RTP timestamp because it is known to the
             transmitting endpoint and has a common definition for all
             media independent of encoding delays or other processing.
             The purpose is to allow synchronized presentation of all
             media sampled at the same time.  Comparing RTP timestamps
             from different media is not effective because they have
             independent random offsets and may advance at different
             rates.  Instead, for each medium the RTP timestamp is
             related to the sampling instant by pairing it with a
             timestamp from a reference clock (wallclock) that
             represents the time when the data corresponding to the RTP
             timestamp was sampled.  The reference clock is shared by
             all media to be synchronized.  The timestamp pairs are not
             transmitted in every data packet, but at a lower rate in
             RTCP SR packets as described in Section 6.4.

             Applications transmitting stored data rather than data
             sampled in real time typically use a virtual clock to
             determine when the next frame or other unit of each medium
             in the stored data should be presented.  In this case, the
             sampling instant would be the presentation time for each
             unit, and it would be related to the wallclock time at
             which the unit was determined to be ready for presentation.

             Live and prerecorded media can also be synchronized, for
             example in a live audio narration of prerecorded video.
             The video would be presented locally to the narrator as
             well as transmitted using RTP.  The sampling instant of the
             video would be established by referencing the timestamp of
             the video frame to the wallclock time when the video frame
             is presented locally and the corresponding audio is
             sampled.

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Jonathan Rosenberg <[email-masked]>
Date: Tue, 07 Jan 2003 23:32:32 -0500

I think that adding some additional explanation would help, but I found 
certain aspects of the text confusing. Specific comments inline:

>              media independent of encoding delays or other processing.

It took me three reads to parse this. I would propose a comma:

The sampling instance is chosen as the point of reference for the RTP 
timestamp because it is known to the transmitting endpoint and has a 
common definition for all media, independent of encoding delays or other 
processing

>              rates. 

Aren't these two sentences contradictory? The first says its for 
synchronized presentation, and the second says it can't be used for that 
because they have random offsets and advance at different rates.

I always thought that the reason we had a separate RTP timestamp and 
wallclock timestamp was overhead. Because fine granularity is needed for 
each wallclock time, it requires too many bits to send in each RTP 
packet. Rather, a timestamp whose granularity is exactly matched to the 
media is used, and then correlated to the less efficient wallclock time 
in the infrequent RTCP packets.

> Instead, for each medium the RTP timestamp is
>              in the stored data should be presented.

I think "determine" is the wrong word here. Its the receiver that 
determines when to present it. The sender indicates when it should be 
presented. So, I would propose:

Applications transmitting stored data rather than data sampled in real 
time typically use a virtual clock to indicate when the next frame (or 
other unit of each medium) in the stored data should be presented.

-Jonathan R.
-- 
Jonathan D. Rosenberg, Ph.D.                72 Eagle Rock Ave.
Chief Scientist                             First Floor
dynamicsoft                                 East Hanover, NJ 07936
[email-masked]                     FAX:   (973) 952-5050
http://www.jdrosen.net                      PHONE: (973) 952-5000
http://www.dynamicsoft.com

---

From: Stephen Casner <[email-masked]>
Date: Wed, 8 Jan 2003 17:51:31 -0800 (PST)

> common definition for all media, independent of encoding delays or other

Yes, that is better.  How about this:

   The sampling instance is chosen as the point of reference for the
   RTP timestamp because it is known to the transmitting endpoint and
   because it has a common definition for all media, independent of
   encoding delays or other processing.

> because they have random offsets and advance at different rates.

Well, what I meant in the first sentence was:

   The purpose of choosing the sampling instance as the point of
   reference is to allow synchronized presentation of all media
   sampled at the same time.

I don't think this conflicts with the second sentence, though perhaps
there should be a paragraph break between them.

> in the infrequent RTCP packets.

Both overhead and accuracy.  Some people were concerned about
off-by-one errors if we used a clock that did not tick at the natural
rate of the media.  There was a long debate about this.

> other unit of each medium) in the stored data should be presented.

What you wrote is true.  But what I was after was the notion that the
sender uses the virtual clock to determine when to take the next unit
of data out of the file and send it.  But I was afraid that if I used
that phrasing, people would say the RTP timestamp should be the
transmit time.  It's not, it is the presentation timestamp from the
file.  The sender is running through the file as if the data would be
decoded and presented locally according to the virtual clock, but
instead the data is transmitted.  Depending on the encoding, the
timestamp in the file to which the sender compares the virtual clock
may be a presentation timestamp or a decoding timestamp.

                                                        -- Steve

---

From: Dave Singer <[email-masked]>
Date: Tue, 14 Jan 2003 16:56:49 -0800

>determines when to present it. The sender indicates when it should 
>be presented. So, I would propose:
>
>Applications transmitting stored data rather than data sampled in 
>real time typically use a virtual clock to indicate when the next 
>frame (or other unit of each medium) in the stored data should be 
>presented.

This has always been my problem about how this is presented;  it is 
an unusual protocol description which describes what happened at the 
source.  Usually protocols describe characteristics of the stream or 
what the receiver ought to do.  In particular, there is no 'sampling 
instant' for streams sent from disk under rtsp control, for example. 
However, there is a reference common clock which can be used to 
relate the RTP timestamps in the various streams.
-- 
David Singer
Apple Computer/QuickTime
_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Stephen Casner <[email-masked]>
Date: Tue, 14 Jan 2003 17:24:17 -0800 (PST)

Dave,

> relate the RTP timestamps in the various streams.

I don't think it is a description of what happened at the source.  We
are choosing what particular reference point in the life of the data
represented by the RTP timestamp in the packet corresponds to the
associated reference clock timestamp.  Choosing a common reference
point in the life of the data allows data from separate sources to be
related.  Specifying some amount of time to be added to the sampling
instant in order to hypothesize a presentation instant seems
artificial to me, and adds an extra complication.

I've proposed some additional text to try to explain the motivation
for the choice of reference point, and to explain how "sampling
instant" should be interpreted for the case of streams sent from
disk.  Do you think the words I added made the situation better or
worse?  Do you have other words to suggest?

I already replied to the message you were replying to; I explained why
I used the phrase "determine when the next frame or other unit of each
medium in the stored data should be presented".  How would you
describe the process that the sending is doing when it uses a
reference clock to determine when it is time for the next piece of
data to be transmitted?

                                                        -- Steve

---

From: Jonathan Rosenberg <[email-masked]>
Date: Thu, 16 Jan 2003 09:48:58 -0500

inline.

>    encoding delays or other processing.

OK.

> there should be a paragraph break between them.

Might help.

> of data out of the file and send it. 

That is not how I interpreted it.

> But I was afraid that if I used
> file. 

Wouldn't it be better to just say that, then? That is, say that its used 
as the timestamp to guide when its sent, but its not the transmit time, 
its the presentation time that the receiver will use.

-Jonathan R.