Subject: Re: [AVT] RTP profile for pro apps [was: draft-ietf-avt-rtp-ac3-01.txt]

From: Colin Perkins <[email-masked]>
Date: Sun, 6 Feb 2005 10:55:04 +0000

Chuck,

Indeed. I would suggest that the first step is for those interested in 
this subject to write an internet-draft summarising the issues, so the 
group can get a common understanding of what problem is to be solved 
and why RTP does not solve it currently. If there is an issue to be 
addressed, we should try to address it in a way that works across 
payload formats, rather than in an ad-hoc solution for a particular 
format.

Colin

On 22 Jan 2005, at 17:02, Chuck Harrison wrote:
> I agree with John about taking that "serious look" and considering
> an RTP profile for the pro world.
>
> I would like to revisit some of the topics I raised 4 years ago
> with draft-harrison-avt-precision-av-00.txt . I think this needs
> to go farther than just tacking SMPTE timecode (with all its
> notorious "features") into a header field.
>
> Cheers,
>   Chuck Harrison
>   Far Field Associates, LLC
>
>
> lazzaro wrote:
>>
>> On Jan 20, 2005, at 10:21 AM, Link, Brian wrote:
>>> Colin,
>>>
>>> I'm open to this idea, and I note the support from Dave and John. I'm
>>> not familiar enough with the infrastructure of RTP to know the 
>>> possible
>>> mechanisms for associating data with any payload format. How would 
>>> this
>>> be done? And from a procedural perspective, what is needed?
>>
>> I think we should consider taking a serious look at the range of
>> RTP pro uses for SMPTE time code, to see if there is real value in
>> having time code inline with the media in RTP (as opposed to
>> sent over a secondary channel, like the signalling solution
>> Colin proposes).  If there is, then I think that real value will
>> exist for a range of payload formats, not just AC3, and so we
>> should think hard about an RTP profile for pro uses.  If in
>> fact the signalling solution is the right one, then a general purpose
>> SMPTE time code signalling mechanism is something we
>> should think hard about.  But duplicating time code functionality
>> in each payload format that might be of interest to pros
>> seems to be sub-optimal to me ... part of the benefit of
>> RTP/SDP is that the header/parameters includes common features,
>> so that each payload format doesn't have to reinvent the wheel.
>>
>>>
>>> Regards,
>>> Brian
>>>
>>>
>>> ---
>>> Brian Link
>>> Dolby Laboratories, 100 Potrero Ave., San Francisco, CA 94103
>>> phone: 415-558-0324  fax: 415-863-1373
>>> email: [email-masked]   alt email: [email-masked]
>>>
>>>
>>> -----Original Message-----
>>> From: Colin Perkins [mailto:[email-masked]]
>>> Sent: Thursday, January 20, 2005 6:56 AM
>>> To: Link, Brian
>>> Cc: [email-masked]
>>> Subject: Re: [AVT] I-D ACTION:draft-ietf-avt-rtp-ac3-01.txt
>>>
>>> Brian,
>>>
>>> This use of SMPTE time codes would seem more general than for the AC3
>>> format. If this feature is going to be useful, would it make sense to
>>> take it out of the AC3 format, and define a general mechanism that
>>> could
>>> be used by any RTP payload format?
>>>
>>> Colin
>>>
>>>
>>>
>>> On 19 Jan 2005, at 15:31, Link, Brian wrote:
>>>> Hi Colin,
>>>>
>>>> Thanks for these comments and the editorial ones, too. I'll
>>>> incorporate the 'ptime' and 'SDP offer/answer' suggestions you have
>>> made.
>>>>
>>>> I'd like to offer some motivation for including SMPTE time code. The
>>>> SMPTE time code feature is not needed by DLNA for their consumer
>>>> application of A/V streaming over a home network. However, while I 
>>>> was
>>>
>>>> updating the I-D, I learned that AC-3 over RTP is also used in a
>>>> professional application and I'd like the payload format to
>>>> accommodate that, too.
>>>>
>>>> The particular professional application is quality control for the
>>>> mastering of DVDs. The audio and video are stored on separate 
>>>> storage
>>>> media, each "striped" with SMPTE time code so that in the final
>>>> mastering step they can be correctly multiplexed together into an 
>>>> MPEG
>>>
>>>> stream. Prior to the multiplexing step, there is a quality control
>>>> step where the audio and video are auditioned together to make sure
>>>> the multiplex will be performed correctly. The purpose of the 
>>>> audition
>>>
>>>> is to check the combination of the audio and video data together 
>>>> with
>>>> the time code as stored on disk. If, instead, the time code is
>>>> discarded by the server and regenerated at the client from the RTP
>>>> timestamp, the value of this test is negated. Any errors in the 
>>>> SMPTE
>>>> time on the storage medium are replaced by the regenerated time 
>>>> code.
>>>>
>>>> It's also interesting to note that the SMPTE time code is not used 
>>>> to
>>>> synchronize two RTP streams. In this case the audio, in an RTP 
>>>> stream,
>>>
>>>> is synchronized with an external video device through a hardware
>>>> interface. RTP is being used as a "drop in" replacement for an
>>>> existing audio transport in a heterogeneous facility. The video 
>>>> stream
>>>
>>>> is not
>>>> (necessarily) carried over RTP. Also, the SMPTE time code for the
>>>> audio is not generated as the RTP stream is created, instead it was
>>>> created earlier and in an external part of the overall system.
>>>>
>>>> For most end-consumer-type applications, SMPTE time code is 
>>>> redundant,
>>>
>>>> so it is optional in the payload header. For professional
>>>> applications, it is not necessarily redundant (and that market often
>>>> demands redundancy, anyway) and is already used by products in the
>>>> field, so I feel SMPTE time code is needed, as an option.
>>>>
>>>> Regards,
>>>> Brian
>>>>
>>>> ---
>>>> Brian Link
>>>> Dolby Laboratories, 100 Potrero Ave., San Francisco, CA 94103
>>>> phone: 415-558-0324  fax: 415-863-1373
>>>> email: [email-masked]   alt email: [email-masked]
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Colin Perkins [mailto:[email-masked]]
>>>> Sent: Saturday, January 15, 2005 6:57 AM
>>>> To: Link, Brian
>>>> Cc: [email-masked]
>>>> Subject: Re: [AVT] I-D ACTION:draft-ietf-avt-rtp-ac3-01.txt
>>>>
>>>> Brian,
>>>>
>>>> On 3 Jan 2005, at 18:39, Link, Brian wrote:
>>>>> As the quoted announcement states, I have revived the RTP payload
>>>>> format I-D for AC-3. I have made a number of changes to take into
>>>>> account comments that had previously been offered, to simplify the
>>>>> draft, and to add some time code functionality that is used in some
>>>>> products.
>>>>>
>>>>> An industry consortium called the Digital Living Network Alliance
>>>>> (DLNA)
>>>>> plans to stream AC-3 over RTP in some of its home networking
>>>>> applications. In order for DLNA to reference the payload format in
>>>>> this document (instead of publishing their own), it will need to be
>>>>> published in a relatively short time.  I will very much appreciate 
>>>>> it
>>>
>>>>> if interested people could review and comment on this I-D to move 
>>>>> it
>>>>> along quickly.
>>>>
>>>> Mostly this looks good, although I have a couple of technical
>>> concerns:
>>>>
>>>>   - Including a SMPTE time code in each packet is unusual, and
>>>> redundant
>>>>     with the RTP timestamp (which is mapped to an NTP-format 
>>>> timestamp
>>>
>>>> via
>>>>     RTCP SR packets) in many cases. Is it possible to use the RTP
>>>> timestamp
>>>>     instead, possibly with a signalled mapping from SMPTE time code 
>>>> to
>>>
>>>> an
>>>>     NTP format timestamp in the SDP during session setup?
>>>>
>>>>   - The second is that the definition of "ptime" is unusual. I think
>>> it
>>>>     would be better as:
>>>>
>>>>      ptime: The duration of time in milliseconds represented by each
>>>> AC-3
>>>>      frame. This corresponds to the duration of each RTP packet in
>>> cases
>>>>      where fragmentation is not used.
>>>>
>>>>     without trying to define what it means for fragments.
>>>>
>>>>   - Finally, the draft should define how SDP is used in offer/answer
>>>> mode to
>>>>     negotiate parameters. It should also describe how the MIME
>>>> parameters are
>>>>     copied into the SDP (section 5.1 of RFC 3557 is a good example).
>>>>
>>>> I've also sent a few minor editorial comments privately.
>>>>
>>>> Colin
>>>>
>>>>
>>>> -----------------------------------------
>>>> This message (including any attachments) may contain confidential
>>>> information intended for a specific individual and purpose.  If you
>>>> are not the intended recipient, delete this message.  If you are not
>>>> the intended recipient, disclosing, copying, distributing, or taking
>>>> any action based on this message is strictly prohibited.
>>>>
>>>>
>>>> _______________________________________________
>>>> Audio/Video Transport Working Group
>>>> [email-masked]
>>>> https://www1.ietf.org/mailman/listinfo/avt
>>>>
>>>
>>>
>>> -----------------------------------------
>>> This message (including any attachments) may contain confidential
>>> information intended for a specific individual and purpose.  If you
>>> are not
>>> the intended recipient, delete this message.  If you are not the
>>> intended
>>> recipient, disclosing, copying, distributing, or taking any action
>>> based on
>>> this message is strictly prohibited.
>>>
>>>
>>>
>> ---
>> John Lazzaro
>> http://www.cs.berkeley.edu/~lazzaro
>> lazzaro [at] cs [dot] berkeley [dot] edu
>> ---
>>
>> _______________________________________________
>> Audio/Video Transport Working Group
>> [email-masked]
>> https://www1.ietf.org/mailman/listinfo/avt
>
> _______________________________________________
> Audio/Video Transport Working Group
> [email-masked]
> https://www1.ietf.org/mailman/listinfo/avt
>

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Colin Perkins <[email-masked]>
Date: Sun, 6 Feb 2005 10:58:04 +0000

Brian,

On 20 Jan 2005, at 18:21, Link, Brian wrote:
> not familiar enough with the infrastructure of RTP to know the possible
> mechanisms for associating data with any payload format. How would this
> be done?

One possibility might be to standardise a new RTP profile, with 
additional header fields (e.g. as is being done for the RTP profile for 
TFRC). Another might be a standard header extension, or new RTCP packet 
types.

> And from a procedural perspective, what is needed?

Initially, a draft explaining the problem, and some analysis of why is 
can't be solved using RTP + existing signalling mechanisms.

Colin

---

From: lazzaro <[email-masked]>
Date: Sun, 06 Feb 2005 10:35:01 -0800

On Feb 6, 2005, at 2:58 AM, Colin Perkins wrote:
> additional header fields (e.g. as is being done for the RTP profile 
> for TFRC). Another might be a standard header extension, or new RTCP 
> packet types.

Reaching out to the folks involved in AES standardization efforts would
be a good thing to do early in the process:

http://www.aes.org/technical/nas/

And also, to reach out to other standards bodies who are stake holders.

Part of the reason why RTP MIDI has taken so long to bring to closure
(the research effort began in Fall 2000) was the number of times the 
protocol
needed to be reworked to take into account the views of new parties 
that joined
the effort mid-stream.  In large part, because I didn't engage and 
evangelize
the global MIDI constituency up front, but instead waited for them to 
discover
the project themselves ...

---
John Lazzaro

---

From: Dave Singer <[email-masked]>
Date: Mon, 7 Feb 2005 09:53:48 -0800

At 10:58 AM +0000 2/6/05, Colin Perkins wrote:
>Initially, a draft explaining the problem, and some analysis of why 
>is can't be solved using RTP + existing signalling mechanisms.
>
>Colin

Given that at least SMPTE time-codes tend to be calculable in 
segments, initial values in SDP and updates in RTCP was the way I was 
thinking.  I'll read 
<http://quimby.gnus.org/internet-drafts/draft-harrison-avt-precision-av-00.txt> 
also to see what else is on the table.
-- 
David Singer
Apple Computer/QuickTime

---

From: Dave Singer <[email-masked]>
Date: Thu, 10 Feb 2005 16:24:43 -0800

Here are some thoughts on carrying SMPTE time-code information in RTP.

First a brief background on SMPTE time-codes.

SMPTE time-codes count frames.  There are two common forms of 
display:  either a simple counter, or what looks like a normal clock 
value (hh:mm:ss.frame).  When the frame rate is truly integer, then 
this can be a normal clock value, in that seconds tick by at the same 
rate as the seconds we know and love.

However, NTSC video infamously runs slightly slower than 30 
frames/second.  Some people call it 29.97 (which isn't quite right) 
and some say that a frame takes 1001 ticks of a 30000 tick/second 
clock (which is closer).  Be that as it may, SMPTE time codes count 
30 of these frames and deem that to make a second.

This causes a SMPTE time-code display to 'run slow' compared to 
real-time.  To ameliorate this, a format called drop-frame is used. 
Some of the frame numbers are skipped, so that the counter 
periodically 'catches up' (so some time-code-seconds actually only 
have 28 or 29 frames in them).

So, what we desire is a system that allows us to associate a SMPTE 
time-code with some media.  Since in RTP all media has a clock 
already, we can leverage that fact.  If we treat the media as having 
'runs' of time in which the time-code is simply counting up, then the 
time-code anywhere within a run can be calculated if you know
1. the RTP timestamp of the start of the run
2. the RTP timestamp where you want to know the time-code
3. the time-code of the start of the run
4. the counting rate and other parameters of the time-code.

My proposal is that we put periodic mappings between (1) and (3) into 
RTCP packets, and provide (4) the 'setup' information out-of-band, 
for example in SDP.

The setup information includes:
   (the timescale of the RTP stream, already provided)
   the duration, in that timescale, of a single frame-count in the 
'frames' portion of the time-code
   the number of those frames that make a time-code-second
   the following booleans:
     is-NTSC-drop-frame:  should the usual 'left out numbers' of 
drop-frame be applied or not?
     wrap-at-24-hours:  should the hours portion wrap from 23 to 0, or 
keep counting up?
     allow-negative-time-codes:  are negative time-codes used in this stream?
     display-time-code-as-counter:  should the display be an integer 
frame-count, or hh:mm:ss.fr format?
     time-code-displayed:  is it intended that this time-code be 
displayed somehow?

For example, if associated with a video track with the common 
time-scale of 90000, then frame-duration of 3003 and 
frames-per-tc-second of 30 would yield a 'normal' SMPTE time-code for 
NTSC video.  Similarly values of 3750 and 24 yield a time-code for 24 
fps film content, and so on.

Now, we put into an RTCP APP packet (or a new RTCP packet), a mapping 
between an RTP time-stamp value and the time-code.  The RTP timestamp 
and the that time-code are 32 bits;  the time-code is either a signed 
counter value (if we're in counter format), or it is the format
   hours(8) -- 0 to 255
   sign(1) -- 1 for negative, 0 for positive
   minutes(7) -- 0 to 59
   seconds(8) -- 0 to 59
   frames(8) -- 0 to (frames-per-tc-second - 1)

This establishes the time-code for all RTP times greater than or 
equal to the one given, until a subsequent APP packet reestablishes 
the mapping.  It's unfortunate that the sign is in the middle, but 
that allows the hours to use the full range, and the minutes don't 
need to.

* * *

It might be argued that we could set the initial mapping also in the 
SDP, since RTCP packets might get lost.  But this means that the SDP 
now has to have knowledge of the RTP random offset (nasty);  and if 
one puts this APP packet into all sender reports, it's probably good 
enough.  Then if you don't have time-codes, you don't have 
audio-video-sync either.

This associates the time-code with a particular media stream.  An 
alternative would be to make it an RTP stream in its own right;  but 
the data rate is so low, this seems egregious.  And by packing it 
inline, we can do this backwards-compatible for gateways etc. that 
already handle dual-stream.

Comments?

-- 
David Singer
Apple Computer/QuickTime

---

From: lazzaro <[email-masked]>
Date: Thu, 10 Feb 2005 17:24:24 -0800

> Here are some thoughts on carrying SMPTE time-code information in RTP.

Thanks for sending this along ...

> This establishes the time-code for all RTP times greater than or equal 
> to the one given, until a subsequent APP packet reestablishes the 
> mapping.

Let's take the simplest example of non-monotonic timecode --
a TV show segment followed by 4 30-second commercials
followed by another TV show segment.  Each new footage
resets the timecode to 00:00:00:00.  From what I understand,
this sort of resetting is common in some video uses, but I'm not
an expert ...

Is your proposal meant to address situations like this? It would
seem possible, if you added some sending semantics to the

> 3. the time-code of the start of the run

to cover how senders should signal upcoming transitions
(say, 15 seconds before the transition start sending the new
RTP/SMPTE start pairing) and how receivers should interpret
APP packets that describe future RTP values (keep using the
old pairing until the new RTP start time arrives in the stream).

There's probably a limit to how short footage can be,
given the 5 second RTCP minimum sending time.  And of
course, this won't work well for interactive applications, where
a new run is starting because a human hits a rewind button,
and so pre-sending new run start times far in advance won't work.
The danger with sending RTCP "late" is that receivers may
associate the wrong SMPTE/RTP pair if the receiver is not aware
that a new run has begun.

> SDP, since RTCP packets might get lost.

I agree, avoiding SDP for initial mappings is a good idea ...

> An alternative would be to make it an RTP stream in its own right;  
> but the data rate is so low, this seems egregious.  And by packing it 

If we go this route, there is a lot to be said for sending an RTP MIDI 
stream,
and the using MIDI Time Code or MIDI Machine Code tools to give 
full-function
SMPTE support.  No extra work to do on AVTs part ...

> 'frames' portion of the time-code

Some apps may want timecode to tick backwards ... MTC supports this
by reversing the sending direction of MIDI Quarter Frame sequences.
One way to do this would be to provide negative durations.  However,
since direction may change for every new start of run, it may be better
to code it in RTCP ...

>  frames(8) -- 0 to (frames-per-tc-second - 1)

There may be a standardized SMPTE packing into 32-bits ...
MIDI has a specialized 32-bit packing, the 8 4-bit nibbles
of an 8-command MIDI Quarter Frame sequence, but there
may be other standardized packings too ...

---
John Lazzaro

---

From: Dave Singer <[email-masked]>
Date: Fri, 11 Feb 2005 09:25:00 -0800

At 5:24 PM -0800 2/10/05, lazzaro wrote:
>>the mapping.
>Is your proposal meant to address situations like this?

Yes, one would include new APP packets marking the RTP timestamp of 
the discontinuity.

>  It would

The latter I agree with.  The former is only possible if you have 
look-ahead (i.e. stored content, or source-buffered live).  If you 
have client-buffered live content, the source has to notice the 
discontinuity and get that RTCP APP packet out PDQ, so it arrives at 
the client before the RTP packet that crosses the bouncary is 
interpreted.

>given the 5 second RTCP minimum sending time.

I think we can allow multiple of these in one RTCP packet, but I hope 
that need is rare!

>And of

Yes, I don't see how to avoid this.

>since direction may change for every new start of run, it may be better

OK, we could allow negative durations.  I think the math still works...

>may be other standardized packings too ...

If anyone knows, I am all ears.  I lifted this from our current practice...
-- 
David Singer
Apple Computer/QuickTime

---

From: lazzaro <[email-masked]>
Date: Fri, 11 Feb 2005 10:06:57 -0800

> OK, we could allow negative durations.  I think the math still works...

There may be an "off-by-one" issue lurking somewhere when you let the 
tape
reverse directions ... I remember having to deal with this issue when 
doing
the recovery journal semantics for MTC.

---
John Lazzaro