Subject: Re: Interleaving MP3 for RTP

From: Dave Singer <[email-masked]>
Date: Fri, 19 May 2000 15:17:48 -0700

>http://www.live.com/rtp-mp3/rtp-mp3.txt

Ross

Overall, I like this draft, but on thinking it over, I think there 
are a few comments to make.

The draft attempts to retain some compatibility with the MPEG 
packings, but since it is a different packing, I'm not sure what this 
compatibility really gains us.  The most obvious point here is the 
4-byte header, which really only serves a useful function if an MP3 
frame is split across multiple RTP packets.  Given that MP3 frames 
are usually small, and we'd only split if a frame won't fit into a 
packet by itself, I expect such splitting to be extremely rare.  I'd 
rather see us use a bit in the header of a frame to say that this 
'frame' is actually a continuation of a previous fragment with the 
same sequence number.

In the header byte, you do need to know the number of this frame 
within its group, but I don't think you need a group number.  You can 
tell exactly where this MP3 belongs in time without it, so there is 
no danger of de-interleaving two groups together.  Basically, the RTP 
timestamp R is (as always) the timestamp of the oldest (earliest) MP3 
frame within the packet.  Say that this has group index M, and the 
frame you want to time has group index N.  Since the MP3 frames have 
constant duration D, the RTP time of this frame is (N-M)*D + R.

You don't discuss timescale and timestamps, and so I think you expect 
the scale to be 90kHz, like general MPEG.  I think it would be better 
if it were the sampling rate of the audio, just like other audio 
packing formats.

It might (?) help de-interleavers, and provide a consistency check, 
if there was a bit used in the header to indicate that more MP3 
frames occur in this packet (or not).

I also think that it can really help the de-packetizer if it knows 
the interleave group size.  I think that this can be indicated in 
very few bits, like 2.  Say these have value I.  Since any group size 
does for the case where we are not going to interleave (we can always 
pack sequential MP3 frames into packets), we don't need a group size 
value of 0 or 1.  If this has value X, I'd say it would indicate to 
the receiver that the maximum de-interleave buffer it needs is of 
size 2**(X+2), i.e. 4,8,16,32.

Comments?
-- 
David Singer
Apple Computer/QuickTime

---

From: Ross Finlayson <[email-masked]>
Date: Fri, 19 May 2000 21:47:24 -0700

At 03:17 PM 5/19/00, Dave Singer wrote:
>The draft attempts to retain some compatibility with the MPEG packings, 
>but since it is a different packing, I'm not sure what this compatibility 
>really gains us.

Dave (cc. "rem-conf" et al),

Here's the motivation for basing the new payload format on RFC2250:  RFC 
2250 is the existing standards-track format for streaming MPEG 1 or 2 audio 
using RTP, so any application that wishes to receive RTP-streamed MPEG 
audio (including MP3 - i.e., MPEG 1 or 2 layer III) already *has to* 
implement the RFC 2250 format.  However, because RFC 2250 doesn't handle 
MP3 well, we'd like implementors to also support the use of the new, 
alternative payload format for streaming MP3.  This seems like a much 
easier sell if the new payload format is based upon the existing payload 
format (RFC 2250) that implementors are already familiar with - it makes 
the new format easier to understand and implement.  (In particular, because 
the new payload format uses the same packetization/depacketization scheme 
as RFC 2250, much of an existing RFC 2250 implementation can be used.)

I.e., basing the new payload format on RFC 2250 seems like "the path of 
least resistance" to having it become widely implemented and used.

>   The most obvious point here is the 4-byte header, which really only 
> serves a useful function if an MP3 frame is split across multiple RTP packets
>Given that MP3 frames are usually small, and we'd only split if a frame 
>won't fit into a packet by itself, I expect such splitting to be extremely 
>rare.

Agreed - this is probably the worst aspect of RFC 2250: The presence of a 
4-byte "MPEG audio-specific header" (for fragmentation) that's usually just 
wasted space.  Fortunately, however, this wastage is typically quite small: 
Just 0.4% in a 1000-byte RTP packet, and less than 0.3% in a 1500-byte RTP 
packet.  If you're in an environment where this level of wastage is 
considered important, then you're probably better off using a more 
efficient codec than MP3 anyway (e.g., AAC or MPEG-4 GA).

>   I'd rather see us use a bit in the header of a frame to say that this 
> 'frame' is actually a continuation of a previous fragment with the same 
> sequence number.

Note that any 'continuation' frame has to include a byte offset, because 
otherwise you wouldn't know how many bytes were in any preceding RTP 
packet(s) that got lost.  (Note also that in RTP, sequence numbers 
increase, even for packets that make up the same 'frame'.)

>You don't discuss timescale and timestamps

Actually, I mention timestamps (briefly) in section 6.  Because it's an 
important point that's relevant both to your comments and to Chris Sloan's 
(which I'll respond to in my next message), I'll repeat it here:

         "Step 3 is the packetizing of a sequence of (interleaved) ADU frames
         into RTP packets - exactly as specified in RFC 2250.  When
         computing the RTP timestamps for each packet, this step ignores
         any reordering that took place in step 2, and treats the sequence
         of packetized ADU frames as if they were generated sequentially.
         I.e., the RTP timestamps on outgoing packets are always
         monotonically increasing, regardless of interleaving."

This makes an implementation much simpler.  It allows the 
packetizing/depacketizing code (which can also be shared with an existing 
implementation of RFC 2250) to be independent of the reordering/tagging code.

> the scale to be 90kHz, like general MPEG.  I think it would be better if 
> it were the sampling rate of the audio, just like other audio packing formats.

Note, however, that in either RFC 2250 or the new format it's possible 
(although strange) for a RTP packet to include consecutive frames that were 
encoded using different sampling frequencies.  In any case (in either 
format), the actual sampling frequency is always known from the (MP3 or 
ADU) 'frame'; the RTP timestamp is not needed for this purpose.

>In the header byte, you do need to know the number of this frame within 
>its group, but I don't think you need a group number.  You can tell 
>exactly where this MP3 belongs in time without it, so there is no danger 
>of de-interleaving two groups together.

The group number (which I refer to as an "Interleave Cycle Count") is for 
the benefit of receivers.  It allows a receiver to easily distinguish 
between frame(s) that are missing merely because they haven't yet appeared 
in the reordering, and frames that are missing because they were in lost 
packets.  E.g., suppose a receiver sees a sequence of frames with 
"Interleave Index" 1, 3, 4 (i.e., with 2 missing).  If it then starts 
seeing frames with a different "Interleave Cycle Count", it then knows for 
sure that frame 2 has been lost, and so can immediately release frames 
1,3,4 to the decoder (or ADU->MP3 converter), rather than having to wait 
around to see if frame 2 ends up appearing later in the order.

I think this also addresses your later question about receivers knowing the 
interleave group size.  In this scheme the receiver knows the maximum 
possible group size (this is a fixed number, given by the number of bits in 
the "Interleave Index"), but doesn't know *in advance* the size of the 
current cycle.  It does, however, know when the currently cycle ends , by 
the change in the "Interleave Cycle Count".

         Ross.

---

From: Dave Singer <[email-masked]>
Date: Mon, 22 May 2000 13:48:53 -0700

At 21:47 -0700 5/19/00, Ross Finlayson wrote:
>>packings, but since it is a different packing, I'm not sure what 
>>this compatibility really gains us.
>
>Here's the motivation for basing the new payload format on RFC2250: 
>RFC 2250 is the existing standards-track format for streaming MPEG 1 
>or 2 audio using RTP, so any application that wishes to receive 
>RTP-streamed MPEG audio (including MP3 - i.e., MPEG 1 or 2 layer 
>III) already *has to* implement the RFC 2250 format.  However, 
>because RFC 2250 doesn't handle MP3 well, we'd like implementors to 
>also support the use of the new, alternative payload format for 
>streaming MP3.  This seems like a much easier sell if the new 
>payload format is based upon the existing payload format (RFC 2250) 
>that implementors are already familiar with - it makes the new 
>format easier to understand and implement.  (In particular, because 
>the new payload format uses the same packetization/depacketization 
>scheme as RFC 2250, much of an existing RFC 2250 implementation can 
>be used.)
>
>I.e., basing the new payload format on RFC 2250 seems like "the path 
>of least resistance" to having it become widely implemented and used.

I think at the point where I'm implementing a new payload format, its 
similarity to others is probably moot as long as it is simple.

>>   The most obvious point here is the 4-byte header, which really 
>>only serves a useful function if an MP3 frame is split across 
>>multiple RTP packets
>>Given that MP3 frames are usually small, and we'd only split if a 
>>frame won't fit into a packet by itself, I expect such splitting to 
>>be extremely rare.
>
>Agreed - this is probably the worst aspect of RFC 2250: The presence 
>of a 4-byte "MPEG audio-specific header" (for fragmentation) that's 
>usually just wasted space.  Fortunately, however, this wastage is 
>typically quite small: Just 0.4% in a 1000-byte RTP packet, and less 
>than 0.3% in a 1500-byte RTP packet.  If you're in an environment 
>where this level of wastage is considered important, then you're 
>probably better off using a more efficient codec than MP3 anyway 
>(e.g., AAC or MPEG-4 GA).

Well, perhaps.  Compressed audio doesn't tend to use very large (1K) 
packets, because of buffering delay considerations.  I am unconvinced 
that it is a good idea to waste space because it doesn't matter very 
much!  I'd prefer a 'tight' design at the outset.

>>   I'd rather see us use a bit in the header of a frame to say that 
>>this 'frame' is actually a continuation of a previous fragment with 
>>the same sequence number.
>
>Note that any 'continuation' frame has to include a byte offset, 
>because otherwise you wouldn't know how many bytes were in any 
>preceding RTP packet(s) that got lost.  (Note also that in RTP, 
>sequence numbers increase, even for packets that make up the same 
>'frame'.)

Not really.  You can't use the second fragment if you don't get the 
first, and then you'll know the length.  You'll know that a fragment 
is missing if either (a) you get a continuation piece without the 
start piece or (b) the packet sequence number of the continuation is 
not 1 greater than the sequence number of the previous piece.

>Actually, I mention timestamps (briefly) in section 6.  Because it's 
>an important point that's relevant both to your comments and to 
>Chris Sloan's (which I'll respond to in my next message), I'll 
>repeat it here:
>
>packetizing/depacketizing code (which can also be shared with an 
>existing implementation of RFC 2250) to be independent of the 
>reordering/tagging code.

OK, I think we are in agreement about different things.  I think that 
we should say explicitly that the RTP time-stamp is the time-stamp of 
the oldest (first?) data in the packet *and* that interleaving should 
be done in such a way time-stamps are non-decreasing (they would only 
be the same for a split frame).

>>  You don't discuss timescale and timestamps, and so I think you 
>>expect the scale to be 90kHz, like general MPEG.  I think it would 
>>be better if it were the sampling rate of the audio, just like 
>>other audio packing formats.
>
>Note, however, that in either RFC 2250 or the new format it's 
>possible (although strange) for a RTP packet to include consecutive 
>frames that were encoded using different sampling frequencies.  In 
>any case (in either format), the actual sampling frequency is always 
>known from the (MP3 or ADU) 'frame'; the RTP timestamp is not needed 
>for this purpose.

But I think that there is a lot of benefit in sample-accurate 
time-stamping, and actually saying that if you change sample-rates 
you change payload ID (with a new RTPMAP) is a really good thing.

>>within its group, but I don't think you need a group number.  You 
>>can tell exactly where this MP3 belongs in time without it, so 
>>there is no danger of de-interleaving two groups together.
>
>The group number (which I refer to as an "Interleave Cycle Count") 
>is for the benefit of receivers.  It allows a receiver to easily 
>distinguish between frame(s) that are missing merely because they 
>haven't yet appeared in the reordering, and frames that are missing 
>because they were in lost packets.  E.g., suppose a receiver sees a 
>sequence of frames with "Interleave Index" 1, 3, 4 (i.e., with 2 
>missing).  If it then starts seeing frames with a different 
>"Interleave Cycle Count", it then knows for sure that frame 2 has 
>been lost, and so can immediately release frames 1,3,4 to the 
>decoder (or ADU->MP3 converter), rather than having to wait around 
>to see if frame 2 ends up appearing later in the order.

But you can as easily work that out from the time-stamp, as I said; 
the time-stamp will tell you that the earliest frame in this packet 
belongs in a different group from the group in hand.

>I think this also addresses your later question about receivers 
>knowing the interleave group size.  In this scheme the receiver 
>knows the maximum possible group size (this is a fixed number, given 
>by the number of bits in the "Interleave Index"), but doesn't know 
>*in advance* the size of the current cycle.  It does, however, know 
>when the currently cycle ends , by the change in the "Interleave 
>Cycle Count".
>
>         Ross.

Sure, we know a max.  This may not be worth worrying about if the max 
is as small as 32.
-- 
David Singer
Apple Computer/QuickTime