Subject: RE: MPEG-4 (including Flexmux), over RTP/RTSP

From: "Paul Christ" <[email-masked]>
Date: Tue, 13 Jun 2000 19:52:15 +0200

Dear Dave, dear all,

please find attached our first comment to your/Dave's mail - now evolved
into
ISO/IEC JTC1/SC29/WG11 N3381.

We welcome such a paper and will try to help further to improve it.
If you don't mind: We should avoid  "memoryless" discussions :-)
AS agreed in the IETF

   C.Guillemot et al.,
   "RTP Payload Format for MPEG-4 with Flexible Error Resiliency",
   IETF Draft, draft-ietf-avt-mpeg4streams-00,
   March 1 2000, expires Sept 1 2000

   R. Civanlar et al.,
   " RTP Payload Format for MPEG-4 Streams",
   IETF Draft, draft-ietf-avt-rtp-mpeg4-02, ?
   ? 2000, expires ?? 20000

are to become experimental RFCs - for technical reasons.
which state should not simply be overwritten
by declarative statements.

All the best

P. Christ

p.s.: Discussed in the context of Project SoNG
      with Christine Guillemot

> -----Original Message-----
> From: [email-masked]
> [mailto:[email-masked]]On Behalf Of Dave
> Singer
> Sent: Freitag, 19. Mai 2000 20:13
> To: [email-masked]'
> Cc: 'Colin Perkins'; CURET Dominique FTRD/DMI/REN; [email-masked];
> [email-masked]
> Subject: MPEG-4 (including Flexmux), over RTP/RTSP
>
>
> This is a portmanteau message, trying to cover the whole space of
> carriage of MPEG-4 in IETF protocols.  I'm trying to pull everything
> together.
>
> It started by the resurgence of comment on FlexMux in RTP, so I'll
> start with packing formats.
>
>
> PACKING FORMATS
>
> I have been thinking about packing formats for MPEG-4 data, and I
> feel that we need to be a little more flexible.  I'd suggest that at
> least the following have value:
>
> a) a simple packing of a single, arbitrary, elementary stream into
> RTP packets.  The packing would not be stream-format-aware, and so
> could carry any one ES.  [MPEG4-GENERIC]
>
> b) a packing of MPEG-4 video, with care to place RTP packet
> boundaries at sub-frame boundaries, such that partial frame display
> is possible at the reciever.  I suspect that the video group's "video
> packet" meets the need, and may not require any RTP-specific header.
> However, it should be a distinct payload name so that receivers can
> know that video-aware packetization has happened.  [MPEG4-VIDEO]
>
> c) a packing of MPEG-4 audio which interleaves the audio frames.  See
> the existing specifications for QCELP and the draft from Ross
> Finlayson for MP3 for examples.  Many audio decoders can do some
> sensible fill-in for single missing frames, but this fill-in error
> concealment rapdily gets worse if multiple consecitive frames are
> lost.  By intrerleaving, the decoder gets a sporting chance to
> conceal errors.  A small RTP header is needed to indicate that this
> has happened.  (With a suitable choice of header, it's possible that
> packing (a) could be a special case of this.)  [MPEG4-AUDIO]
>
> d) a packing of flexmux into RTP.  This enables the carriage of large
> numbers of streams, where the use of separate RTP streams becomes
> prohibitive.  [MPEG4-FLEXMUX]
>
> It may be that some formats become common enough, and have such
> characteristics, that a unique payload format for it becomes
> warranted (e.g. AAC).  That should be possible down the road.
>
> For those streams requiring reliable delivery, I'd like to see us
> leverage existing work in the IETF in this area (including, but not
> limited to FEC, re-transmission, or repetition), rather than making
> it a characteristic of the packing scheme itself.
>
> I'd like to think that all these, though separate payload schemes
> with separate names (in the RTPMAP) can be otherwise treated
> uniformly, in terms of timescale and timestamp, ES-ID to RTP mapping,
> and so on.
>
>
> TIME-STAMP HANDLING
>
> I think we are all agreed on this: that the MPEG-4 timescale for an
> ES becomes the RTP timescale, that the composition time maps to the
> RTP time, and the decode time maps to transmission order.
>
>
> SDP INFORMATION
>
> I think we can assume, for the moment, that any session described by
> SDP (e.g. in SAP, as a file download, as a DESCRIBE over RTSP) has at
> most one MPEG-4 session, though I'd like to see how we could lift
> this restriction.
>
> We need
> i) RTP stream to elementary stream mapping.  This needs to cover the
> Flexmux case as well as the single stream.
> a=x-mpeg4-esid a.b.c
> or
> a=x-mpeg4-esids m.i:a.b.c,n.k:p.q
>
> The first attribute is used for single streams; the second for
> flexmux.  In this, a is the ESID of the top-level OD stream, b is the
> ESID within that scope of another OD stream, and c is the ESID within
> that scope of the stream;  similarly for p and q.  m and n are
> muxcodes (identifying the table used), and i and k are flexmux
> channel numbers within the indicated muxcode.
>
> The entire muxcode table(s) also need to be transferred.  For the
> moment, I propose another attribute for them.  Since these tables
> have a length indicator, multiple tables can be concatenated
> together, if required.
> a=x-mpeg4-muxcodetable <location>
>
> where <location> is a URL that will supply the table(s).  If they are
> small, a DATA: URL will probably suffice to carry them in-line. If
> not, the URL should use a file-retrieval scheme (e.g. HTTP, FTP).
> The data at the indicated URL consists of some number of muxcode
> tables, complete, in binary format (but note that DATA URLs allow for
> base64 encoding of binary data, which would be needed here).  The
> mime type of a muxcode table needs deciding
> (application/x-mpeg4-muxcodetable?).
>
>
> ii)  Some indication that an MPEG-4 session is carried, and the means
> to get the IOD, if the terminal doesn't already have it by some other
> means.
> In SDP, as a session-level attribute:
>       a=x-mpeg4-iod [<location>]
>       In an RTSP session, the location is optional.  If not supplied,
> the IOD is retrieved over the RTSP session by using either DESCRIBE
> with an accept of type application/x-mpeg4-iod, or a GET-PARAMETER
> (we need to decide which). Where the SDP information is supplied by
> some other means (e.g. as a file, in SAP), the location is obligatory
> and should be a URL which will supply the IOD (e.g. small ones may be
> encoded using DATA:, or otherwise HTTP: or other suitable file-access
> URL).
> --
> David Singer
> Apple Computer/QuickTime
>

---

From: Dave Singer <[email-masked]>
Date: Tue, 13 Jun 2000 12:48:22 -0700

>ISO/IEC JTC1/SC29/WG11 N3381.

Thanks for your comments.  I appreciate them.  We really need to get 
this area nailed down...

>by declarative statements.

Right.  The purpose of my draft was mostly to focus on the areas on 
which there was probably agreement, but no actual specification, and 
to admit that we were likely to have a plurality of RTP packings 
going ahead.  I certainly have no desire to discard existing work in 
the area;  rather to build on it and flesh out all the other areas.

It seems, from reading your comments, that your disagreements are 
only in the area of packing formats.

1) You don't want to require support for the simple packing done by 
Carsten et al., whereas I feel that it would be good if there is one, 
simple, general packing that everyone implements and that can carry 
any MPEG-4 stream, perhaps in a non-optimal fashion.  I think it 
would be disappointing if there was no such spec., and Carsten's 
draft seems like the best contender.  Do you disagree with the 
principle (that there should be a simple interoperable draft), or the 
specification itself?

2) My recommendation that we look at existing techniques in the IETF 
for reliability -- FEC, re-transmission.  I didn't mention redundant 
audio, though as you say that is another existing technique which is 
orthogonal to the encoding schemes used.  I only phrased this as a 
recommendation, and I guess it could be weakened from there.  I 
certainly have no quibble in principle with combined source/channel 
coding, which is, I think, the essence of your draft.  My draft was 
explicitly written to get us out of the deadlock of trying to decide, 
now, a single packing scheme, and permit (maybe even encourage) 
research, development, interoperation, and promotion of schemes. 
It's clear from the multiplicity of drafts that we are not yet in 
consensus.

>p.s.: Discussed in the context of Project SoNG

My regards to all of you, and thanks for the good insights and work 
you are doing.
-- 
David Singer
Apple Computer/QuickTime

---

From: "Olivier Avaro" <[email-masked]>
Date: Wed, 14 Jun 2000 10:38:40 +0200

Hi all,

I would second Dave on the following aspect : let's get the framework first,
we have many matters of consensus there, let's have all the brains on this
reflector geting it right.

cu,

O.

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Wed, 14 Jun 2000 19:29:37 +0900

Dear Dave, Olivier, all,

I'm resending the attached mail which may be related to this discussion
because I haven't see this mail through the reflector.

I agree with the basic concept of Dave's proposal if it tries to propose a
framework covering all MPEG-4 RTP formats. Actually, I misunderstood that it
might be proposing like (c) in the attached mail. But, we need to be care
that we should never have a solution like (a). By this meaning, I think that
N3381 needs further revisions. The selection of RTP payload proposals and
MUST/SHOULD/MAY wording should be decided with much care.

If it is acceptable, I would like to make a contribution from one side of
the MPEG-4 RTP world.

Best regards,
Yoshihiro

----- Original Message -----
From: Olivier Avaro <[email-masked]>
To: Paul Christ <[email-masked]>; Dave Singer
<[email-masked]>
Cc: <[email-masked]>; 'Colin Perkins'
<[email-masked]>; CURET Dominique FTRD/DMI
<[email-masked]>; <[email-masked]>; <[email-masked]>;
Yan Xiang Chen <[email-masked]>; Stefan Wesner
<[email-masked]>
Sent: Wednesday, June 14, 2000 5:38 PM
Subject: Re: MPEG-4 (including Flexmux), over RTP/RTSP

> reflector geting it right.

Subject: Resolution 3.2.3 and N3381
From: "Yoshihiro Kikuchi" <[email-masked]>
To: "'Dave Singer'" <[email-masked]>
Cc: "Yoshihiro Kikuchi" <[email-masked]>, "Herpel Carsten" <[email-masked]>, "Colin Perkins" <[email-masked]>, <[email-masked]>
Date: Mon, 12 Jun 2000 10:45:26 +0900

Dear Dave and MPEG-4 on IP experts,

I read through the output documents from the MPEG Geneva meeting and found
some document and descriptions that seem to be related to MPEG-4 on IP.
These are 'resolution 3.2.3' and 'N3381'. Because I could not attend the
Geneva meeting, I would like to ask some questions to know what do these
mean.

As stated in N3381, there are some (not one) RTP payload formats for the
carriage of MPEG-4 streams. Among them, two payload formats,
draft-ietf-avt-rtp-mpeg4-02[8] and draft-ietf-avt-rtp-mpeg4-es-01[6] are
both proposed firstly in the MPEG committee, sent to the IETF as liaison
statements, and demonstrated in the MPEG meetings.

What I would like to ask is which one of the followings is the correct
purpose of the framework proposed in N3381.

(a) It forces to use only one RTP payload format
(draft-ietf-avt-rtp-mpeg4-02[8]), or plus one
(draft-rgcc-avt-mpeg4flexmux-00[5]), in all cases when the proposed
framework is used.

(b) It proposes a framework to treat all MPEG-4 RTP payload formats. It
leaves the selection of RTP payload format depending on the media types
(e.g. MPEG-4 Audio/Visual), applications, network environments, users needs,
etc, while keeping a framework commonly to all RTP payload formats. We can
use MPEG-4 Audio/Visual media aware packetization scheme, e.g.
draft-ietf-avt-rtp-mpeg4-es-01[6], if it satisfies the conditions [2.1] to
[2.6] in section 2 of N3381.

(c) It tries to propose new payload formats other than the existing ones.

If the answer is:

(a) It is really disappointing. I cannot understand why only RTP payload
formats where some Systems groups members are favor on are proposed as an
MPEG common framework. I must disagree such framework be proposed as "on
behalf of ISO/IEC JTC1/SC29/WG11(MPEG)".

(b) It is interesting. This may enable MPEG-4 Audio/Visual aware
packetization scheme stored in the MP4 file format. This may also enable the
combination of OD/BIFS streams and Audio/Visual streams transferred over the
RTP payload formats suited for each audio/visual media type.

(c) Although anyone is free to propose any new payload formats, it needs
*very* good reasons to adopt them considering that there have been several
(not one) MPEG-4 RTP payload formats proposed.

I may be missing something because I was not at the Geneva meeting. I want
to know what was discussed in Geneva. Would you please give us a clarify for
the correct understanding of the proposal?

Best regards,
Yoshihiro

----
        Yoshihiro Kikuchi

E-mail: [email-masked]
Corporate Research and Development Center
TOSHIBA Corporation
TEL: +81 +44 549 2288    FAX: +81 +44 520 1267

---

From: "Olivier Avaro" <[email-masked]>
Date: Wed, 14 Jun 2000 15:16:12 +0200

Dear Yoshihiro, all,

Indeed, your previous mail did not reach the reflector.

The goal of the proposed framework is to achieve end-to-end interoperability
when transporting and operating MPEG-4 content over IP-based protocols. This
includes the packing of MPEG-4 data, but also session set up and control
mechanism, mime types, ... This document is not going to develop new
specifications (like new packing), unless needed, but will rather refer to
existing ones and improve them, if needed as well.

Such framework is expected by the industry for a while, and so far, we have
not been able to deliver something consistent and covering all the aspects
of the delivery and operation of MPEG-4 content on IP.

There are several reasons for this, at least :
1- The complexity of the process. Such framework implies the participation
of several communities with different process (not only MPEG and IETF/AVT,
but also other IETF groups like MMUSIC);
2- The complexity and multiple usage of the MPEG-4 standard. Different
industries have different needs and have so far focused to provide solutions
for their own business.

Some industry in MPEG felt the need of an integrated framework providing
end-to-end interoperability, therefore this initiative. It comes originally
from MPEG since this is were most of the customers are so far. But of course
it respects the process of the IETF groups since this is where the
standardization should occur, and encourages collaboration, as it already
happened in the past.

Still, interoperability is about making choices, and this framework is not
supposed to be a catalog of options. The framework will point to well
identified IETF spec. and will have normative statements. Concerning its
status :

1- From an MPEG point of view : this is a regular output document. It's
construction has been based on a consensus basis and a well defined timeline
has been proposed to finalise it. If divergences of opinions appear, they
should be solved by core experiment process in due time. This has not
happened so far.

2- From the IETF point of view : this is a regular internet draft that is
proposed by one or more specific company who can also mention the status of
this document in MPEG. I am not sure though to which group this draft
belongs to. Any idea ? It will follow the regular process in IETF.

There may be competing documents, but only one will have the MPEG stamp.

Kind regards,

Olivier

---

From: Dave Singer <[email-masked]>
Date: Wed, 14 Jun 2000 09:35:17 -0700

My intention was your (b).  Enable to the support of multiple RTP 
payloads for MPEG-4 data;  require the support of one (so as to have 
a chance at interoperability).  Make sure all schemes share some 
common characteristics, and share other behavior with respect to 
IP-based protocols (SDP, RTSP in particular).

I didn't actually propose any new payload formats, merely observed 
that there was probably good reason to make some.

(By the way, I have reduced the recipient list here to just the 
relevant email groups, with a cc to the person I am replying to, to 
be sure it gets to him, as I assume that everyone is one or both 
lists.)

-- 
David Singer
Apple Computer/QuickTime

---

From: "Yoshihiro Kikuchi" <[email-masked]>
Date: Fri, 16 Jun 2000 22:15:49 +0900

Dear Dave, Olivier, all,

Thank you very much for your answer.

I'm now happy that my misunderstandings of (a) and (c) have been cancelled.
I'm sorry to have the misunderstanding.

I support the trial to have a framework covering multiple MPEG-4 RTP payload
formats. However, I still have my opinion that it still needs careful
discussions before deciding recommended payload formats. (Is it really
single format?) This is nothing to prevent the work, but just to have a
solution that all related party are agreeable. We should NOT have a solution
like (a). I think that N3381 is a good starting point, but may still need
further revisions before stamping MPEG mark.

Fortunately, there is one more MPEG meeting in Beijing before the next IETF
meeting in August. Is it too late to have a discussion before/at the
meeting?

Best regards,
Yoshihiro

---

From: "Olivier Avaro" <[email-masked]>
Date: Fri, 16 Jun 2000 22:35:28 +0200

Dear Yoshihiro,

> I think that N3381 is a good starting point, but may still need
N3381, as it's name indicate, already have an MPEG mark.

> meeting in August. Is it too late to have a discussion before/at the
No, this is not too late, this is just the beginning of the discussion, and
your comments are mostly wellcomed. Please send them on the list for
discussion. A new version of the document will be proposed to the next
meeting based on consensus on this list. Controversial issues should be
proposed to the Beijing meeting as separate document. They will be proceeded
during the meeting. In Beijing, we will output a new version of the
document.

Kind regards,

Olivier

---

From: Dave Singer <[email-masked]>
Date: Fri, 16 Jun 2000 13:46:11 -0700

>further revisions before stamping MPEG mark.

OK.  I agree, I put out 3381 for comment, not as a final solution.

I also agree, that though I feel strongly we need a GP packing format 
which is required (otherwise we cannot achieve interoperability), it 
is possible that I have picked the wrong one, though I don't think so.

>meeting in August. Is it too late to have a discussion before/at the

Let's discuss away.  I will not be in Beijing, but will be on email...
-- 
David Singer
Apple Computer/QuickTime

---

From: Colin Perkins <[email-masked]>
Date: Thu, 22 Jun 2000 21:09:29 -0400

Dave (et al),

Finally managing to catch up on the MPEG-4 related email after changing
jobs, I have a couple of comments...

>principle (that there should be a simple interoperable draft), or the 

With the proviso that we may need one format for elementary streams and 
one for systems streams, I agree on the principle that we should have a
single standard format which MUST be supported. This should be as simple
as possible, perhaps even to the extent of being knowingly less resistent
to packet loss than the alternatives.

>It's clear from the multiplicity of drafts that we are not yet in 

Also agreed. It would be helpful, I think, if the reliability schemes were
written as add-ons to the base format. This would let us mix-and-match the
reliability schemes as appropriate. 

Colin