Subject: [AVT] Re: FYI: Structured-Audio-related IETF AVT individual contribution

From: John Lazzaro <[email-masked]>
Date: Mon, 27 Aug 2001 20:22:54 -0700

[note my reply also cc'd to [email-masked], IETFers, I'm
 answering a few questions from Philippe Gentric on
 draft-lazzaro-avt-mwpp-midi-mpeg4-00.txt]

Hi Philippe, 

> Did you have a look at "draft-ietf-avt-mpeg4-multiSL-01.txt" ?

Yes.

> Can you point to specific items that
> would be missing in the above memo
> and that your application requires ?

If the application goal is network
musical performance, then multiplexing
a Structured Audio ES with any other
ES in the same RTP packet isn't a good
idea, even if the other ES is also a 
Structured Audio stream. Here are some
of the reasons why:

  [1] The chief challenge here is keeping
  the total latency -- from motor movement
  of musician #1 to eardrum motion of 
  musician #2 -- to a very low number.
  Under 5ms is best, 25ms or so is the
  point where things start to fall apart. 

  To meet this latency challenge, a sender grabs
  an incoming MIDI command (created by
  an asynchronous musician gesture) off
  the soundcard port as soon as it completes,
  stuffs it in an RTP packet, and sends it off;
  oftimes, the receiver uses no buffering.

  This sending pattern isn't compatible with 
  normal streaming media, and so trying to 
  multiplex them together is a bad idea.

  [2] The recovery journal scheme only works
  if every RTP packet sent contains a recovery
  journal in it -- there are several reasons
  for this, I can explain in detail if needed.
  But this requirement means that if you were
  multiplexing the Structured Audio ES with
  another stream, every RTP packet sent would
  need a recovery journal for the SA stream,
  even if the SA stream had no new MIDI data
  to send.

  [3] MIDI events are indeed events -- they
  execute at a discrete moment in time. The
  receiver needs to know that moment in time,
  so that it can model the nominal latency
  of the channel, and so it can determine
  when a MIDI command has arrived "too late"
  relative to this latency, and execute 
  recovery semantics. This usage of timestamps
  is special, and is unlike normal media codecs.

So in short, using an SL packetization like
draft-ietf-avt-mpeg4-multiSL-01.txt doesn't
work well for this application because the
recovery journal system really wants an RTP
packet to itself, and because the sending
patterns of the application are so different
than most media.

In addition, many applications don't need the
features of MPEG 4 Systems; network musical
performance is solved well in many cases with
a single Elementary Stream, and it seems best
to design an optimized packetization for that
Elementary Stream.

So, we designed 

http://www.ietf.org/internet-drafts/draft-lazzaro-avt-mwpp-midi-mpeg4-00.txt

as an Elementary Stream packetization that
can stand alone w/o MPEG 4 Systems, and use
the IETF multimedia toolsuite for all of its 
session needs.

> For example can you qualify what are the conceptual differences between
> "checkpoint packets" and "sequence break packet" and
> a "random access point" (as defined in MPEG)?

I'll try to answer this, but its hard because I only have
a superficial knowledge of MPEG 4 Systems at that level. 
The real trick with the recovery journal scheme is
that you can lose an _arbitrary_ number of RTP packets sent
after the checkpoint packet, and the _first_ RTP packet that
arrives after the loss packets ends all catastrophic problems
immediately upon receipt. In a practical sense for MIDI, this
means all "stuck notes" get "unstuck", and the "MIDI state
machine", as it were, gets back into a known state. 

Getting this to work (including safe sender movement of the
checkpoint packet forward) requires:

-- The sender can say "I know receiver X has received a specific
   previous packet Y, via the RTCP LSR field I looked at recently".
-- The sender can say "I know packet Y has a recovery journal,
   because every packet does. And so, receiver had a recovered
   state at the time of packet Y."

It would seem to me that this sort packet-by-packet knowledge
isn't a part of MPEG 4 Systems, since as I describe it its
so IP-datagram specific. So, that would be the essence of
the difference, I would guess ...

-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu     www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
http://www1.ietf.org/mailman/listinfo/avt

---

From: "Lim, Young-Kwon" <[email-masked]>
Date: Tue, 28 Aug 2001 14:01:28 +0900

Dear John, phillipe and all,

> Structured Audio stream. 

It doens't require to multiplex multiple ESs in one RTP stream.
It only allows to concatenate multiple SL packets from one ES in one RTP packet for efficiency.

> Here are some

Again, you don't need to wait for multiplexing. 
You can send your RTP packet as soon as it's ready.

>   even if the SA stream had no new MIDI data

Would you explain more about "recovery journal"?

>   is special, and is unlike normal media codecs.

I think there is no problem to use generic payload format to fulfill this requirements, isn't it?

> patterns of the application are so different

I don't understand what is real difference.

> to design an optimized packetization for that

You can configure generic payload format containing null header except CTS. It doesn't require to use SL if you don't need it.

BTW, this discussion is more relavant to 4onIP AHG. Would you continue this discussion there?

Sincerely,
Young.