Subject: [AVT] FWD: Explaining AVT to the computer music folks ...

From: John Lazzaro <[email-masked]>
Date: Mon, 1 Jul 2002 18:12:20 -0700

Hi everyone,

	Thought I should send this along, this weekend I posted a link
to draft-ietf-avt-mwpp-midi-rtp-04.txt to a collection of mailing
lists where different segments of the computer music community hang
out. I try to do this every few months, to keep the impacted
communities in the loop ...

	As often happens, a lot of the feedback was of the "please
change MIDI itself too." However, the person who raised the issue
posted it across a few other mailing lists, starting a public
discussion. So, it seemed appropriate for me to address the matter on
the musical mailing lists, and in the process explain where AVT fits
into the standards world to the computer musician folks ...

	For context, Michael is Michael Gogins, NYC composer and music
software developer, who originally raised the issue, Richard Dobson is
a UK-based composer and music software developer, and Eric Schierer
worked extensively on the Structured Audio standard for MPEG.

	I reference IETF and AVT a lot in the reply (which is why I'm
sending it along here), but hopefully in an appropriate way ...  I'll
probably reuse this text in the future, so please let me know if there
are potential issues with the way AVT is characterized. 

---

[ my posting to media_api and csound-dev mailing lists starts below]

        The short reply to Michael's original note is, as Richard
Dobson notes, the IETF is not the keeper of change control for MIDI,
and is not in the business of defining new gestural computer music
languages (thank goodness :-).

        The part of the IETF that is involved with the MWPP effort,
the Audio Visual Transport Working Group, DOES have change control
over its own protocol -- the Real Time Protocol, or RTP -- and a big
part of what changing RTP means is adding new "codecs" to the
supported list. For example, along with MIDI, RTP packetizations under
construction at the moment include Dolby AC-3 and motion JPEG 2000.

        So, most of Michael's suggestions are not in scope for what
I'm doing in the IETF, although there is one exception:

> In the same spirit, it also would be extremely useful to have options
> for extra MIDI controller numbers and extra MIDI channels (beyond 16).

        MWPP supports setting up multiple MWPP RTP streams, with tools
for describing the relationship between the MIDI name spaces of the
stream. This effectively gives you the "beyond 16 channels", just as
buying a MIDI interface with multiple jacks does. See Appendix C.4 of:

http://www.cs.berkeley.edu/~lazzaro/sa/pubs/txt/current-mwpp.txt

        But this configuration option, as well as the other MIDI
"enhancements" in MWPP, act to simulate what MIDI users do when they
buy a MIDI switch box or configure the MIDI screen on MIDI software
and hardware devices. The integrity of the MIDI command language as
defined by the MMA (http://www.midi.org) is kept intact.

        However, I do think IETF technologies have a strong role to
play in the broad problems Michael wants to solve -- but the path is
one that follows the telephony tradition, not the computer science
tradition. This isn't surprising -- RTP is often used with another
IETF protocol, the Session Initiation Protocol (SIP), that is the
protocol that lets Internet Phones place calls to each other.

        How does telephony solve upward compatibility? Let's take
modems as an example. 33.6 kB modems can call 2400 baud modems and a
connection can happen -- a connection at 1980's bandwidths, but a
usable connection nonetheless, if your needs are 1980's-era needs.

        This compatibility isn't the sort of backward-compatibility
that computer instruction sets maintain, which is the sort of
compatibility Michael was suggesting with his MIDI extensions.  The
algorithms and waveforms used for 33.6 kB modems look nothing like the
waveforms of 2400 baud modems.

        Instead, the modem session starts with a negotiation, where
the two parties find the fastest modem standard they both know. The
downside of this is that every new generation of modem needs to be
able to implement all previous modem standards. The good side of this
is, its easy to come up with a negotiation algorithm that is forward
and backward compatible, so its possible to produce modems that you
know will be able to negotiate with modems designed 10 years from now.

        Many telephony standards use this negotiation method as the
primary tool for compatibility -- including the IETF tool SIP for
starting Internet Phone conversations. When two parties call each
other, part of what SIP coordinates is a negotiation between the two
phones, to find a common RTP codec. Phones usually have a few
"baseline" audio codecs, along with a collection of higher-quality
codecs. If the two phones share a common higher-quality codec, SIP
switches both phones to use that codec; if all else fails, the phones
default to one of the baseline codecs.

        One vision for MWPP is that, when you walk into a musical
instrument store, the rack-mounts and controller keyboards all have
RJ-41 Ethernet jacks alongside the MIDI DIN jacks, and the software in
the boxes speak SIP and RTP. You plug your boxes into a Ethernet
switch, they are peers on the network, and a SIP session negotiation
happens, to find the right codecs for musical gestures.

        In this scenario, MWPP is the baseline codec -- all the boxes
know how to speak MIDI because they have DIN jacks to drive and sense,
and so if all else fails, they use RTP and MWPP to make a virtual MIDI
network that replicates the physical MIDI network that would happen if
you plugged DIN cables into all of the boxes. This scenareo has a
variety of incremental advantages over simply shipping boxes with DIN
jacks and no RJ-41 Ethernet jacks, which is why manufacturers might
consider doing it.

        However, once the SIP and RTP infrastructure is in place, MWPP
is "just another codec." It's easy to imagine Roland (for example)
designing a new gestural language to replace or extend MIDI, and an
RTP packetization for that language. When a SIP session starts, the
Roland keyboard and the Roland rack-mount do negotiations and discover
they have a "better" codec than MWPP to use, and so they switch to use
it instead. The Roland keyboard and the EMU rackmount only both know
MWPP, so they speak MIDI to each other. Voila, upward compatibility,
without hacking the MIDI opcode format.

        Some replacement codecs might be standardized -- maybe an MMA
effort, maybe an effort from the CSound community, or maybe, as Eric
Scheirer notes, from MPEG:

> The generic MPEG-4 over RTP packetization supports the streaming
> SASL specification from the Structured Audio spec.  There's no
> reason you couldn't use that even if you weren't controlling SAOL
> with it.

        so you could use SASL. However, I should note:

> I suspect John Lazzaro's SA implementation (GPL licensed)
> already supports SASL over RTP and so you could steal the
> source from sfront.  Is that right, John?

        MWPP doesn't use the midi_event bitstream coding in Section
5.5 of the SA standard. RTP is designed for use over unreliable
network protocols (like UDP; RTP can work over reliable protocols like
TCP too, but an RTP packetization that only works for TCP probably
would be a difficult sell for standards-track), and the simple
midi_event coding has no resiliency. Most of the 70 pages of the MWPP
I-D describes the recovery journal, a scheme for adding resiliency to
packetized MIDI, so that lost packets don't result in stuck notes and
console faders that are in the wrong position forever.

        Likewise, a SASL packetization that has the resiliency to work
well with RTP won't be based on the score_line bitstream coding --
like MWPP, the SASL packetization will need resiliency that works in
the context of how SAOL and SASL interact. I actually have such a
packetization under development and partially implemented in an
internal version of sfront, but its not in ready for release -- right
now, the focus is on reaching closure on MWPP, so its been on the back
burner.

-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu     www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt