Subject: [AVT] Re: [linux-audio-dev] Introducing DMIDI

From: John Lazzaro <[email-masked]>
Date: Wed, 19 Dec 2001 14:34:44 -0800

[note to AVT folks: MWPP discussions on linux-audio-developers are
 ongoing, including posts by Dominique Fober, whose paper:

 http://www.grame.fr/~fober/RTESP-Wedel.pdf 

 describes MIDI network coding; this message is my reply to these
 discussions, CC'd to LAD]

Hi everyone,

	Responding to MWPP-related comments made in multiple email
postings here ...

> Dominique Fober writes:
> 
> There are some interesting ideas in these works and I would like to
> add a contribution to the discussion:

Yes, please do -- more specifically, please consider joining the IETF
AVT working group mailing list, and becoming part of the consensus
process; we have many network experts there, but we're short on MIDI
expertise. See:

http://www.ietf.org/html.charters/avt-charter.html

For details on how to join the mailing list (its a low volume list).

In general, the main thrust of the comments on MWPP at IETF52 in Salt
Lake was that it was important to figure out the minimal amount of
MWPP semantics that needs to be normative, so that implementors can be
free to innovate under MWPP, as long as interoperability can be
maintained.

The next rewrite of the MWPP (probably in Feb 2002 timeframe) is about
figuring out what to remove and what to add to meet this goal;
figuring out what to remove and what to add to permit the
RTESP-Wedel.pdf ideas from working in the MWPP framework would be a
good contribution to this effort.

>
> Therefore, a mecanism to compensate for the latency variation seems to
> me to be necessary.
>

I don't think its always necessary -- MWPP should provide the freedom
to implement latency variation compensation, but I don't think it
should mandate it. In our experience playing over the CalREN 2
Berkeley-Stanford and Berkeley-Caltech links, jitter compensation
wasn't necessary; "play when received" worked well, as long as the
"outliers" of very late packets were handled separately, using
semantic rules (which is what the "ontime" and "late" flags codify).

> [... a later message ...]
>
> Sure, it may be used for this but it isn't: MIDI events are played
> at reception time. In MWPP for example, the timestamp is only used
> to determine wether a packet is too late or not. 

Note that "play when received" isn't a normative part of MWPP.
Quoting from page 7 of the MWPP I-D, first paragraph:

   The most common occurrence is for packets to be normal/ontime. In this
   case, the receiver schedules all commands in the MIDI command section
   of the MWPP payload for execution on the local SA decoder.  The
   details of scheduling are implementation-specific: simple decoders
   (such as [3]) may execute MIDI commands as soon as they arrive, but
   other approaches are possible.

So, MWPP itself is agnostic about when you actually play notes which
are deemed "ontime" -- an implementor is permitted to use a playout
scheme more sophisticated than "play when received", that handles
latency variation. And since the algorithm for deciding when a note is
late or ontime is also implementation specific, I don't see any
show-stoppers for implementing the latency variation ideas in
RTESP-Wedel.pdf in an MWPP-compliant implementation (although I
haven't read the paper yet so I can't say for sure).

In practice, though, any scheme that compensates for latency variation
is going to do so by delaying the playing out some notes, relative to
a "play when received" mode -- jitter reduction has a cost in added
latency. The question is whether the cost is worth the benefit -- and
it would be best if MWPP left that decision in the implementors hands,
rather than making it for them.

> Another point is the efficient use of the transmitted packets: sending
> one packet for each event is probably not the best solution. 

Note that MWPP has a "MIDI Command Section" with a 6-bit length field,
and that except for the first MIDI command, running status coding is
permitted:

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
|R|R|    LEN    |          MIDI Command Payload ...             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Recovery Journal ...                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

So, schemes that code more than one MIDI command per packet are
supportable under MWPP; sfront puts one command per packet in its MWPP
implementation, but its not a normative part of the spec. What is lost
in this simple approach is information about the temporal delay
between commands -- as MWPP is written, the sender is claiming that
all commands in the MIDI Command Payload happened at the moment of the
RTP timestamp.

One addition that could be made to MWPP, is a simple, efficient way to
code this delta-t information, that doesn't impact the packetization
for the "single-command-per-packet" case. However, the AVT culture is
not to "add a feature because it might possibly be used" -- if the
bandwidth reduction (relative to sending one-command-per-packet, each
with its precise RTP timestamp) achievable by adding this complexity
isn't seen as serious issue by the working group, the consensus might
well be to leave the feature out.

> In this case and due to the underlying protocols overhead, the useful
> information part of a packet may become less than 10% of the packet
> size.

We talked about efficiency of MWPP at the Working Group meeting in
Salt Lake, and there was a sense that looking at efficiency without
regards to the absolute bandwidth of the packetization is a mistake.

For fast piano work, we're measured the median MWPP RTP payload
bandwidth (i.e. everything in the bitfield diagram above, but not the
RTP, UDP and other headers) to be about 4700 bits per second.  Since
this compares favorably with the lowest-bandwidth speech codecs that
RTP currently supports, there wasn't much support in the room for
re-engineering the packetization to have better efficiency if there
was a price to be paid in other areas. 

> Considering a 44 bytes overhead (IP + UDP) + the 4 DMIDI header bytes
> intended to address a specific node and device, sending a full MIDI
> data flow (about 1000 3-bytes events per second) requires nearly 400
> kbs when the MIDI rate is 31.25 kbs.

We've clocked sustained, two-handed, fast piano improvisation at about
20 events per second -- significantly less than the 1000-event "maxing
out the MIDI cable" number above. For the MWPP target application of
network musical performance of musical groups, the "number of events
produced by a human" metric seems more appropriate than the "max out
the cable" metric. BTW, note that using controllers seems to decrease,
not increase the number -- its hard to generate data faster than fast
fingers sweeping across a piano keyboard.

This fast, two-handed playing is the sort of input that yields the
4700 bits per second figure for MWPP's payload (the RTP way of
comparing bandwidth is to use packetization payloads, since there are
good header-compression techniques for RTP, UDP, and IP headers that
can be brought into play if saving every bit is important).

If you wanted to send these "max out the MIDI cable" flows through
MWPP, you'd have two alternatives using the existing scheme:

 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+  
|R|R|    LEN    |          MIDI Command Payload ...             |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|                       Recovery Journal ...                    |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

-- Place one MIDI command per RTP packet, to minimize time skew
at the expense of bandwidth.

-- Pack multiple MIDI commands into each MIDI Command Header, and
pay the price in systematic jitter.

The alternative is a more sophisticated coding of the MIDI Command
Payload, that codes time deltas between commands; this information
wouldn't be of use for simple "play when received" implementations,
but would be of use for implementations which did latency variation
removal. This would be an example of the sort of thing that could be
added to a future MWPP I-D revision ...

-------------------------------------------------------------------------
John Lazzaro -- Research Specialist -- CS Division -- EECS -- UC Berkeley
lazzaro [at] cs [dot] berkeley [dot] edu     www.cs.berkeley.edu/~lazzaro
-------------------------------------------------------------------------

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
http://www1.ietf.org/mailman/listinfo/avt

---

From: Dominique Fober <[email-masked]>
Date: Thu, 20 Dec 2001 19:31:21 +0100

>semantic rules (which is what the "ontime" and "late" flags codify).

I agree: MWPP may let the compensation task to the client responsability, provided that the client get the necessary information to do so. Are RTP timestamped packets enough ? I'm not sure: dealing with time on different stations may rapidly reach the problem of clock skew. 
Lets take an example in http://www.cs.berkeley.edu/~lazzaro/sa/pubs/pdf/nossdav01.pdf appendix B which is refered in draft-lazzaro-avt-mwpp-midi-nmp-00 section 3:
the model of time is based on what I call the apparent clock offset (ACO) ie the clock offset between the sender and the receiver + the current transport latency at the time of the measurement (tf minus t0). Deciding wether a packet is "on time" or "late" is based on the latency variation which is evaluated as the current ACO minus the initial ACO. But if the receiver clock is running faster than the sender clock, the latency will appear as constantly increasing and will reach 'maxlate' sooner or later, triggering a set of 'late packets' up to the 'late window' exhaustion (3.5 s).
Of course, the clock skew depends on the clocks used to timestamp the packets and to collect the current time on receiver side. My experience of software clocks (based for example on timer tasks) is that you can get significant drift: for example 1 ms per 10 sec which means that maxlate is then reached in about 6 mn.
I don't want to claim that it's a real problem: using more acurate clocks may solve the problem for a given time. However, fixing the limits (the skew tolerance and the corresponding time limit) would be useful for the protocol implementation.

--df

----------------------------------------------
Dominique Fober               <[email-masked]>
----------------------------------------------
GRAME - Centre National de Creation Musicale -
9 rue du Garet  69001 Lyon France
tel:+33 (0)4 720 737 06 fax:+33 (0)4 720 737 01
http://www.grame.fr

---

From: Dominique Fober <[email-masked]>
Date: Fri, 21 Dec 2001 12:01:24 +0100

>can be brought into play if saving every bit is important).

The two-hands piano performance is a simple case. We may consider a more general case where the performance appears as remote control of audio synth. The controler may be a MIDI instrument but also an acoustic instrument which signal is transformed into MIDI using a pitch tracker for example. In this case, you may have a lot of control information (pitch bend, controlers) generated, which rate depends on the tracker audio input buffer size. Using a fft on a 512 frames buffer for example may reach a 200 events per sec. data flow.
More generally, the MIDI limitations in regard of control have been pointed out very early and some proposals have been made for improvements or replacement [see Computer Music Journal Vol 18 No 4 Winter 1994 for example]. This was just to highlight the fact that in common musical situations and depending on what king of control you want, you may rapidly reach a substantial data flow.

>MWPP, you'd have two alternatives using the existing scheme:

not necessary in systematic jitter, you can choose to group the events at regular interval. Of course, the interval length is to be added to the end to end delay but as it is constant, it should have no effect on the jitter. The problem then is to evaluate wether such an interval exists, which meets the real-time performance requirements while providing a real bandwith economy.

>added to a future MWPP I-D revision ...

You can consider delta time between commands or time offsets to the packet timestamp. In both cases, the delta values are bounded by the grouping interval which should be always be kept low. Therefore I don't think that any specific strategy (as in MIDI Files for example) is to be provided to code delta times.

--df