Subject: [AVT] Re: avt Digest, Vol 30, Issue 23

From: lazzaro <[email-masked]>
Date: Wed, 25 Oct 2006 20:31:41 -0700

On Oct 25, 2006, at 10:32 AM, Randell Jessup wrote:
>
> "Tom-PT Taylor" <[email-masked]> writes:
>>  Precise time synchronization so that every network element will
>>  have access to a very high quality clock (done via a layer 2
>>  profile of the new revision of IEEE 1588). This is being done as
>>  project 802.1AS, and it¹s well on its way.
>
> Just wondering: why is precise synchronization important?   
> Especially since
> sync will drift.  And why do the network elements need to know?   
> Perhaps
> you're trying to instill knowledge into the network of how long a  
> packet
> has been "in-transit"?

Imagine having a set 801.11 wireless microphones that you put
around a drum kit (one on the hi-hat, one on the snare drum, etc).
The antenna and the network stack CPU is inside the mic. Each
mic is sending an independent RTP stream out over the air,
which a DAW is recording simultaneously, one track per mic.

All the mics are picking up the sound of every drum a little bit ...
and so you really want the audio as tightly synchronized as possible,
down to the audio sample if you can (@96 kHz, we're talking  
microseconds).

If you had delay-compensated precise clocking multicast over the  
network,
and if you had a Layer 2 stack on the microphone that recognized it was
sending RTCP packets and rewrote the NTP field on the way to the  
antenna,
the RTP sender app in the mic could just write 0s in the NTP field --  
it's code
could be written to know that the Layer-2 part of the network stack  
would be
rewriting in tight NTP.

Admittedly a pro app, but I'm sure the consumer types on the list
can describe similar sorts of scenarios ...

---
John Lazzaro
http://www.cs.berkeley.edu/~lazzaro
lazzaro [at] cs [dot] berkeley [dot] edu
---

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Randell Jesup <[email-masked]>
Date: Thu, 26 Oct 2006 10:02:29 -0400

lazzaro <[email-masked]> writes:
>> Just wondering: why is precise synchronization important?  Especially
>> since sync will drift.  And why do the network elements need to know?
>> Perhaps you're trying to instill knowledge into the network of how long
>> a packet has been "in-transit"?
>
>

Ok.  Nothing odd here.

>down to the audio sample if you can (@96 kHz, we're talking  microseconds).

Ok.  There are a number of ways to approach this.

>If you had delay-compensated precise clocking multicast over the network,
>sending RTCP packets and rewrote the NTP field on the way to the antenna,
>the RTP sender app in the mic could just write 0s in the NTP field -- it's
>code could be written to know that the Layer-2 part of the network stack
>would be rewriting in tight NTP.

Uhhhh.  That sounds like a horrible layer-violation to me off the top of my
head.   Not to mention that it's wrong - you need the sample time, NOT the
transmission time here.  If you have the sample times (accurately) in the
RTP headers as RTP timestamps, and use RTCP often enough to avoid noticable
drift (see the AVPF spec if needed), then you're set.  No network transit
times needed.

Note also that the speed of sound is on the order of 12"/millisecond, so
you're not going to get microsecond-sample-precise matching on the mics
anyways.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: lazzaro <[email-masked]>
Date: Thu, 26 Oct 2006 07:41:21 -0700

>> would be rewriting in tight NTP.

[...]

> head.

Layers are good for clean specification.  But once the specifications  
are
written, implementors have always merged layers of the stack in the
codebase where there is a compelling reason to do so.

>   Not to mention that it's wrong - you need the sample time, NOT the
> drift (see the AVPF spec if needed), then you're set.

The assumption here is that the path from your application layer to
Layer 2 is so short that the sample time is the transmission time --
the last thing your app does to the RTCP packet is add the RTP timestamp
(setting the sample time), then it hands it off the stack which  
immediately
writes the NTP time and sends the packet.  And you use an OS (or the
lack thereof) that lets the path down to stack be short enough for this
to work.

> No network transit times needed.

If you want the sampling instant for these 5 microphones that have
no wires connecting them to be synchronized, you have to have a
way to get a common clock to all of them, that is good down
to microseconds (see below), and a network-layer clock is not a bad  
approach.

> you're not going to get microsecond-sample-precise matching on the  

Not the issue of interest here.

The high-hat mic hears the high-hat with N millisecond latency, the  
snare
at M millisecond latency, etc, because of the speed of sound M != N.
The snare mic hears the snare with P millisecond latency and the hi-hat
with Q millisecond latency, because of the speed of sound P != Q.  The
values of N, M, P, and Q depend on the relative positions of the mics to
each other and the drums.

If you want to mix those two microphones together and not have phasey  
sounds
happening as the two clock drift (even by a few samples -- cymbals  
and snares
have lots of high frequencies), you want sample-accurate synchronization
between the two mics.

In other words, you want the "sound stage" created by the relative  
timings to
be rock solid.

That's why people run analog wires from all the mics to one A/D
box where all the mics are sampled from the same clock.  The
challenge is to get rid of those wires, and replace them with  
standardized
digital wireless networks.

---
John Lazzaro

---

From: Chuck Harrison <[email-masked]>
Date: Thu, 26 Oct 2006 11:29:08 -0700

John,

I agree with Randell's observation that what is needed here is
an accurate (submicrosecond) relationship between sample instant
and NTP clock. This relationship is communicated through the
RTCP mechansim. The delays between sampling and packet transmission
(whether they arise in the OS, the hardware, or whatever) are
simply part of the transmission jitter/delay budget and are
irrelevant to the precision of the end-to-end system. Especially
so in recording-system or one-way scenarios where latency bounds
can be extremely relaxed.

>From the review in
http://www.ieee802.org/1/files/public/docs2006/as-garner-use-of-p2p-tc-in-avb-0406.pdf
I gather that the Precision Time Protocol & Audio Visual
Bridging work fits into the RTP architecture simply as a precise
way to maintain NTP synchronization. There is nothing broken
about the RTP architecture (although the interpretation of SR
transmission time in RFC3550 sec. 6.4.1 must be finessed to
RTP-timestamp granularity). This work just gives us a better
wallclock.

The only weakness in RTP's sync architecture is that is over-
loads two functions on the same mechanism: (1) transport jitter
measurement and (2) multistream media synchronization. In pro
AV applications, the latter function is far more demanding, but
the RFC text usually emphasizes the former function. Perhaps for
this reason, implementations often perform much more poorly in
regard to cross-media timing than is possible.

One thing that *is* missing from RTP is the concept of sample
clock synchronization across multiple sources. The 5-year-old
AES42 standard for digital microphones can be used for
inspiration. A brief description is on page 10 of this document:
http://www.neumann.com/download.php?download=lect0042.PDF

As an aside, the actual jitter performance of the sample clocks
used for capture and reproduction must be *way* below one sample
(sub-nanosecond, in fact, for pro quality audio). Otherwise the
sampling jitter introduces distortion proportional to the
instantaneous slope of the waveform. In those cases where the
clock jitter is correlated with the signal you can get audible
artifacts at a few hundred picoseconds. No joke! That's why
PLL design for high-end audio equipment is a fine art and often
relies on VCXOs. 

The thing that jumps out at me in looking (for the first time) at
the 802.1as work and your drum-kit scenario is that wireless is
such an important application area and that 802.1as seems to be
ducking the implications of that for now. I have had some
thoughts on that but they are O.T. to this avt forum.

In peace and low jitter,
  Chuck

---

From: lazzaro <[email-masked]>
Date: Thu, 26 Oct 2006 12:43:09 -0700

On Oct 26, 2006, at 11:29 AM, Chuck Harrison wrote:

> The delays between sampling and packet transmission

Performers on stage are typically using monitoring systems (in-ear,
or speakers in front of them on the floor or a stand).  So, the latency
from the wireless mic, through the mixing console, and back up
to the stage is critical, and can't be relaxed -- 2 ms would make
many but not all performers happy (vocalists in particular are
very sensitive to this -- especially using in-ear).

Likewise, using these networks as the first-hop for long-distance
network musical performance systems are also an application
where every millisecond counts.

So, I'm wary of declaring this effort to be latency relaxed ... for
some applications it isn't.

---
John Lazzaro

---

From: lazzaro <[email-masked]>
Date: Thu, 26 Oct 2006 12:43:09 -0700

---
John Lazzaro

---

From: lazzaro <[email-masked]>
Date: Thu, 26 Oct 2006 14:16:22 -0700

On Oct 26, 2006, at 1:47 PM, Chuck Harrison wrote:

>
> However the layer-2 timestamping you seemed to suggest in
>> [...] a Layer 2 stack on the microphone that recognized
>> it was sending RTCP packets and rewrote the NTP field on
>> the way to the antenna [...]
> is ill-conceived for this task as far as I can tell.

All I was suggesting here is an implementation alternative to:

[1] Using a Layer-2 timing service to generate an NTP clock
value that is tightly time-synchronized between all the nodes
on the network (in my case, the multiple wireless mics).

[2] Letting an application call a function to get the current
NTP clock value and putting it in the RTCP NTP slot next
to the matching RTP time stamp clock.

My alternative tried to take a user-land function call/return, with  
all its
return timing uncertainty, out of the picture.  If you like [1] and  
[2] above,
you like what I was trying to do in my quick example ... it
was intended to do exactly as above, but replacing the function
call jitter with the jitter of passing the IP packet down to Layer 2
where the clock code would sit.

In retrospect, my idea is not the right one here. But what I
was attempting to do is important -- if you're going to go through
the effort of designing a way to have all the nodes on a network
have a wall-clock accurate to the sub-audio-sample level, and
you also want to use RTP, and you want to have RTCP
NTP/RTP timestamp pairs that use that wall clock in a way that
captures its accuracy, you HAVE to think about where the code
handling IP and RTP sit in the stack relative to the Layer-2 code,
and how timestamp accuracy is maintained in the implementation.

That's part of why the IEEE folks seem to be coming to talk to us --
they want to use RTP, but they also want to provide timing and
other services related to media at Layer 2, and they want to
figure out a sensible way of integration that keeps the benefits
of adding Layer 2 timing services.  We should help them do that,
otherwise we shouldn't be surprised if they design their own media
transport protocol to use instead of RTP, that handles this problem
by its design.

---
John Lazzaro

---

From: Chuck Harrison <[email-masked]>
Date: Thu, 26 Oct 2006 14:59:11 -0700

Right on, John.

The canonical operation required here is a function call that
returns correlated values of two timebases -- in this case, the
wallclock and the sample clock. This wants to be an atomic
operation at the API level. For the most demanding market niches,
it might be necessary to implement this at the hardware level
(no problem, if you've got a few gates left over in an FPGA),
but in the more usual case I figure it's an issue of tight
coding in a kernel driver.

Note that for this to work it is not critical that the function
call return "very quickly" or even that the two correlated values
it reports are accurately "now". "What time is it now" can 
and probably should be answered with an entirely different
function call.

Unfortunately in typical PC hardware I'm told it's difficult to
get hold of honest information about the audio sample clock,
no matter how slick your layer-2 NTP sync is.

IMHO the "timebase correlation" API call shouldn't be dedicated
to just the sampleclock-to-wallclock function. For example,
querying for the correlation between audio sample clock and
processor clock (e.g. from RDTSC) could be useful and implementable.
Combined with another call correlating NTP to RDTSC value,
and a little fudging, you'd be in pretty good shape to compute
the wallclock:sampleclock relationship.

Peace,
  Chuck

lazzaro wrote:
> [...] what I

---

From: lazzaro <[email-masked]>
Date: Thu, 26 Oct 2006 16:07:42 -0700

[...]

> the wallclock:sampleclock relationship.

For better or for worse, though, we're in the network protocol business
in the IETF, not the API business (with a few exceptions, like  
sockets API).

And so, if there were an implicit way to get wall-clock (as maintained
by Layer 2's network service) and sample-clock (as maintained by the
application level program) timebase correlation through the act of
passing packets up and down the Layer-2/Layer-3 stack interface, then
the IETF and IEEE could do business as usual and create the network  
protocol
that specifies the Layer-2/Layer 3 interface.  And so, I think it may  
be worth going
off and looking for a solution of that nature ...

---
John Lazzaro

---

From: Chuck Harrison <[email-masked]>
Date: Thu, 26 Oct 2006 13:47:38 -0700

John,

I'm all in favor of low latency, and I enjoy rising to that
challenge.

> the way to the antenna [...]
is ill-conceived for this task as far as I can tell. It does
nothing about latency itself, and while it might improve the
accuracy of RTP transit jitter measurements it seems it would
actually *decrease* the accuracy of the sample-clock correlation.
It appears to be another case of letting RTP's packet-jitter-
measurement function trump the precise-media-clock function.
I don't think we want to go there.

Incidentally, getting syntonic sample clocks (i.e. equal
frequency, no phase drift) for multiple audio sources has
long been recognized in pro audio as a *latency* benefit
because it avoids the need to resample the signal. Low-
distortion audio resampling requires a long accurate sinc
filter kernel and therefore introduces a dozen or so samples
worth of delay. That's one reason for the mechanism designed
into AES42 mics.

And to anyone writing a decent musical audio application...don't
even *think* about handling sample clock drift by dropping
or dup'ing samples. Leave that stuff to VOIP and conferencing-
grade apps.

Peace,
  Chuck