Subject: [AVT] Re: Layer 2 synchronization and clock

From: lazzaro <[email-masked]>
Date: Sat, 28 Oct 2006 12:23:07 -0700

On Oct 28, 2006, at 7:16 AM, Randell Jesup wrote:

> Ok, if this is a replacement for NTP that's fine, though requiring  
> anything
> of network elements needs a really good reason, and it'd better be  
> cheap,
> easy to do and not get wrong, and give some sort of benefit outside  
> the
> very limited space mentioned here if you want hardware vendors to  
> implement
> it in enough equipment for this to be really useful to you.

Consumer stereo and 5.1 speakers are not a limited space, in the  
sense of
yearly revenue of the market.  It will be many years before video- 
conferencing
makes the revenue numbers of consumer audio speakers :-).

And while any of the individual pro applications we've mentioned may  
seem
like a "limited market" in comparison to the consumer speaker market,  
you
have to consider the "long-tail" effect.  If you build core  
technology that works
well enough to be usable for these small markets, all of the small  
markets
adopt it and together it becomes a sizable market.  And also, content  
creation
technology needs to be nurtured so that new music people want to  
listen to
comes to be -- Bing Crosby sold a lot of radios, the Beatles sold a  
lot of turntables.

Focusing on stereo speakers, here's is the basic problem.  If you  
want to send
digital audio to stereo speakers today, you have two choices:

[1]  The left speaker and the right speaker are housed in separate  
cabinets.
You send the digital signal to the left speaker (wireless or wired),  
and connect
the left and right speakers with a (usually) analog cable.  This lets  
you get good
stereo separation (you place the speakers in different locations in  
the room), but
the cable connecting the left and right speakers defeats the goal of  
a wireless
speaker system.  It doesn't solve "the spouse problem" -- one spouse  
wants the
best sounding speakers, the other spouse doesn't want ugly wires  
running around
the living room, so the couple leave the store without buying your  
product.

[2] You put the left and right speakers in the same enclosure (the  
boom-box form
factor).  The problem here is that the box needs to be pretty big to  
get any
stereo separation at all, and even if you work very hard, the  
separation is
not very good compared to what you can do with two cabinets.

The IEEE proposal is about adding a third choice. Two speaker  
cabinets, each
with its own wired or wireless Ethernet (PoE lets you get rid of the  
power cord,
which in some markets is more interesting than getting rid of network  
cables).
For this to work, you really need the sort of sub-sample-period  
timing accuracy
that has been mentioned on this thread, for all the nodes  
participating in this network.

> What bothers me the most is the "insert time in the network stack"  
> part.
> a) latency from sample to stack is rarely fixed except in the most
> special-purpose hardware.

The conjecture is, if smart people come to the problem with an open  
mind,
they'll find the abstraction that hasn't been invented yet to solve  
this problem.

Application Layer Framing (ALF) is at the heart of RTP. The  
conjecture was
that for real-time media, the application is the entity that needs to  
break up
the stream into packets and fill in the headers with timing and  
sequence info,
not the network stack.

The opportunity the IEEE folks pose -- the potential for maintaining  
a very accurate
clock across a network span, an accuracy that can only be maintained  
using
hardware directly managed by Layer 2 -- is a challenge to ALF  
itself.  When done
in a simplistic way, the act of passing the clock between Layer 2 and  
the application
layer destroys what makes it valuable -- it's accurate timing.

The goal is to come up with new abstractions (or repurpose existing  
ones)
that finds a way to split media between Layer 2 and Layer 3 while  
maintaining
the clock accuracy.

There's a strong economic incentive to (1) have the new abstractions  
be compatible
with RTP, and (2) have the new abstractions take the form of a  
network protocol
instead of an operating system API.   The incentive for (1) is to use  
existing
payload format protocols and to play existing streams.  The incentive  
for (2) is
that network protocols are one of the few places for interoperability  
left in computing.

> b) the more I hear the more I think you want to
> get the true sampling time for a set of samples.  This is really a  
> hardware
> and system issue; it's not a network or protocol issue.  IMHO

If you look at it the way you suggest, it leads to you to designing a  
new
media transport protocol for Layer 2, which someone (Layer 2 for the
protocol, the operating system, or applications) has to transcode from
something  (an audio API like CoreAudio, or a network media API like
RTP).  There are many economic incentives to avoid this sort of  
reinvention,
and that's why the IEEE folks are here kicking our tires.  Telling  
them they
really don't want to buy our car is not what we should be all about  
here in AVT.

---
John Lazzaro
http://www.cs.berkeley.edu/~lazzaro
lazzaro [at] cs [dot] berkeley [dot] edu
---

_______________________________________________
Audio/Video Transport Working Group
[email-masked]
https://www1.ietf.org/mailman/listinfo/avt

---

From: Randell Jesup <[email-masked]>
Date: Mon, 30 Oct 2006 14:18:15 -0500

lazzaro <[email-masked]> writes:

>> anything of network elements needs a really good reason, and it'd better
>> be cheap, easy to do and not get wrong, and give some sort of benefit
>> outside the very limited space mentioned here if you want hardware
>> vendors to implement it in enough equipment for this to be really useful
>> to you.
>
>Consumer stereo and 5.1 speakers are not a limited space, in the sense of
>conferencing makes the revenue numbers of consumer audio speakers :-).

Though most of those aren't attached to ethernet... :-)

>And while any of the individual pro applications we've mentioned may seem
>like a "limited market" in comparison to the consumer speaker market, you
>have to consider the "long-tail" effect.  If you build core technology
>that works well enough to be usable for these small markets, all of the
>small markets adopt it and together it becomes a sizable market. 

Yes, that can happen, but you need a large enough driver to get it built
in the first place.  If you're dependent on other people putting it in
there for close to free, you had better have a good reason for them to do
so.  If it requires single-sourced (or small-number-sourced) hardware
that's much more expensive (due to low volume/competition), then it's
really hard to get those small users to jump on and build the market.
Not quite a direct example, but look at 802.11 vs. Homeplug vs HomePNA.

>And also, content creation technology needs to be nurtured so that new
>music people want to listen to comes to be -- Bing Crosby sold a lot of
>radios, the Beatles sold a lot of turntables.

Ok, I suppose, kinda, but I don't see how that has any relevance.

>Focusing on stereo speakers, here's is the basic problem.  If you want to
>send digital audio to stereo speakers today, you have two choices:
>
>cabinets.  You send the digital signal to the left speaker (wireless or
>wired), and connect the left and right speakers with a (usually) analog
>cable.  This lets you get good stereo separation (you place the speakers
>in different locations in the room), but the cable connecting the left and
>right speakers defeats the goal of a wireless speaker system.  It doesn't
>solve "the spouse problem" -- one spouse wants the best sounding speakers,
>the other spouse doesn't want ugly wires running around the living room,
>so the couple leave the store without buying your product.

Ok, so that's a more widely-applicable problem set.  It also has majorly
relaxed requirements compared to some of the requirements mentioned.  This
doesn't need a tightly synced network-wide clock.  It merely needs
synchronization between multiple receivers.  Your two main choices here are
a) multiple unicast wireless streams and
b) a single multicast wireless stream.

A multicast wireless stream has a big advantage here, since reception time
should be identical (modulo speed-of-light).  In that case, you don't need
actual clock synchronization; you just need a fixed delay from reception to
playout.

If you go multiple-unicast, you'll use more bandwidth and the reception
times to playout time will be both longer and much more variable.  This
requires some level of clock sync, but since it's all single-network stuff,
there are quite a few ways to do it either with NTP or SNTP or some
derivative.  Again, a clock-sync protocol that uses some type of multicast
clock time will solve the normal "how do I deal with estimating network
latency" problem with NTP.

I suspect you were trying to solve this "pulse" problem by effectively
recording in the packet the time it went from internal buffers to the
antenna/wire (or, somewhat equivalently, the delta time between submission
to the stack and transmit time).  This would be better modeled on how IP
handles hop counts.  If you want this sort of scheme to really work well,
the receiver of a packet has to add the apparent transmission
period from the approximate time-in-transit, or the sender has to add the
expected transmission period before actually transmitting it based on
knowledge of the channel bitrate.  In theory the receiver could in hardware
record the time difference between first and last bits and use that.  And
you'll still be missing speed-of-medium (light, electricity) delays and a
non-knowledgable network element (router/bridge/etc) could induce
unaccounted-for delay.

Another problem with this is that the line protocol might have issues
that interfere with trying to record the transmission time before actually
transmitting.  This is something more of a hardware issue than software,
but it's tied to design of the lower layers of the network, and if it's
some form of "wait until free then xmit and check for collision" (like
ethernet), you may have a problem with not being (quite) ready to start
transmitting as soon as the channels goes free.

>The IEEE proposal is about adding a third choice. Two speaker cabinets,
>each with its own wired or wireless Ethernet (PoE lets you get rid of the
>power cord, which in some markets is more interesting than getting rid of
>network cables).  For this to work, you really need the sort of
>sub-sample-period timing accuracy that has been mentioned on this thread,
>for all the nodes participating in this network.

Or use power cords, and use HomePlug (which supports QoS, and generally is
lower latency than 802.11).  That also avoids the very limited PoE
voltage/amp limits, which would be a bummer in a speaker.

>> What bothers me the most is the "insert time in the network stack"  part.
>The conjecture is, if smart people come to the problem with an open  mind,
>they'll find the abstraction that hasn't been invented yet to solve  this
>problem.

Which is where I'm trying to push you - it seems like you've gone too far
down a rabbit hole already with assumptions about how to solve the (vaguely
defined) problem.  Nail down that problem-set first.  How widely applicable
do you want/need this to be.  How accurate?  How widely distributed (local
transmission, local subnet, local set of subnets, WAN, etc)?  What cost?

>Application Layer Framing (ALF) is at the heart of RTP. The conjecture was
>break up the stream into packets and fill in the headers with timing and
>sequence info, not the network stack.
>
>The opportunity the IEEE folks pose -- the potential for maintaining a
>very accurate clock across a network span, an accuracy that can only be
>maintained using hardware directly managed by Layer 2 -- is a challenge to
>ALF itself.  When done in a simplistic way, the act of passing the clock
>between Layer 2 and the application layer destroys what makes it valuable
>-- it's accurate timing.

All you need is for that network-level sync to feed an offset value and
(possibly) drift compensation rate that the local hardware and software can
use to accurately convert local hardware time to network time.  Once you
have that, you can sample and playback accurately.

>The goal is to come up with new abstractions (or repurpose existing ones)
>maintaining the clock accuracy.

You're assuming again that splitting the layers is the right solution,
which you certainly haven't proved.

>There's a strong economic incentive to (1) have the new abstractions be
>compatible with RTP, and (2) have the new abstractions take the form of a
>network protocol instead of an operating system API.  The incentive for
>(1) is to use existing payload format protocols and to play existing
>streams.  The incentive for
>(2) is that network protocols are one of the few places for
>interoperability left in computing.
>
>
>> hardware and system issue; it's not a network or protocol issue.  IMHO
>
>If you look at it the way you suggest, it leads to you to designing a new
>something (an audio API like CoreAudio, or a network media API like RTP).
>There are many economic incentives to avoid this sort of reinvention, and
>that's why the IEEE folks are here kicking our tires.  Telling them they
>really don't want to buy our car is not what we should be all about here
>in AVT.

Just tell them the right car to buy.  :-)

Seriously, I think a strong candidate is a better (and more-local) variant
of NTP (perhaps dependent on changes in the network elements and their
stacks), combined with normal RTP.  But here I'm making assumptions about
the problem-set myself, since I don't know what it is exactly.

-- 
Randell Jesup, Worldgate (developers of the Ojo videophone), ex-Amiga OS team
[email-masked]
"The fetters imposed on liberty at home have ever been forged out of the weapons
provided for defence against real, pretended, or imaginary dangers from abroad."
		- James Madison, 4th US president (1751-1836)

---

From: lazzaro <[email-masked]>
Date: Mon, 30 Oct 2006 11:57:27 -0800

> transmission, local subnet, local set of subnets, WAN, etc)?  What  

The approach I'm going to take personally is to spend a few weeks  
thinking
this thread over, and if I come up with a set of answers to your  
questions
worthy of documentation, I'll write an I-D.  Then we can resume the  
discussion
with concrete ideas in a document to reference.

However, a fair number of folks on the list have thought these issues  
through
already, and may be ready to respond to your points now.  If so, I'd  
recommend
those folks step up and continue the discussion now -- both here on  
the list and
in San Diego next week.

---
John Lazzaro

---

From: "Joe Kelsey" <[email-masked]>
Date: Mon, 30 Oct 2006 12:26:14 -0800

I think that Randall just doesn't see the NTP problem.

Clock synchronization is a very difficult problem and it has to occur as soon as you apply power to the device.  NTP synchronization takes almost a day to get to the microsecond level.  Wireless sound device synchronization has to occur within seconds of power or it is not useful for real-world devices.

If you look at IEEE 1588, the biggest difference from NTP is that the protocol requires definite timing of network latency by insertion of a timestamp at both transmission and reception.  NTP only inserts one of these timestamps and assumes a fixed value for the other.  If you have the application transmission time stamp, the network stack transmission time stamp, the network stack reception time stamp and the application reception time stamp, then you can synchronize your clocks very quickly to a very fine point.  Without the network stack transmission time stamp (missing in NTP), this takes hours to do.

Now, since you had to modify NTP to introduce layer violations to actually synchronize clocks quickly, what can you do to RTP to make it more acceptable.  I think that Randall is correct here, there really is no reason to open up RTP to network layer violations, but the clock synchronization issue is one that many computer people just seem to assume has already been solved.  NTP is nice and it works, but it takes a long time to produce its results.  Not every consumer device can afford to wait half a day in order to start working.  Sometimes even waiting a few seconds is too much.

/Joe

>>> lazzaro <[email-masked]> 10/30/2006 11:57 AM >>>

---
John Lazzaro

---

From: "Yaakov Stein" <[email-masked]>
Date: Mon, 30 Oct 2006 22:40:20 +0200

Sorry to come in on this thread so late.

At the last IETF meeting there was a bar BOF
on extending NTP in order to solve several problems,
including ones similar to those being discussed here.

A mailing list has been set up
https://www1.ietf.org/mailman/listinfo/tictoc 
and two design teams are working on requirements documents.

We decided at the last moment NOT to hold the BOF in San Diego, 
amongst other reasons in order to enable 1588v2 to progress
a bit more. However, we intend holding this BOF in Prague.

Y(J)S

---

From: Randell Jesup <[email-masked]>
Date: Mon, 30 Oct 2006 17:02:54 -0500

>I think that Randall just doesn't see the NTP problem.

I do, though I know relatively little about NTP - I trust your comment that
it synchronizes too slowly for this purpose.  This was why I suggested that
one of the first things that should be considered is a "better" NTP,
perhaps one that only works within a limited network domain (subnet, local
layer 2, etc).

>Clock synchronization is a very difficult problem and it has to occur as
>soon as you apply power to the device.  NTP synchronization takes almost a
>day to get to the microsecond level.  Wireless sound device
>synchronization has to occur within seconds of power or it is not useful
>for real-world devices.

Agreed.

>If you look at IEEE 1588, the biggest difference from NTP is that the
>protocol requires definite timing of network latency by insertion of a
>timestamp at both transmission and reception.  NTP only inserts one of
>these timestamps and assumes a fixed value for the other.  If you have the
>application transmission time stamp, the network stack transmission time
>stamp, the network stack reception time stamp and the application
>reception time stamp, then you can synchronize your clocks very quickly to
>a very fine point.  Without the network stack transmission time stamp
>(missing in NTP), this takes hours to do.

Ok; not knowing the details of NTP I can accept that.  It sounds like
there's a way there to make an improved (perhaps local-only) NTP.

>Now, since you had to modify NTP to introduce layer violations to actually
>synchronize clocks quickly, what can you do to RTP to make it more
>acceptable.  I think that Randall is correct here, there really is no
>reason to open up RTP to network layer violations, but the clock
>synchronization issue is one that many computer people just seem to assume
>has already been solved.  NTP is nice and it works, but it takes a long
>time to produce its results.  Not every consumer device can afford to wait
>half a day in order to start working.  Sometimes even waiting a few
>seconds is too much.

Right; there should be a series of time-synchronization protocols (or
variants of one protocol) for different domains.  A fast-converging but
limited to (fairly) local network protocol has value.  You probably don't
want to run it on a WAN/Internet for general synchronization of servers and
workstations.  The best solution is one where it's effectively the same
protocol but with different options or data (supplied by hardware if
needed).

The other side is that if you want this to work on local/household/etc
networks, it's much easier if you can handle it with minimal or no support
from the intermediate network elements.  Alternatively, make it of value
for other (common) applications (and simple) so that router/etc makers will
have no reason _not_ to include it.  If it requires a hardware spin or
non-trivial changes in the software, it may never be more than a niche
protocol.  Can it be done entirely at the level of an NTP-replacement,
without modifying network stacks?  That would be a major advantage.