Subject: Re: [ntpwg] ntpwg Updated NTPv4 Protocol Specification/timestamp location

From: "STUART VENTERS" <[email-masked]>
Date: Fri, 22 Feb 2008 11:11:52 -0600

Kurt:

Thanks for the support.  In the notes Dave published, checkout the section titled 'Errors Due to Interworking Between Software, Hardware, and Driver Schemes' and see if it addresses your test case.  

http://www.eecis.udel.edu/~mills/timestamp.txt

As you can see, Dave and I are now pretty much in agreement with the analysis.  Hopefully it can morph into an appendix to the NTP spec.

The notes describe a variety of timestamping schemes and shows how they interoperate.  The simplest scheme (called hardware) uses timestamps at the start of packet.  This is the scheme the 1588 spec and hardware support.  Another scheme (called software) uses tx timestamps at the start of packet and rx timestamps at end of packet. This is the scheme used by the reference NTP implementation, and I suspect also the one which interoperates best with vanilla switches and routers.

My main goal in this discussion was to get some specific scheme in the spec so that as implementations get more precise, they will continue to interoperate.  The notes propose a single scheme (hardware) that implementations should use for interoperability.  This meets my original goal and I'm ok with this choice on the grounds that it is simple.

An unexpected outcome in the discussion was that the simple choice that 1588 made was not optimal for some cases without on-path support.  The software scheme, appears to sometimes offer better accuracy.  Unfortunately, it's also more complex and different.  I'm torn between the simple hardware scheme and a slightly weird software scheme which should work better with existing implementations and on paths without on-path support.  (Note that this is not a question of what measurement means are available, because the implementation can transpose from one scheme to another.)

As I said before, I can live with the choice of the hardware scheme (SOPtx, SOPrx), but am curious as to how the group feels about the software scheme (SOPtx, EOPrx).

Regards,

Stuart

It looks like the analysis Dave published 21st, 

Kurt Roeckx wrote:
> On Thu, Feb 07, 2008 at 05:34:08PM +0000, David L. Mills wrote:
>> Stuart,
>>
>> No, compare your equation with mine. The view has to be from A, so the 
>> delay cancels.
> 
> 
> I'm still not convinced.  So I'll try to make an example.  Let's assume
> that it takes 2 second for the packet to travel from A to B and the
> other way around.  That is, if A sends a physical signal from low to
> high, B will see that change 2 seconds later.  Let's also assume that it
> takes A 1 second to transmit the packet.  Then let's assume that A's and
> B's clock are in sync, and that A takes the timestamp at the start of
> the packet, and B and the end.
> 
> We start with A sending a packet at time 0.0.  A takes the time
> when the packets starts to be transmitted, so sets T1 to 0.0.
> 
> The packet will start to arrive at B at 2.0, and stop at
> 3.0.  B takes the timestamp at the end of the packet
> and will set T2 to 3.0.
> 
> It takes B 1 second to process the packet.  It will start to transmit
> the packet at 4.0.  But since it timestamps the packets after
> it's completly send, it will set T3 1 second later to 5.0.  B will inform
> A about T3 with an other packet it will send later.
> 
> The packet starts to arrive at A at 6.0, which is when A takes
> the timestamp, getting us T4 = 6.0.  The packet completly arrives at
> 7.0, but that doesn't matter.
> 
> This gives:
> offset = 1/2 * [(T2-T1) + (T3-T4)] = 1/2 * [(3-0)+(5-6)] = 1.
> 
> While we expected the result to be 0.
> 
> Note that for the delay we don't have the problem.
> delay = (T4-T1) - (T3-T2) = (6-0) - (5-3) = 4.
> 
> 
> Kurt

_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: Brian Utterback <[email-masked]>
Date: Fri, 22 Feb 2008 13:07:24 -0500

I seem to recall that there is an implementation in FreeBSD to
get the transmit interrupt timestamps. Is anybody here familiar
with how that was implemented? I talked to some Solaris interface
driver writers and they seemed to think that it was not practical.

STUART VENTERS wrote:
> Kurt:
> 
> Thanks for the support.  In the notes Dave published, checkout the 
> section titled 'Errors Due to Interworking Between Software, Hardware, 
> and Driver Schemes' and see if it addresses your test case. 
> 
> _http://www.eecis.udel.edu/~mills/timestamp.txt_
> 
> 
> As you can see, Dave and I are now pretty much in agreement with the 
> analysis.  Hopefully it can morph into an appendix to the NTP spec.
> 
> The notes describe a variety of timestamping schemes and shows how they 
> interoperate.  The simplest scheme (called hardware) uses timestamps at 
> the start of packet.  This is the scheme the 1588 spec and hardware 
> support.  Another scheme (called software) uses tx timestamps at the 
> start of packet and rx timestamps at end of packet. This is the scheme 
> used by the reference NTP implementation, and I suspect also the one 
> which interoperates best with vanilla switches and routers.
> 
> My main goal in this discussion was to get some specific scheme in the 
> spec so that as implementations get more precise, they will continue to 
> interoperate.  The notes propose a single scheme (hardware) that 
> implementations should use for interoperability.  This meets my original 
> goal and I'm ok with this choice on the grounds that it is simple.
> 
> An unexpected outcome in the discussion was that the simple choice that 
> 1588 made was not optimal for some cases without on-path support.  The 
> software scheme, appears to sometimes offer better accuracy.  
> Unfortunately, it's also more complex and different.  I'm torn between 
> the simple hardware scheme and a slightly weird software scheme which 
> should work better with existing implementations and on paths without 
> on-path support.  (Note that this is not a question of what measurement 
> means are available, because the implementation can transpose from one 
> scheme to another.)
> 
> As I said before, I can live with the choice of the hardware scheme 
> (SOPtx, SOPrx), but am curious as to how the group feels about the 
> software scheme (SOPtx, EOPrx).
> 
> Regards,
> 
> Stuart
> 
> 
> ------------------------------------------------------------------------
> 

-- 
blu

There are two rules in life:
Rule 1- Don't tell people everything you know
----------------------------------------------------------------------
Brian Utterback - Solaris RPE, Sun Microsystems, Inc.
Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom
_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: Danny Mayer <[email-masked]>
Date: Fri, 22 Feb 2008 13:15:30 -0500

> driver writers and they seemed to think that it was not practical.

Contact Poul Henning-Kemp or Frank Kardel who did this for FreeBSD and 
NetBSD respectively.

Danny

---

From: "M. Warner Losh" <[email-masked]>
Date: Fri, 22 Feb 2008 11:13:37 -0700 (MST)

In message: <[email-masked]>
            Brian Utterback <[email-masked]> writes:
: I seem to recall that there is an implementation in FreeBSD to
: get the transmit interrupt timestamps. Is anybody here familiar
: with how that was implemented? I talked to some Solaris interface
: driver writers and they seemed to think that it was not practical.

It all depends on your hardware and what performance hit you want to
take with the newer, more sophisticated hardware.

The way it is implemented in FreeBSD is geared more towards older,
simpler hardware.  The time of the interrupt is recorded on the theory
that it will be more or less constant between when the packet arrives
and when the interrupt happens.  This works well on old hardware, but
newer hardware falls down badly since interrupts are deferred,
offloading happens, etc.

Warner

: STUART VENTERS wrote:
: > Kurt:
: > 
: > Thanks for the support.  In the notes Dave published, checkout the 
: > section titled 'Errors Due to Interworking Between Software, Hardware, 
: > and Driver Schemes' and see if it addresses your test case. 
: > 
: > _http://www.eecis.udel.edu/~mills/timestamp.txt_
: > 
: > 
: > As you can see, Dave and I are now pretty much in agreement with the 
: > analysis.  Hopefully it can morph into an appendix to the NTP spec.
: > 
: > The notes describe a variety of timestamping schemes and shows how they 
: > interoperate.  The simplest scheme (called hardware) uses timestamps at 
: > the start of packet.  This is the scheme the 1588 spec and hardware 
: > support.  Another scheme (called software) uses tx timestamps at the 
: > start of packet and rx timestamps at end of packet. This is the scheme 
: > used by the reference NTP implementation, and I suspect also the one 
: > which interoperates best with vanilla switches and routers.
: > 
: > My main goal in this discussion was to get some specific scheme in the 
: > spec so that as implementations get more precise, they will continue to 
: > interoperate.  The notes propose a single scheme (hardware) that 
: > implementations should use for interoperability.  This meets my original 
: > goal and I'm ok with this choice on the grounds that it is simple.
: > 
: > An unexpected outcome in the discussion was that the simple choice that 
: > 1588 made was not optimal for some cases without on-path support.  The 
: > software scheme, appears to sometimes offer better accuracy.  
: > Unfortunately, it's also more complex and different.  I'm torn between 
: > the simple hardware scheme and a slightly weird software scheme which 
: > should work better with existing implementations and on paths without 
: > on-path support.  (Note that this is not a question of what measurement 
: > means are available, because the implementation can transpose from one 
: > scheme to another.)
: > 
: > As I said before, I can live with the choice of the hardware scheme 
: > (SOPtx, SOPrx), but am curious as to how the group feels about the 
: > software scheme (SOPtx, EOPrx).
: > 
: > Regards,
: > 
: > Stuart
: > 
: > 
: > It looks like the analysis Dave published 21st,
: > 
: > Kurt Roeckx wrote:
: >  > On Thu, Feb 07, 2008 at 05:34:08PM +0000, David L. Mills wrote:
: >  >> Stuart,
: >  >>
: >  >> No, compare your equation with mine. The view has to be from A, so the
: >  >> delay cancels.
: >  >
: >  >
: >  > I'm still not convinced.  So I'll try to make an example.  Let's assume
: >  > that it takes 2 second for the packet to travel from A to B and the
: >  > other way around.  That is, if A sends a physical signal from low to
: >  > high, B will see that change 2 seconds later.  Let's also assume that it
: >  > takes A 1 second to transmit the packet.  Then let's assume that A's and
: >  > B's clock are in sync, and that A takes the timestamp at the start of
: >  > the packet, and B and the end.
: >  >
: >  > We start with A sending a packet at time 0.0.  A takes the time
: >  > when the packets starts to be transmitted, so sets T1 to 0.0.
: >  >
: >  > The packet will start to arrive at B at 2.0, and stop at
: >  > 3.0.  B takes the timestamp at the end of the packet
: >  > and will set T2 to 3.0.
: >  >
: >  > It takes B 1 second to process the packet.  It will start to transmit
: >  > the packet at 4.0.  But since it timestamps the packets after
: >  > it's completly send, it will set T3 1 second later to 5.0.  B will 
: > inform
: >  > A about T3 with an other packet it will send later.
: >  >
: >  > The packet starts to arrive at A at 6.0, which is when A takes
: >  > the timestamp, getting us T4 = 6.0.  The packet completly arrives at
: >  > 7.0, but that doesn't matter.
: >  >
: >  > This gives:
: >  > offset = 1/2 * [(T2-T1) + (T3-T4)] = 1/2 * [(3-0)+(5-6)] = 1.
: >  >
: >  > While we expected the result to be 0.
: >  >
: >  > Note that for the delay we don't have the problem.
: >  > delay = (T4-T1) - (T3-T2) = (6-0) - (5-3) = 4.
: >  >
: >  >
: >  > Kurt
: > 
: > 
: > ------------------------------------------------------------------------
: > 
: > _______________________________________________
: > ntpwg mailing list
: > [email-masked]
: > https://lists.ntp.org/mailman/listinfo/ntpwg
: 
: -- 
: blu
: 
: There are two rules in life:
: Rule 1- Don't tell people everything you know
: ----------------------------------------------------------------------
: Brian Utterback - Solaris RPE, Sun Microsystems, Inc.
: Ph:877-259-7345, Em:brian.utterback-at-ess-you-enn-dot-kom
: _______________________________________________
: ntpwg mailing list
: [email-masked]
: https://lists.ntp.org/mailman/listinfo/ntpwg
: 
: 
_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: Brian Utterback <[email-masked]>
Date: Fri, 22 Feb 2008 13:23:13 -0500

I meant the *transmit* interrupt. How do you get the timestamp
back to ntp and how does ntp relate the timestamp to the packet?

-- 
blu

---

From: "Greg Dowd" <[email-masked]>
Date: Fri, 22 Feb 2008 11:51:08 -0800

What I did in linux was extend the socket buffer to add a tx timestamp
and then used ioctl to query.  It's pretty lame as there is no
correlation other than socket, just last tx.  Best way would be to
reflect the pkt back into the receive queue with that timestamp.  I've
played with this in my ethernet driver but I don't know how to make the
kernel do it automagically in linux.

Greg Dowd
gdowd at symmetricom dot com (antispam format)
Symmetricom, Inc.
www.symmetricom.com
"Everything should be made as simple as possible, but no simpler" Albert
Einstein

-----Original Message-----
From: ntpwg-bounces+gdowd=[email-masked]
[mailto:ntpwg-bounces+gdowd=[email-masked]] On Behalf Of
Brian Utterback
Sent: Friday, February 22, 2008 10:23 AM
To: M. Warner Losh
Cc: [email-masked]; [email-masked]
Subject: Re: [ntpwg] ntpwg Updated NTPv4 Protocol
Specification/timestamp location

I meant the *transmit* interrupt. How do you get the timestamp back to
ntp and how does ntp relate the timestamp to the packet?

> : > The notes describe a variety of timestamping schemes and shows how

> they
> : > My main goal in this discussion was to get some specific scheme in

> the
> : > An unexpected outcome in the discussion was that the simple choice

> that
> : > 1588 made was not optimal for some cases without on-path support.

> The
> : > means are available, because the implementation can transpose from

> one
> : >  > that it takes 2 second for the packet to travel from A to B and

> the

--
blu

---

From: "David L. Mills" <[email-masked]>
Date: Fri, 22 Feb 2008 21:01:37 +0000

Brian,

The link you quote is broken. The _ at the beginning and end should be 
removed.

Dave

---

From: "David L. Mills" <[email-masked]>
Date: Fri, 22 Feb 2008 21:24:00 +0000

Guys,

Stuart and I went around this issue at some length. Yes, even the 
experiments I did with old and new NICs show up to 100 us unaccounted 
for. Certainly, 1588 hardware snaps the stamp just right. However, the 
experiments involved very little traffic other than NTP, so the 
nonreciprocal delay must be due to someting other than buffer queueing. 
I'm looking for something like the old TCP trick I once implemented to 
avoid ick-ack. The trick with interactive TELNET keystokes is to wait a 
little bit after the end of a received packet before sending the ACK. 
This increases the chance that a fast echo can be piggybacked in the ACK 
packet. Maybe instead of ick-ack we have ick-nic?

The FreeBSD scheme with older hardware is pretty darn good and might get 
better if we can snap an output stamp. We agreed that it is not likely 
that driver and NIC firmware will be willingly changed to support more 
accurate stamps as long as expensive 1588 NICs are available.

I suggested a sneaky carrot that could be implemented for applications 
other than just NTP. Call it expedited delivery, or priority queueing by 
any other name. The maximum delay of a 16-K buffer at 100 Mbs is about 
1.3 ms, which sounds kinda long for some things. If the driver and/or 
firmware can be buggered to put a tagged frame first on the FIFO, that 
might make several applications, including NTP very happy.

Dave

---

From: "David L. Mills" <[email-masked]>
Date: Fri, 22 Feb 2008 21:48:16 +0000

Greg,

Just getting a timestamp on the last packet sent might not be enough, as 
there may be several packets queued behind it. There needs to be a way 
for ntpd to associate the stamp with the relevant association. There 
probably also need a way for the output driver to recognize what packets 
should be stamped.

Excuse me while I dust off 30 years from my first TCP implementation. 
For historic and hilarious reasons we had something called rubber-EOL, 
since then wisely deprecated, and something called the urgent pointer. 
Reason was to provide an interrupt capability something like the 
red-ball/green-ball scheme implemented in TYMNET. I implemented it in 
the Fuzzballs to avoid the nasty delay between jamming the BREAK key and 
waiting for several second before the data in the pipe emptied. I don't 
think that has been deprecated, even if the need for it is probably not 
still with us.

I drop the other foot. Did I read somewhere that the Unix I/O system 
still hasthe  ablity to read out-of-band (OOB) data? That would be 
necessary to volley the balls. If such still be the case, why not have 
the output driver snatch the pseudo header of the last transmitted UDP 
packet, stick it and timestmap in a buffer and shoot up the OOB 
function? There still needs to be a socket bit to enable the function.

Dave

Greg Dowd wrote:

>> : > analysis. Hopefully it can morph into an appendix to the NTP
>
> spec.
>
>> : >

---

From: Danny Mayer <[email-masked]>
Date: Sat, 23 Feb 2008 21:22:57 -0500

> analysis.  Hopefully it can morph into an appendix to the NTP spec.

At this point in time (:)) I hope not. At best it should be a reference 
in the references section. Revisiting the NTPv4 protocol draft would 
require reviewing carefully the text and discussing it in the WG in depth.

Danny

---

From: "David L. Mills" <[email-masked]>
Date: Sun, 24 Feb 2008 06:34:11 +0000

Danny,

I couldn't agree more strongly. Get this puppy out the door and save the 
timestamp bickering for later.

Dave

---

From: "STUART VENTERS" <[email-masked]>
Date: Mon, 25 Feb 2008 14:50:04 -0600

Danny,
Dave,

Considering where we are, shipping the puppy is understandable.

The consequence of not having the timestamp location in the spec is:
  Interoperable time transfer accuracy is limited to 2 packet times on the slowest interconnecting link.
  Frequency transport is still at the full accuracy of the implementations and packet path.

This appears to be ok with the majority of current implementations.

Interoperability for new applications may be better served by waiting
  until the group has the time to carefully consider the matter
    in a future spec.

The current version of the spec is draft-8 which I assume is the puppy in question.

Since there doesn't appear to be much talk about draft-8 on the reflector,
   and assuming there are no squeaks about the 2 packet limit,
   does that say you have about completed a successful last call for draft-8?

Regards,

Stuart

David Mills wrote:
>>Stuart Venters wrote:
>>> Hopefully it can morph into an appendix to the NTP spec.
>>>