Subject: Re: [ntpwg] ntpwg Updated NTPv4 Protocol Specification/timestamp location

From: "STUART VENTERS" <[email-masked]>
Date: Fri, 1 Feb 2008 16:49:42 -0600

Dave,

My apologies for not responding sooner to your Jan 23 E-mail, I got tied up.
The delay was good in the it has given me time to consider what you said.

Thank you for the further clarification as to the constraints of a S/W implementation of NTP on unix.  I had suspected as much from things done in an earlier life, but it is nice to hear from the person who best knows.  I agree with you that even with heroic efforts, it is likely impossible to implement an exact capture point in the environment where NTP mostly runs now.  That said, making a spec which requires doing so is futile.

Although I haven't had experience building NTP in S/W, I have experimented with it in H/W.  Many of the S/W constraints go away, and without heroic efforts I can strike timestamps with maybe 40ns accuracy.   The residual errors are due mostly to clock domain crossings between the Ethernet Rx and Tx clocks and the clock I use for timekeeping.  With more work these could be reduced but there will always be some residual error.  Even with H/W, making a spec which requires exact accuracy is futile.

So why do I still think that things that can't be controlled in the implementation need to be in the spec?

Stepping back, the NTP spec is strange compared to most IETF protocol specs.  Most protocol specs provide specific rules for communicating.  Close doesn't count, things have to be exact.  You wouldn't expect to see a spec that says use approximately this value for a message type or checksum.  As demonstrated above, timekeeping is different.  Close not only counts, it's all we can do.  But to get close, we need to match the reciprocal delays.  To do this with multiple implementations, we need to agree on CP1 and CP2.  The CP1/CP2 definitions are needed because without them, there are multiple incompatible ways to achieve matched reciprocal delays.

Perhaps a way out is to provide exact definitions for CP1 and CP2 as goals for the implementer.  Then provide profiles stating how close the implementer has to get to the goals.

Regards,

Stuart

ps, I wonder if there is some fpga eval card which would be a better deal than a $600 special 1588 nic.

pps, As for your other shoe, perhaps it should stay on.  It's not clear that the two step stuff is necessary and it's clearly not desirable to change the protocol.  If just before calculating the message digest, the transmit routine could look into the transmit system and predict when the packet would go out then it would be unnecessary.  If I remember correctly, the most popular communications processors used to provide enough information to do this.  Namely, a buffer descriptor list of packets yet to be sent and a current buffer pointer for the one being sent.  Seems like converting these to the approximate number of Ethernet clocks before the NTP packet goes out would be doable.  I expect that any 1588 implementation which needs the two step correction also needs special 1588 hardware for striking timestamps.  Maybe the 1588 hardware just needs to provide this prediction capability.  (Hopefully the 1588 H/W folks are listening.)  Of course this only works if the 1588 hardw
 are is tightly integrated with the Etherent transmitter.  If the hardware is just watching the packets go by at the transmitter output, then maybe we see your sock.  (So, what's the url for the architecture briefing?)

ppps, There's another thing different about the NTP spec.  One technique for defining a protocol is to detail a sample implementation and then say that the protocol is what the implementation does.  It seems like the NTP spec does this a bit more than usual.  This is great as long as we remember we are making a protocol spec.

_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: "David L. Mills" <[email-masked]>
Date: Sun, 03 Feb 2008 04:53:34 +0000

Stuart,

I hear you, but we are not in agreement. I think we agree that NTP does 
not synchronize the clock to prevailing wire time; it synchronizes to a 
server clock by minimizing the computed offset relative to that clock. I 
think we agree that the only thing that matters is the reciprocal delays 
and the delay variations along the paths. Therefore, a definitive 
statement on the achievable accuracy depends only on the rigor with 
which this can be achieved.

I submit this has nothing to do with the formal NTP specification. 
Whether timestamps are scarfed on the wire, in the drivers or even as 
NTP does now is a matter of design, implementation and software/hardware 
capabilities. A clear statement about this should be in the spec along 
with plausible scenarios such as I summarized in a previous message.

I would hope the TICTOC WG would take this issue up and consider the 
proposition that, should wire timetamps such as captured in 1588 
hardware be available and the two-step NTP on-wire protocol be adopted, 
NTP accuracy could rival 1588. There are lots of wherefores and whyfores 
in this agenda and it is not intendded to compete with current 1588 
development, but it is an issue for research and experimentation.

Dave

STUART VENTERS wrote:

> Thank you for the further clarification as to the constraints of a S/W 
> implementation of NTP on unix. I had suspected as much from things 
> done in an earlier life, but it is nice to hear from the person who 
> best knows. I agree with you that even with heroic efforts, it is 
> likely impossible to implement an exact capture point in the 
> environment where NTP mostly runs now. That said, making a spec which 
> requires doing so is futile.
>
> Although I haven't had experience building NTP in S/W, I have 
> experimented with it in H/W. Many of the S/W constraints go away, and 
> without heroic efforts I can strike timestamps with maybe 40ns 
> accuracy. The residual errors are due mostly to clock domain crossings 
> between the Ethernet Rx and Tx clocks and the clock I use for 
> timekeeping. With more work these could be reduced but there will 
> always be some residual error. Even with H/W, making a spec which 
> requires exact accuracy is futile.
>
> Stepping back, the NTP spec is strange compared to most IETF protocol 
> specs. Most protocol specs provide specific rules for communicating. 
> Close doesn't count, things have to be exact. You wouldn't expect to 
> see a spec that says use approximately this value for a message type 
> or checksum. As demonstrated above, timekeeping is different. Close 
> not only counts, it's all we can do. But to get close, we need to 
> match the reciprocal delays. To do this with multiple implementations, 
> we need to agree on CP1 and CP2. The CP1/CP2 definitions are needed 
> because without them, there are multiple incompatible ways to achieve 
> matched reciprocal delays.
>
> Perhaps a way out is to provide exact definitions for CP1 and CP2 as 
> goals for the implementer. Then provide profiles stating how close the 
> implementer has to get to the goals.
>
> Regards,
>
> Stuart
>
>
>
> pps, As for your other shoe, perhaps it should stay on. It's not clear 
> that the two step stuff is necessary and it's clearly not desirable to 
> change the protocol. If just before calculating the message digest, 
> the transmit routine could look into the transmit system and predict 
> when the packet would go out then it would be unnecessary. If I 
> remember correctly, the most popular communications processors used to 
> provide enough information to do this. Namely, a buffer descriptor 
> list of packets yet to be sent and a current buffer pointer for the 
> one being sent. Seems like converting these to the approximate number 
> of Ethernet clocks before the NTP packet goes out would be doable. I 
> expect that any 1588 implementation which needs the two step 
> correction also needs special 1588 hardware for striking timestamps. 
> Maybe the 1588 hardware just needs to provide this prediction 
> capability. (Hopefully the 1588 H/W folks are listening.) Of course 
> this only works if the 1588 hardware is tightly integrated with the 
> Etherent transmitter. If the hardware is just watching the packets go 
> by at the transmitter output, then maybe we see your sock. (So, what's 
> the url for the architecture briefing?)
>
> ppps, There's another thing different about the NTP spec. One 
> technique for defining a protocol is to detail a sample implementation 
> and then say that the protocol is what the implementation does. It 
> seems like the NTP spec does this a bit more than usual. This is great 
> as long as we remember we are making a protocol spec.
>
>
>
>
>

---

From: "TS Glassey" <[email-masked]>
Date: Sun, 3 Feb 2008 08:32:55 -0800

This may be my fault... let me possibly spin a justification for what the 
group is trying to do.

Perhaps the issue here is that we have all forgotten that most SW models 
were built as emulations - so a SW version of the NTP system, is an 
emulation of a HW based appliance or service. That said there is a 
reasonable argument for assigning a particular point in time (or rather a 
point of the structure of the NTP Packet itself, which would be considered 
that point in time.

I think that the problem is that unlike 1588, the NTP process depends on 
some stack receiving and framing the data prior to its actually being 
reliable data. Since there are no precision timer/receipt processes 
available for logging in TCP, UDP,  or the NTP process to address this it 
becomes a real issue in that the data could sit in the stack for any amount 
of time...

However. In rethinking mo commentary I would now agree that tools like the 
TCP accelerator engines which reduce TCP and UDP communications to more 
appliance style communications in and of themselves demonstrate  a reason 
for allowing this event.

But perhaps the real issue isn't whether its a part of the packet frame that 
constitutes the physical "its now" marker  but rather when the 'device' (a 
Network Agent) actually constitutes the receipt point since nothing can be 
done by the stack until it passes those datum (no pun intended) to the NTP 
process or TOD services of the Host OS.

Does this make sense?

Todd

----- Original Message ----- 
From: "David L. Mills" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, February 02, 2008 8:53 PM
Subject: Re: [ntpwg] ntpwg Updated NTPv4 Protocol 
Specification/timestamplocation

---

From: Kurt Roeckx <[email-masked]>
Date: Sun, 3 Feb 2008 18:27:24 +0100

> becomes a real issue in that the data could sit in the stack for any amount 

You can ask the kernel when it received a packet.  This is currently
mostly done in the kernel itself without any help from hardware.  Ntpd
is using this for a long time now.  Complete software based PTP
does the same thing and has exactly the same problem as ntpd.

What could be changed to get better results is:
- Have more hardware support so that you can capture the time
  in the PHY for both send and receive of a packet;
- Have the kernel export that to user space;
- Change ntpd so that is can send a "Follow Up" message that
  contains the real time of when the previous packet was send
  as captured by the PHY.

But those are all just current implementation details.  This is not
a limitation of the NTP protocol.

Kurt

---

From: "David L. Mills" <[email-masked]>
Date: Sun, 03 Feb 2008 19:57:50 +0000

TOd,

I beg to differ.

You speak of NTP as a software emulation of a hardware device. You might 
have missed a point stressed in previous messages. While the 1588 
on-wire protocol is different than NTP, both use the same t1, t2, t3 and 
t4 timestamps and the same offset and delay calculations. The 1588 
calculations are done in the software driver or even by a utility 
program interfaced via USB(!) The actual math could in principle be done 
entirely using hardware chips, but that wouldnt really change anything.

The only real difference between 1588 and NTP is that 1588 defines a 
capture point on the media itself. If you stand back and look at it, 
this doesn't really matter for the same reasons as in NTP. In both 
protocols the object is to synchronize a slave clock in a kernel or NIC 
to a master clock in another machine, so the same arguments apply. The 
1588 capture point is specifically chosen to minimize jitter and so 
intervening switches can compensate for queueing delays. I don't think 
this detail belongs in the NTPv4 spec.

By intent, 1588 does not specify the nature of the feedback loop used to 
discipline the slave clock, but NTP does for the reasons discussed 
previously on this list and in the spec. The loop does not emulate 
hardware; it emulates an equation most often studied in control theory 
and digital signal processing textbooks. It is possible to implement 
these equations in hardware, software or a combination of both. DSP 
chips on the market today use both, as does your cellphone. NTP doesn't 
need hardware assist.

Dave

TS Glassey wrote:

> This may be my fault... let me possibly spin a justification for what 
> the group is trying to do.
>
> Perhaps the issue here is that we have all forgotten that most SW 
> models were built as emulations - so a SW version of the NTP system, 
> is an emulation of a HW based appliance or service. That said there is 
> a reasonable argument for assigning a particular point in time (or 
> rather a point of the structure of the NTP Packet itself, which would 
> be considered that point in time.
>
> I think that the problem is that unlike 1588, the NTP process depends 
> on some stack receiving and framing the data prior to its actually 
> being reliable data. Since there are no precision timer/receipt 
> processes available for logging in TCP, UDP,  or the NTP process to 
> address this it becomes a real issue in that the data could sit in the 
> stack for any amount of time...
>
> However. In rethinking mo commentary I would now agree that tools like 
> the TCP accelerator engines which reduce TCP and UDP communications to 
> more appliance style communications in and of themselves demonstrate  
> a reason for allowing this event.
>
> But perhaps the real issue isn't whether its a part of the packet 
> frame that constitutes the physical "its now" marker  but rather when 
> the 'device' (a Network Agent) actually constitutes the receipt point 
> since nothing can be done by the stack until it passes those datum (no 
> pun intended) to the NTP process or TOD services of the Host OS.
>
> Does this make sense?
>
> Todd
>
> ----- Original Message ----- From: "David L. Mills" <[email-masked]>
>> > snip

---

From: "David L. Mills" <[email-masked]>
Date: Sun, 03 Feb 2008 20:04:44 +0000

Kurt,

Exactly. A proposal for a follow-up (two-step) function is in the 
architecture briefing on the NTP project page. The intent is that use of 
the original and two-step mode be automatically detected on-fly.

Dave

---

From: "TS Glassey" <[email-masked]>
Date: Sun, 3 Feb 2008 12:43:14 -0800

----- Original Message ----- 
From: "David L. Mills" <[email-masked]>
To: <[email-masked]>
Sent: Sunday, February 03, 2008 11:57 AM
Subject: Re: [ntpwg] ntpwg Updated NTPv4 
ProtocolSpecification/timestamplocation

> have missed a point stressed in previous messages.

No... I cought the point.

>  While the 1588
> t4 timestamps and the same offset and delay calculations.

But the question is one of how they use the same timestamps.  NTP is 
desinged to run in the OS context of some server. There are in fact NTP 
appliances, but they also suffer the same issues with the stack framing 
overhead and packet reception.

>  The 1588
> program interfaced via USB(!)

Depends whether they are executed by a FPGA or by an OS... Your push back is 
specific to one very particular type of 1588 device & there are others which 
are more appliance based.

The same can be said of NTP boxes from those embedded type systems which run 
on an integrated service. Others that are NTP Server's running on an OS 
context are 'emulators' by the very definition.

> The actual math could in principle be done

But doing the math in HW and the Network Interface in HW would 
significantly.

> this doesn't really matter for the same reasons as in NTP.

OK OK - but then the reality is that we need a performance specification for 
a minimum NTP or 1588 performance model and then in that case the frame 
point on the packet becomes relevent.

> In both

---

From: "STUART VENTERS" <[email-masked]>
Date: Mon, 4 Feb 2008 11:23:05 -0600

Dave,

Life is pretty good, we are generally in agreement on the 1st and 3rd paragraph and the 2nd may have gone from futile to only un-necessary.

Let me try a scenario based on my understanding of your second paragraph and see where we end up.

Say the NTP spec turns out like you want and an implementer (named Herbie Worker) takes it and builds an NTP client and server.  He's building in an fpga and decides to strike timestamps when the start of each packet enters and exits his host.  (IE scarfed on the wire at SOP.)  He's careful and his timestamps have variances on say 40ns.  Furthermore, he decides to use the same timestamping design at both the client and server.  This makes guaranteeing the reciprocal match automatic, since each direction passes a packet through identical time strikers.  He gets it all running and installs test wires on the time counters at the client and server.  From the test points, he notes that on a point to point link the client is time aligned to the server within 40ns.  Herbie has successfully built a rigorous NTP implementation and he is a happy camper.  Marketing looks at the tests and decides to sell an NTP implementation with an advertised accuracy of 40ns.

Another implementer (named Sue Worker) also decides to build an NTP client and server from the spec.  She's doing it in software, but has a dedicated fast processor and after thinking about it decides to strike timestamps as close to the hardware as she can which is in her drivers.  (IE scarfed in the drivers.)  She's pretty sharp and figures out that with a wire between the RX interrupt and her Hardware timers, she can timestamp incoming packets to within 100ns.  She also figures out that by waiting until the transmitter is quiet before starting a packet, she can control the timestamp outgoing packets to within 100ns.  She also has testpoints and her client is aligned to her server within 100ns.  Sue has successfully built a rigorous NTP implementation an we have another happy camper.  Marketing looks at the tests and decides to sell an NTP implementation with an advertised accuracy of 100ns.

A third guy (named Pesky User) decides to do some time keeping and builds a system using Herbie's server and Sue's client.  He does so and notices a time offset of a few microseconds.  This seems contrary to the specs, what's wrong?  One answer is that Pesky, by plugging together 2 stunning NTP implementations, essentially constructed a third NTP implementation, but without enough rigor to ensure that the reciprocal delays match.  

Pesky finds this answer unsatisfying, (after all, he's a user not an implementer), and so the IETF reopens the NTP spec.  The new spec affirms Herbie's choice and Sue adds a minor compensation factor to bring her implementation into compliance.  Now Pesky can plug different implementations together at will.  Everybody is happy except for the users with islands of Sue's original implementation.

Ideally there is something wrong with my scenario or in my understanding of what you are saying.  I'd be perfectly happy to be wrong if there is a simpler way to make a complete spec.  

If not, then IMHO, the NTP spec still needs tell Herbie and Sue that they have to assure a reciprocal delay match. Since there are multiple non-interoperable ways of doing so (for example the 2 above), it also needs to give them a clue as to which one to use.  I don't pretend to know the best way to do this, but nailing down CP1 and CP2 is a possibility.

Regards,

Stuart

ps, It's possible that the reason we are here is that this is the first time that it's become reasonable to implement NTP to a degree of accuracy where this sort of precision in the spec matters.

pps,  It seems to me that NTP may already be superior to 1588 in precision timekeeping applications where the packets have to go through vanilla packet switches.  It this case more measurement packets means better odds of getting a lucky one.  The followup packets are a hindrance to more measurement packets.  (Unless the followup packets are also measurement packets, or you build a 1588 implementation which doesn't need/use followup packets.)

---

From: "David L. Mills" <[email-masked]>
Date: Tue, 05 Feb 2008 04:11:36 +0000

Stuart,

All three citizens are told only to keep the reciprocal delays as close 
as possible and minimize incidendal jitter. They don't see any capture 
point in the specification.

Media Herbie has no problem since his NIC strikes a transmit timestamp 
only if the carrier is down and the preamble is not jammed. He knows the 
packet he stamped got successfully out the door. Driver Sue has to be 
careful to strike the timestamps at NIC interrupt time; that is, just 
before the checksum for the transmit side and just after the checksum on 
the receive side. This makes sure that, even if the frame is 
retransmitted the strikes are true. If she is clever, she will know the 
line speed and add in 320 ns (at 100 Mb) on the transmit side to account 
for the 32-bit checksum.

Now, on the fastest machine I have it takes about 500 ns to read the 
system clock and the only thing she knows is that the true time is 
somewhere in the middle. Sue will need something better, like the PCC or 
TSC. THe real trick is to pass these timestamps upstack to the protocol 
machine, whether in FPGA (only for sissies), embedded RISC CPU on-chip 
(Intel) (for real programmers) or dreary ntpd.

Notice that the boys and girls can interoperate in a truly gender 
independent way.

So, where;s the beef?

Dave

STUART VENTERS wrote:

> Dave,
>
>
> Life is pretty good, we are generally in agreement on the 1st and 3rd 
> paragraph and the 2nd may have gone from futile to only un-necessary.
>
> Say the NTP spec turns out like you want and an implementer (named 
> Herbie Worker) takes it and builds an NTP client and server. He's 
> building in an fpga and decides to strike timestamps when the start of 
> each packet enters and exits his host. (IE scarfed on the wire at 
> SOP.) He's careful and his timestamps have variances on say 40ns. 
> Furthermore, he decides to use the same timestamping design at both 
> the client and server. This makes guaranteeing the reciprocal match 
> automatic, since each direction passes a packet through identical time 
> strikers. He gets it all running and installs test wires on the time 
> counters at the client and server. From the test points, he notes that 
> on a point to point link the client is time aligned to the server 
> within 40ns. Herbie has successfully built a rigorous NTP 
> implementation and he is a happy camper. Marketing looks at the tests 
> and decides to sell an NTP implementation with an advertised accuracy 
> of 40ns.
>
> Another implementer (named Sue Worker) also decides to build an NTP 
> client and server from the spec. She's doing it in software, but has a 
> dedicated fast processor and after thinking about it decides to strike 
> timestamps as close to the hardware as she can which is in her 
> drivers. (IE scarfed in the drivers.) She's pretty sharp and figures 
> out that with a wire between the RX interrupt and her Hardware timers, 
> she can timestamp incoming packets to within 100ns. She also figures 
> out that by waiting until the transmitter is quiet before starting a 
> packet, she can control the timestamp outgoing packets to within 
> 100ns. She also has testpoints and her client is aligned to her server 
> within 100ns. Sue has successfully built a rigorous NTP implementation 
> an we have another happy camper. Marketing looks at the tests and 
> decides to sell an NTP implementation with an advertised accuracy of 
> 100ns.
>
> A third guy (named Pesky User) decides to do some time keeping and 
> builds a system using Herbie's server and Sue's client. He does so and 
> notices a time offset of a few microseconds. This seems contrary to 
> the specs, what's wrong? One answer is that Pesky, by plugging 
> together 2 stunning NTP implementations, essentially constructed a 
> third NTP implementation, but without enough rigor to ensure that the 
> reciprocal delays match.
>
> Pesky finds this answer unsatisfying, (after all, he's a user not an 
> implementer), and so the IETF reopens the NTP spec. The new spec 
> affirms Herbie's choice and Sue adds a minor compensation factor to 
> bring her implementation into compliance. Now Pesky can plug different 
> implementations together at will. Everybody is happy except for the 
> users with islands of Sue's original implementation.
>
>
> Ideally there is something wrong with my scenario or in my 
> understanding of what you are saying. I'd be perfectly happy to be 
> wrong if there is a simpler way to make a complete spec.
>
> If not, then IMHO, the NTP spec still needs tell Herbie and Sue that 
> they have to assure a reciprocal delay match. Since there are multiple 
> non-interoperable ways of doing so (for example the 2 above), it also 
> needs to give them a clue as to which one to use. I don't pretend to 
> know the best way to do this, but nailing down CP1 and CP2 is a 
> possibility.
>
>
> Regards,
>
> Stuart
>
>
>
> ps, It's possible that the reason we are here is that this is the 
> first time that it's become reasonable to implement NTP to a degree of 
> accuracy where this sort of precision in the spec matters.
>
> pps, It seems to me that NTP may already be superior to 1588 in 
> precision timekeeping applications where the packets have to go 
> through vanilla packet switches. It this case more measurement packets 
> means better odds of getting a lucky one. The followup packets are a 
> hindrance to more measurement packets. (Unless the followup packets 
> are also measurement packets, or you build a 1588 implementation which 
> doesn't need/use followup packets.)
>