Subject: [Ntp] The bump, or why NTP v5 must specify impulse response

From: Watson Ladd <[email-masked]>
Date: Sun, 12 Apr 2020 15:11:31 -0700

Dear ntp WG,

Section 14.5 of Phase Lock Techniques by Floyd Gardner describes
difficult problems caused by the accumulation of errors in a chain of
PLLs under benign conditions, and section 2.2.4 describes the root
cause, namely an inevitable peak in the transfer function of a second
order filter. I don't think these are insolvable in NTP v5, but they
should give pause to the idea that we can avoid needing to specify the
the synchronization algorithm. The peaking of the PLL needs to be
controlled, or some thing more complex needs to be specified.

Sincerely,
Watson Ladd

---

From: Daniel Franke <[email-masked]>
Date: Mon, 13 Apr 2020 10:21:40 -0400

> controlled, or some thing more complex needs to be specified.

Are the algorithms in RFC 5905 vulnerable to this? If so, I think that
shoots down any implication that it's a practical necessity to solve
this in the NTPv5 spec, since we've obviously been getting by okay so
far.

Just to be clear, though, I'm not proposing that the WG should be
*silent* on synchronization algorithms in v5. Just that whatever we
have to say about it should be in a separate, Informational document,
and can discuss the design space and suggest multiple possibilities
rather than require one particular thing.

---

From: Watson Ladd <[email-masked]>
Date: Mon, 13 Apr 2020 12:15:27 -0700

> this in the NTPv5 spec, since we've obviously been getting by okay so

v4 limits the length of the chain and specifies the input response to have,
which solves the problem.

---

From: Daniel Franke <[email-masked]>
Date: Mon, 13 Apr 2020 15:24:38 -0400

I don't think I follow. In what sense does it limit the length of the
chain, other than that stratum is capped at 16? And what
characteristic of RFC 5905's recommended input response address the
problem? (I need to read the book you've referenced before I fully
understand the problem in the first place; I've ordered it but it'll
take some time to arrive).

On Mon, Apr 13, 2020 at 3:15 PM Watson Ladd <[email-masked]> wrote:
> v4 limits the length of the chain and specifies the input response to have, which solves the problem.
>
>>

---

From: Daniel Franke <[email-masked]>
Date: Mon, 13 Apr 2020 16:09:47 -0400

What motivates my skepticism here, by the way, is a belief that it's
possible to get much, much better performance with better
synchronization algorithms. A number of academic researchers have
experimented with Kalman filters [1][2][3] and gotten phenomenal
results, in the case of [1], two orders of magnitude better than stock
ntpd 4.2.0. I suspect that this could be even further improved with
more accurate network models (Kalman filters would be provably optimal
if network jitter were Gaussian, but we all know it isn't). I want to
avoid specifying any requirements that would discourage or obstruct
this sort of research from being developed into production
implementations.

[1] https://ieeexplore.ieee.org/abstract/document/4268082
[2] http://alumni.media.mit.edu/~aggelos/papers/tuffc_nov04.pdf
[3] https://www.sciencedirect.com/science/article/pii/S1474667015361358

---

From: Doug Arnold <[email-masked]>
Date: Mon, 13 Apr 2020 21:22:33 +0000

I think that the most important thing for the clock steering algorithm in ntpv5 is that it is modular.  Define an interface to the clock steering subsystem so that people can make servo loop filters specialized for a specific set of conditions.

If we define a default clock steering algorithm, the most important property is that it always converges, even if it is not the most accurate for any specific network conditions.  It probably has to be fairly simple to have this property.  Complex algorithms usually have weird corner cases.

As for Kalman filters, they work really well for things like GPS receives where the noise properties are relatively constant.  Then you can tune them for those noise properties.  Queuing noise in a network varies wildly, and is often exhibits a highly skewed client clock error measurement probability density with a peak near zero followed by a long tail in one direction.  I suspect that Kalman filters will perform poorly on highly skewed data unless you have a non-linear prefilter to make the noise probability density closer to gaussian.  I know of people who have gotten good results for PTP servo loops using Kalman filters, but with a generalized luckly packet prefilter to eliminate the long tail in the distribution, and with the prefilter also determining the Kalman filter parameters.  That is a complicated algorithm which probably has some catastrophic corner cases, so it wouldn't be a good candidate for a default servo loop which always works at least ok.

Doug

________________________________
From: ntp <[email-masked]> on behalf of Daniel Franke <[email-masked]>
Sent: Monday, April 13, 2020 1:09 PM
To: Watson Ladd <[email-masked]>
Cc: NTP WG <[email-masked]>
Subject: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

> >> rather than require one particular thing.

_______________________________________________
ntp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ntp

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 14 Apr 2020 12:35:43 +0200

> If we define a default clock steering algorithm, the most important property is that it always converges, even if it is not the most accurate for any specific network conditions.  It probably has t…

Simple algorithms may not be useful in real-world applications.
Complex algorithms are difficult to analyze and show that they always
work in given conditions. I'm not sure how much sense it makes for the
draft to have some general requirements.

I'd like to see an example of the issue with a long chain of PLLs. If
anyone has a code of a susceptible PLL, I can try to simulate a long
chain.

> As for Kalman filters, they work really well for things like GPS receives where the noise properties are relatively constant.  Then you can tune them for those noise properties.  Queuing noise in a…

In the context of NTP, which is supposed to synchronize computer
clocks, the clock may be an even bigger problem than the network.
Without a good model of the clock a Kalman filter (at least in the
basic form) cannot perform well. The problem is in random changes in
the frequency of the clock due to temperature changes (due to random
changes in the load of various components in the computer). A Kalman
filter that works well on an idle computer won't likely work well on a
moderately loaded server. It would need to include data from a
temperature probe. Making the loop adaptive is not that easy. In NTPv3
and NTPv4 it is done by adjusting the polling interval (to which is
tied the PLL/FLL time constant).

-- 
Miroslav Lichvar

---

From: "Dieter Sibold" <[email-masked]>
Date: Tue, 14 Apr 2020 13:19:11 +0200

I think the document which specifies the new NTP protocol should not 
consider the clock steering algorithm as Doug suggest. But we should 
start a second document in which we describing at least one possible 
steering algorithm. This might only be informational.

Dieter

On 13 Apr 2020, at 23:22, Doug Arnold wrote:

> I think that the most important thing for the clock steering algorithm 
> in ntpv5 is that it is modular.  Define an interface to the clock 
> steering subsystem so that people can make servo loop filters 
> specialized for a specific set of conditions.
>
> If we define a default clock steering algorithm, the most important 
> property is that it always converges, even if it is not the most 
> accurate for any specific network conditions.  It probably has to be 
> fairly simple to have this property.  Complex algorithms usually have 
> weird corner cases.
>
> As for Kalman filters, they work really well for things like GPS 
> receives where the noise properties are relatively constant.  Then you 
> can tune them for those noise properties.  Queuing noise in a network 
> varies wildly, and is often exhibits a highly skewed client clock 
> error measurement probability density with a peak near zero followed 
> by a long tail in one direction.  I suspect that Kalman filters will 
> perform poorly on highly skewed data unless you have a non-linear 
> prefilter to make the noise probability density closer to gaussian.  I 
> know of people who have gotten good results for PTP servo loops using 
> Kalman filters, but with a generalized luckly packet prefilter to 
> eliminate the long tail in the distribution, and with the prefilter 
> also determining the Kalman filter parameters.  That is a complicated 
> algorithm which probably has some catastrophic corner cases, so it 
> wouldn't be a good candidate for a default servo loop which always 
> works at least ok.
>
> Doug
>
> [3] 
> https://www.sciencedirect.com/science/article/pii/S1474667015361358
>

---

From: Ulrich Windl <[email-masked]>
Date: Tue, 14 Apr 2020 22:37:41 +0200

On 4/14/20 12:35 PM, Miroslav Lichvar wrote:

[...]
> clocks, the clock may be an even bigger problem than the network.

Yes, I also realized that our GPS-based startum-1 server basically 
measures room temperature when looking at the frequency correction.

But asymmetric network delays are also an important factor.
[...]

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 14 Apr 2020 21:05:58 +0000

HA!

But at least you don't have quantum effects if you change floors, right? :)

---

From: Warner Losh <[email-masked]>
Date: Tue, 14 Apr 2020 15:10:43 -0600

> But at least you don't have quantum effects if you change floors, right? :)

Only if you fly in planes or are moving. The Sagnac Effect is quite real
even at low speeds in a spinning environment.

Working at Timing Solutions years ago, we had to rope off an area around a
test system. You could see when a certain manager would lumber through the
lab in the data, which spoiled the calibration that was being done and it
had to start over... :)

But neither of these are likely effects large enough to matter to ntp.

Warner

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 14 Apr 2020 21:19:19 +0000

>But neither of these are likely effects large enough to matter to ntp.

NIST recalibrated because of gravity/general-relativity changes when their F1 clock changed floors (https://www.wired.com/2007/12/time-nist/)

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 14 Apr 2020 16:00:20 -0700

On 4/14/2020 2:19 PM, Salz, Rich wrote:
> *>*But neither of these are likely effects large enough to matter to ntp.
> 
>  
> 
> NIST recalibrated because of gravity/general-relativity changes when
> their F1 clock changed floors (https://www.wired.com/2007/12/time-nist/)

That's not really anywhere near the same thing.

-- 
Harlan Stenn <[email-masked]>
http://networktimefoundation.org - be a member!

---

From: Kyle Rose <[email-masked]>
Date: Tue, 14 Apr 2020 19:19:11 -0400

> That's not really anywhere near the same thing.

I think Rich was just continuing Ulrich's joke.

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 14 Apr 2020 16:57:21 -0700

On 4/14/2020 4:19 PM, Kyle Rose wrote:
> On Tue, Apr 14, 2020 at 7:00 PM Harlan Stenn <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
>     > their F1 clock changed floors
>     (https://www.wired.com/2007/12/time-nist/)
> 

OK, that would be good.  I honestly couldn't tell.

---

From: Harlan Stenn <[email-masked]>
Date: Wed, 15 Apr 2020 02:59:05 -0700

We had the same discussion at the transition from ntpv3 to ntpv4.

If you do this, please do not call it NTP.  Please find another group to
do this in, perhaps TICTOC.

H

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 15 Apr 2020 13:38:38 +0000

>    If you do this, please do not call it NTP. 

Can you explain why 3->4 could still be called NTP but 4->5 should not?

---

From: Harlan Stenn <[email-masked]>
Date: Thu, 16 Apr 2020 02:58:39 -0700

> Can you explain why 3->4 could still be called NTP but 4->5 should not?

You didn't quote enough context.

NTP is a combination of two parts:

- the protocol/packet structure
- algorithmic and response behavior

If you are only specifying the protocol in this new thing then it is
dangerous/misleading/irresponsible to call this v5 document "NTP".

It is only slightly less dangerous/misleading/irresponsible to call it
"NTPv5 Protocol", regardless of whether or not there is also an "NTPv5
Algorithms and Response Behavior".

---

From: Tony Finch <[email-masked]>
Date: Thu, 16 Apr 2020 14:32:12 +0100

> - algorithmic and response behavior

I got the impression from https://chrony.tuxfamily.org/comparison.html
that not all implementations use the same algorithms.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Hebrides: Variable 3 or less, becoming east or northeast, 3 to 5. Moderate,
occasionally rough at first. Fair. Good.

---

From: Harlan Stenn <[email-masked]>
Date: Thu, 16 Apr 2020 06:43:00 -0700

> that not all implementations use the same algorithms.

That's true.

Just to shortcut this discussion...

>From the NTP Project's POV, chronyd is *not* a conforming NTP
implementation, as it uses different algorithms and does not honor the
behavioral characteristics of the application of impulse changes.

---

From: Kurt Roeckx <[email-masked]>
Date: Thu, 16 Apr 2020 15:55:47 +0200

> behavioral characteristics of the application of impulse changes.

Does that mean you're saying that the algorithms can't be
improved?

Does that mean that if you support NTPv4 and NTPv5, and NTPv5 has a
different algorithm, and you talk to something that does NTPv4,
you should have a complete emulation of what an NTPv4 server
should have done, including serving a different time because
the impulse response is different?

Kurt

---

From: "Dieter Sibold" <[email-masked]>
Date: Thu, 16 Apr 2020 19:27:29 +0200

The problem with this combination is that we currently have no 
implementation that conforms to RFC5905 completely.
There are different use cases for potential clients. Some of them are 
not well served with the current algorithm other are. A modular approach 
would allow both: to apply an algorithm most suitable to the application 
and to be compliant with the protocol spec.

Dieter

---

From: Harlan Stenn <[email-masked]>
Date: Thu, 16 Apr 2020 14:12:54 -0700

> Does that mean you're saying that the algorithms can't be

Srsly?  No.

What is your definition of "improved"?

Does it cast the wide net that the current definitions and
specifications cover?

> the impulse response is different?

At least you're thinking about this.

There are cases with regulatory requirements here.

There are liability risks.

Then again, the WG may decide to punt all of this for V5.

Similar to what POSIX did around leap seconds.

---

From: Daniel Franke <[email-masked]>
Date: Thu, 16 Apr 2020 17:44:02 -0400

> There are cases with regulatory requirements here.

What regulations govern NTP's clock synchronization algorithm?

> There are liability risks.

Who risks being liable to whom for what?

---

From: [email-masked]
Date: Fri, 17 Apr 2020 12:20:56 +0200

Hey all,

I wanted to offer my point of view on the question of modularity for NTPv5 as discussed here.

As many of you may know, the place that I'm coming from is not decades of practical experience in NTP implementation and/or maintenance, but instead years of close academic and theoretic scrutiny concerning guarantees for time synchronization methods.

For me personally, the separation of NTP would ideally not only be made into protocol on the one hand and algorithmic and response behavior on the other hand, but go one step further.

I would prefer separation into three layers, specifically:

1) Network protocol (what kinds of messages can one send, and how does one have to respond to incoming messages); this should be specified on a per-association basis and security should be handled mostly here.

2) Algorithmic offset calculations to the desired "true" time scale (how does one calculate time offset, frequency offset, perhaps offset of higher-order derivatives); this can involve multiple associations; security of the individual associations should be known so it can be taken into account if desired; this is the module that needs to be aware of the use of different time scales such as TAI vs. UTC, the handling of leap seconds etc.; this module should also be aware of how clock steering is handled (see below) and account for it.

3) Clock steering (how is the participant's clock manipulated in order to accomodate the measured offset(s) and make the clock read something close to the "true" time, e.g. hard-setting the counter vs. manipulating the frequency via counter increment vs. manipulating the clock in other ways such as via the oscillator's voltage vs. handling all of that via virtual clocks); this can differ between machines, specifically between OS and hardware and should be treated appropriately.

With this separation, we need well-defined interfaces between 1) and 2) and between 2) and 3).

Not dealing with 3) in a specification and leaving it to implementations should be seen as a real option, keeping in mind that knowledge of 3) can increase accuracy of 2).

Regarding the nomenclature, I would strongly suggest getting consensus first on which of the many ways forward is going to be taken in order to arrive at something that can be called "NTPv5".

Specifically, it should be decided early which of 1), 2) and 3) that document should treat, the options as I see them being 1) only, or 1) and 2), or all three.

What do you all think?

Best regards,

Kristof

-----"ntp" <[email-masked]> schrieb: -----

An: "Harlan Stenn" <[email-masked]>
Von: "Dieter Sibold"
Gesendet von: "ntp"
Datum: 16.04.2020 19:29
Kopie: "Watson Ladd" <[email-masked]>, "NTP WG" <[email-masked]>, "Doug Arnold" <[email-masked]>, "Daniel Franke" <[email-masked]>, "Salz, Rich" <rsalz=[email-masked]>
Betreff: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

Dieter

---

From: Harlan Stenn <[email-masked]>
Date: Fri, 17 Apr 2020 03:48:56 -0700

> What regulations govern NTP's clock synchronization algorithm?

Who said the regulations govern NTP's clock synchronization algorithm?

> Who risks being liable to whom for what?

If there is a requirement that time be disciplined by NTP and the
governing NTP spec describes algorithms and behavior and you are running
something that behaves differently, then you are non-compliant, and liable.

If you KNOWINGLY run something that is non-compliant, this may escalate
the violation to the level of fraud.  This may trigger treble damages.

But that's not quite the point here.

More to the point, NTPv4 and all of its predecessors specified:

- the packet format
- the algorithms
- a set of behavioral limits and specifications

This means others were allowed to write specifications (regulations)
assuming and/or relying on "NTP" - the packet format, algorithms, and
behavior.

Removing or separating the algorithmic and behavioral specifications
from the NTP specification at best cost-shifts that discussion
elsewhere, and I submit it does this to groups that are likely
completely unaware that they can no longer rely on behavior that they
had no idea they previously relied on.

Are you planning to just do a protocol spec and "do the algorithms and
behavior spec later", or worse yet, let others do that if they think
it's important?

---

From: Doug Arnold <[email-masked]>
Date: Fri, 17 Apr 2020 14:29:16 +0000

I support this proposed top level architecure, and would be happy to help.

Doug

________________________________
From: ntp <[email-masked]> on behalf of [email-masked] <[email-masked]>
Sent: Friday, April 17, 2020 3:20 AM
To: NTP WG <[email-masked]>
Subject: [Ntp] Antwort: Re: The bump, or why NTP v5 must specify impulse response

Hey all,

What do you all think?

Best regards,
Kristof

-----"ntp" <[email-masked]<mailto:[email-masked]>> schrieb: -----
An: "Harlan Stenn" <[email-masked]<mailto:[email-masked]>>
Von: "Dieter Sibold"
Gesendet von: "ntp"
Datum: 16.04.2020 19:29
Kopie: "Watson Ladd" <[email-masked]<mailto:[email-masked]>>, "NTP WG" <[email-masked]<mailto:[email-masked]>>, "Doug Arnold" <[email-masked]<mailto:[email-masked]>>, "Daniel Franke" <[email-masked]<mailto:[email-masked]>>, "Salz, Rich" <rsalz=[email-masked]<mailto:rsalz=[email-masked]>>
Betreff: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

Dieter

On 16 Apr 2020, at 11:58, Harlan Stenn wrote:

> Harlan Stenn <[email-masked]<mailto:[email-masked]>>

_______________________________________________
ntp mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ntp

---

From: Kurt Roeckx <[email-masked]>
Date: Fri, 17 Apr 2020 16:41:39 +0200

> had no idea they previously relied on.

Maybe it's useful to have performance guarantees specified, but
leave the algorithm how to get there either to the
implementations, or in a separate document.

Kurt

---

From: <[email-masked]>
Date: Fri, 17 Apr 2020 14:51:50 +0000

I recommend starting by identifying the top level subsystems and defining the interfaces among them, as Kristoff has suggested.

I would propose as a next step to define implementations of each subsystem which meet the needs of the most common NTP applications, for example, synchronizing server and router clocks for log file entry timestamps.  This default implementation needs to be simple to configure, robust, and work over the public internet and typical enterprise networks.  It does not need to be super accurate or work over some specialized network, like a high frequency trading system.

Specialized use cases can have different implementations for one or more of the subsystems, although I would expect that we want to keep a few things consistent, such the over the wire packet format, and message sequence to collect 4 timestamps at a client or a higher stratum number server.

Doug

________________________________
From: ntp <[email-masked]> on behalf of Harlan Stenn <[email-masked]>
Sent: Friday, April 17, 2020 3:48 AM
To: [email-masked] <[email-masked]>
Subject: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

- the packet format
- the algorithms

---

From: Daniel Franke <[email-masked]>
Date: Fri, 17 Apr 2020 10:52:43 -0400

> Specifically, it should be decided early which of 1), 2) and 3) that document should treat, the options as I see them being 1) only, or 1) and 2), or all three.

My ontology is similar, but quite identical, to Kristof's:

I. The syntax and semantics of the wire protocol
IIa. The algorithm for estimating the offset between your clock and
that of your peers
IIb. The algorithm for combining estimates collected from multiple
peers into one "consensus" estimate
IIIa. How to use that estimate to steer the clock used for capturing
the timestamps used in the wire protocol
IIIb. How to use that estimate to steer the clock that other
applications see when they ask for the system time

The central question of this thread so far, up until Kristof entered,
could be expressed as "How much is (IIIa), and by extension (IIa) and
(IIb), really a part of (I)? When we say that the timestamps on the
wire confer an estimate of the current time, how particular do we need
to be about the properties of that estimate?".

NTPv4 and prior don't distinguish at all between (IIIa) and (IIIb),
but NTPv5 might have to. My sketch allows discontinuous adjustments to
the `tai_loc_offset` field with each new data point obtained. But a
lot of applications depend on having a single system clock that's all
at once monotonic, reasonably accurate, and reasonably
frequency-stable, which means there has to be some further smoothing
step involved (perhaps a PLL, perhaps something else).

In an ideal world, the contract between the OS and the NTP daemon
would be the same on all platforms, and would be something like this:
the OS provides the NTP daemon with a monotonic time counter driven by
the hardware oscillator, and the NTP daemon provides the OS with a
running estimate (maybe with error bars included) of the first few
Taylor series coefficients of the function from that counter to TAI.
What the OS does with this information and in particular how it
communicates it to other applications on the system is none of the NTP
daemon's concern.

Unfortunately, currently *no* OS provides us with anything this clean,
and even if one did we'd still have to accommodate all the ones that
don't if we want our spec to be platform-agnostic. Since we don't
control the OS landscape and it changes on a different and faster
schedule than NTP spec does, there's only so much we can do here. We
can specify what we wish the interface would look like and maybe some
kernel hackers will feel inspired by it, but for the foreseeable
future NTP implementers are going to have to contend with
less-than-ideal interfaces, and the most the IETF can do for them is
give some abstract general guidance on that.

---

From: Doug Arnold <[email-masked]>
Date: Fri, 17 Apr 2020 15:01:22 +0000

Good point on the NTP/OS interface.  Perhaps in the architecture we will define what NTP needs from and whats to set in the OS, like the idealized OS interface you alluded to.  Then each specific OS will need some kind of adapter shim layer.  That could make NTPv5 easy to port to a new OS.  You only need the OS adaptor layer.

Doug
________________________________
From: ntp <[email-masked]> on behalf of Daniel Franke <[email-masked]>
Sent: Friday, April 17, 2020 7:52 AM
To: [email-masked] <[email-masked]>
Cc: NTP WG <[email-masked]>
Subject: Re: [Ntp] Antwort: Re: The bump, or why NTP v5 must specify impulse response

---

From: Warner Losh <[email-masked]>
Date: Fri, 17 Apr 2020 09:35:17 -0600

On Fri, Apr 17, 2020 at 8:53 AM Daniel Franke <[email-masked]> wrote:

> > As many of you may know, the place that I'm coming from is not decades
> of practical experience in NTP implementation and/or maintenance, but
> instead years of close academic and theoretic scrutiny concerning
> guarantees for time synchronization methods.
> >
> > For me personally, the separation of NTP would ideally not only be made
> into protocol on the one hand and algorithmic and response behavior on the
> other hand, but go one step further.
> > 1) Network protocol (what kinds of messages can one send, and how does
> one have to respond to incoming messages); this should be specified on a
> per-association basis and security should be handled mostly here.
> > 2) Algorithmic offset calculations to the desired "true" time scale (how
> does one calculate time offset, frequency offset, perhaps offset of
> higher-order derivatives); this can involve multiple associations; security
> of the individual associations should be known so it can be taken into
> account if desired; this is the module that needs to be aware of the use of
> different time scales such as TAI vs. UTC, the handling of leap seconds
> etc.; this module should also be aware of how clock steering is handled
> (see below) and account for it.
> > 3) Clock steering (how is the participant's clock manipulated in order
> to accomodate the measured offset(s) and make the clock read something
> close to the "true" time, e.g. hard-setting the counter vs. manipulating
> the frequency via counter increment vs. manipulating the clock in other
> ways such as via the oscillator's voltage vs. handling all of that via
> virtual clocks); this can differ between machines, specifically between OS
> and hardware and should be treated appropriately.
> >
> > Not dealing with 3) in a specification and leaving it to implementations
> should be seen as a real option, keeping in mind that knowledge of 3) can
> increase accuracy of 2).
> >
> > Regarding the nomenclature, I would strongly suggest getting consensus
> first on which of the many ways forward is going to be taken in order to
> arrive at something that can be called "NTPv5".
> > Specifically, it should be decided early which of 1), 2) and 3) that
> document should treat, the options as I see them being 1) only, or 1) and
> 2), or all three.
> >
> > What do you all think?
>

Counters of sufficient resolution wrap. There's often a software counter
too. And getting either the raw software or raw hardware counters and/or
their notion of a relationship to the local system time can be tricky as
well. Some OSes allow access to the raw counters (since CPUs allow turning
that access on/off), and some even publish what's basically a 'phase at
count' testimate and a frequency estimate that the applications can then
use to compute time w/o a round trip into the kernel (this is often hidden
inside of libc's clock_gettime or similar).

Also, how does NTPd get access to this pure counter when it timestamps the
packets? It would need to know that in order to measure the evolution of
that counter over time to make estimates about it.

And I'd maintain that the NTP daemon needs to provide both the phase
estimate of the counter vs the TAI and a shorter-term frequency estimate
for the counter's evolution. Since the counter is likely to be lousy over
time and the frequency will wander all over the place (relative to a high
precision time keeping xtal or clock), the most useful information is that
counter X is TAI time Y and the short-term frequency is Z (maybe with a Z'
estimate as well). The OS will need to read the counter and produce a
TAI/UTC time quite often, and doing more than a simple computation will
show up in benchmarks. But that thinking may be a legacy of NTP steering
the OS's notion of time that it gets from a pure counter under the covers.
This is why we have ntp_adjtime providing a time delta and frequency error
and a bunch of other stuff that talk about ntpd's notion of how good those
are which are on a spectrum from generally useful, to totally NTPd
specific. I have a hard time seeing a way to not push the error in time /
frequency measurements down.

Second, the OS needs to keep a smooth time to its clients w/o time jumps
(but small changes in frequency are OK). So there will likely need to be
some coordination between the OS and the ntp daemon to communicate how much
of the 'steers' or 'tyler series' information it could ingest and what it's
notion of a future counter's time will be so NTP can help steer the OS's
use of the counter so it will be on time in the future and not overshoot as
the frequency wanders around.

> give some abstract general guidance on that.

I think this sounds great from a theoretical view. I think it sounds lousy
from a practical view.

Generally, timekeeping is a critical part of the OS. This makes the above
ideal not implementable in many ways.

CPU Cost: If the OS is having to do something complicated to convert the
internal counters to a time as often as it needs time, the interface will
be a non-starter.

DoS issues: What does the OS do when ntpd information goes away? Because it
will. And then it will come back and the OS will have evolved its time
based on hardware counters whose frequency estimates will have been used as
best they can, but they will likely be wrong.

Convergence: How do we know NTPd's recommendations will converge with
whatever is going on inside the OS? Is there a stable form or do we get
oscillations. This is the same issue, btw, with algorithmic improvements
talked about elsewhere: you need to make sure that whatever is going on
with the steering of time doesn't cause oscillations as you resynchronize
with your upstreams. This will be especially an issue with pools of NTPd
servers that are steering using different algorithms. Steering to a steered
clock is tricky and well known problem in the high-precision time-keeping
biz. Often times I created a measurement of a free-running clock and
published my estimates of that clock so that people making measurements
could make adjustments after the fact since they needed a high precision
measurement of events, but didn't need it at the instant the measurement
was made. Had we steered the clock, there'd be lags to that accuracy. And
if they were steering their clocks, they'd get oscillations.

Warner

---

From: Daniel Franke <[email-masked]>
Date: Fri, 17 Apr 2020 12:08:15 -0400

On Fri, Apr 17, 2020 at 11:35 AM Warner Losh <[email-masked]> wrote:
> Counters of sufficient resolution wrap. There's often a software counter too. And getting either the raw software or raw hardware counters and/or their notion of a relationship to the local system time can be tricky as well. Some OSes allow access to the raw counters (since CPUs allow turning that access on/off), and some even publish what's basically a 'phase at count' testimate and a frequency estimate that the applications can then use to compute time w/o a round trip into the kernel (this is often hidden inside of libc's clock_gettime or similar).
>
> Also, how does NTPd get access to this pure counter when it timestamps the packets? It would need to know that in order to measure the evolution of that counter over time to make estimates about it.
>
> And I'd maintain that the NTP daemon needs to provide both the phase estimate of the counter vs the TAI and a shorter-term frequency estimate for the counter's evolution. Since the counter is likely to be lousy over time and the frequency will wander all over the place (relative to a high precision time keeping xtal or clock), the most useful information is that counter X is TAI time Y and the short-term frequency is Z (maybe with a Z' estimate as well). The OS will need to read the counter and produce a TAI/UTC time quite often, and doing more than a simple computation will show up in benchmarks. But that thinking may be a legacy of NTP steering the OS's notion of time that it gets from a pure counter under the covers. This is why we have ntp_adjtime providing a time delta and frequency error and a bunch of other stuff that talk about ntpd's notion of how good those are which are on a spectrum from generally useful, to totally NTPd specific. I have a hard time seeing a way to not push the error in time / frequency measurements down.
>
> Second, the OS needs to keep a smooth time to its clients w/o time jumps (but small changes in frequency are OK). So there will likely need to be some coordination between the OS and the ntp daemon to communicate how much of the 'steers' or 'tyler series' information it could ingest and what it's notion of a future counter's time will be so NTP can help steer the OS's use of the counter so it will be on time in the future and not overshoot as the frequency wanders around.

I don't think we're disagreeing about anything here. Where I say
"first few Taylor series coefficients" and you say "time delta and
frequency error", we're talking about the same thing — time delta and
frequency error are the first and second coefficients respectively.
I'm considering the kernel and libc all part of the OS, so questions
about how to deal with hardware counters that wrap and what should
happen in kernel versus user space all lives on the "OS" side of
abstraction barrier.

---

From: Warner Losh <[email-masked]>
Date: Fri, 17 Apr 2020 10:22:10 -0600

On Fri, Apr 17, 2020 at 10:08 AM Daniel Franke <[email-masked]> wrote:

> > Also, how does NTPd get access to this pure counter when it timestamps
> the packets? It would need to know that in order to measure the evolution
> of that counter over time to make estimates about it.
> >

Well, the OS provides an 'abstract' counter, but only kinda. Usually it
integrates the past to some point in time after the last counter wrap and
then uses the delta from that point to the current counter to compute the
time. The OS may have a hard time providing a pure counter to NTP, and it
should be designed such that it may only get the low order bits from the
counter and an OS timestamp...

Warner

---

From: Daniel Franke <[email-masked]>
Date: Fri, 17 Apr 2020 12:39:11 -0400

On Fri, Apr 17, 2020 at 12:22 PM Warner Losh <[email-masked]> wrote:
> Well, the OS provides an 'abstract' counter, but only kinda. Usually it integrates the past to some point in time after the last counter wrap and then uses the delta from that point to the current counter to compute the time. The OS may have a hard time providing a pure counter to NTP, and it should be designed such that it may only get the low order bits from the counter and an OS timestamp...

On Linux, at least, there's CLOCK_MONOTONIC_RAW which is the kind of
thing I have in mind.

---

From: Hal Murray <[email-masked]>
Date: Fri, 17 Apr 2020 14:24:24 -0700

[email-masked] said:
> Maybe it's useful to have performance guarantees specified, but leave the
> algorithm how to get there either to the implementations, or in a separate
> document. 

If you are going to discuss performance guarantees, you have to start with a 
description of the input data.

How would you describe the traffic parameters of the internet?

You also get to consider the effects of temperature on the system crystal.  I 
think I could at least come up with some parameters for that.

Interrupt latency, Scheduler, ...

-- 
These are my opinions.  I hate spam.

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 20 Apr 2020 13:52:04 +0200

>>> "Dieter Sibold" <[email-masked]> schrieb am 16.04.2020 um 19:27 in
Nachricht
<17906_1587058115_5E9895C3_17906_233_1_E591F38A-A303-4077-BC4C-A61AB7867F83@gmai
> and to be compliant with the protocol spec.

Hi!

In the past I tried to read "version" to find out what the implementation is
conforming to. One remarkable exception was Solaris NTP that suppressed the
version (obviously for "security" reasons). The other alternative was to guess
from the variable list which version the remote end was. Then (again for
"security" reasons) most implementations shut down the control protocol too.
Maybe NTPv5 should require some more verbose version field. Also considering
the few bits the version field has right now, we need a better mechanism for
the future anyway, I guess...

One of the amazing facts of NTPv4 implementation was that variable names as
presented in the control protocol changed names. Like these:
CS_DRIFT,     RO, "frequency" (once "drift")
CS_ERROR,     RO, "clk_jitter" (once "error")
CS_STABIL,    RO, "clk_wander" (once "stability")

I think a future specification should require fixed names for specific
variables, too Or maybe assign numeric labels like four digits.
So if the semantics of a variable changes, it would get a new number...

Regards,
Ulrich

>>> Can you explain why 3‑>4 could still be called NTP but 4‑>5 should 
>>> not?
>>
>> ‑ the protocol/packet structure
>> ‑ algorithmic and response behavior
>>
>> ‑‑ 
>> http://networktimefoundation.org ‑ be a member!
>

---

From: [email-masked]
Date: Tue, 21 Apr 2020 12:25:39 +0200

Moreover, I think it's always good to ensure that if you demand a certain performance level, you can give at least a description of a proof that that performance level can in fact be reached.

I also agree with Hal that the specification of performance guarantees is going to be tricky, since you're going to have to assume a lot of parameters.

Easiest is probably to formulate guarantees dynamically depending on achieved message in-flight roundtrip time per exchange, but even then you're making assumptions about the drift and other behavior of the client's clock between the two timestamps that it takes.

Best regards,

Kristof

An: "Kurt Roeckx" <[email-masked]>
Von: "Hal Murray"
Gesendet von: "ntp"
Datum: 17.04.2020 23:25
Kopie: [email-masked], [email-masked]
Betreff: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

---

From: Doug Arnold <[email-masked]>
Date: Tue, 21 Apr 2020 14:49:47 +0000

I would expect that the packet format and general packet exchange sequence would not change.  I would expect that the packet rate, clock steering algorithms, and security mechanisms might.

Doug

________________________________
From: Salz, Rich <[email-masked]>
Sent: Tuesday, April 21, 2020 7:08 AM
To: Harlan Stenn <[email-masked]>; Salz, Rich <rsalz=[email-masked]>; Dieter Sibold <[email-masked]>; Doug Arnold <[email-masked]>
Cc: Watson Ladd <[email-masked]>; NTP WG <[email-masked]>; Daniel Franke <[email-masked]>
Subject: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

>    NTP is a combination of two parts:

Thank you.

So if NTPv5 said that it only changes the protocol, and not the algorithms, that would be okay?

---

From: Doug Arnold <[email-masked]>
Date: Tue, 21 Apr 2020 14:40:30 +0000

The ITU does this for their profiles of PTP in wireless backhaul, but it is huge amount of work.  They define network traffic test cases, develop models for local clocks, and run quite sophisticated simulations.  In the case of NTP the use variety of network conditions are more varied.  I fear that we would either end up with a proof which is useless to network operators because of the simplifying assumptions we have to make, or a PhD thesis amount of work.

Doug

________________________________
From: ntp <[email-masked]> on behalf of [email-masked] <[email-masked]>
Sent: Tuesday, April 21, 2020 3:25 AM
To: Hal Murray <[email-masked]>
Cc: [email-masked] <[email-masked]>; Kurt Roeckx <[email-masked]>
Subject: [Ntp] Antwort: Re: The bump, or why NTP v5 must specify impulse response

Best regards,
Kristof

-----"ntp" <[email-masked]<mailto:[email-masked]>> schrieb: -----
An: "Kurt Roeckx" <[email-masked]<mailto:[email-masked]>>
Von: "Hal Murray"
Gesendet von: "ntp"
Datum: 17.04.2020 23:25
Kopie: [email-masked]<mailto:[email-masked]>, [email-masked]<mailto:[email-masked]>
Betreff: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

_______________________________________________
ntp mailing list
ntp@ietf..org<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/ntp

---

From: "Salz, Rich" <[email-masked]>
Date: Tue, 21 Apr 2020 14:08:45 +0000

Thank you.

---

From: Philip Prindeville <[email-masked]>
Date: Tue, 21 Apr 2020 11:56:52 -0600

> On Apr 17, 2020, at 4:48 AM, Harlan Stenn <[email-masked]> wrote:
> 
> [snip]
> 

Not to throw gasoline on the fire, but Postel told me decades ago, “don’t confuse protocol and policy” (he was [ahem] encouraging me to drop 3 paragraphs from the draft of RFC 1048).  It was good advice then and it holds.

Algorithms are, in essence, a “policy” for disciplining a clock.

A compromise might be to define requirements for the protocol parametrically, in terms of drift, jitter, etc. while not defining the algorithms to do that.

This opens the field up to more active research in ways to improve the protocol and the use of clocks without tying anyone’s hands.

The protocol could provide a “reference algorithm” that’s used as a baseline and as an example of a certain performance “envelope” under well-defined circumstances.

Given how long NTPv4 has been around (for both right and wrong reasons), it’s reasonable to expect NTPv5 to live at least half as long.  A lot of technology can potentially emerge during that time and we should have the flexibility to incorporate it as it develops.

Look how far we’ve come since Cesium ovens as reference clocks.

-Philip

---

From: Doug Arnold <[email-masked]>
Date: Tue, 21 Apr 2020 18:59:48 +0000

I agree with Philip.  And I also like the terms protocol and policy.  In my mind the protocol is what has to be the same for implementations to communicate and the policy is what allows a specific node to make use of information received via the protocol.  It is possible to have nodes in the same network using the same protocol, but with different policies.

There could be a generic NTPv5 which works for 90% or more of the users, but a modular architecture which allows for some alternative policies for niche uses.  People are already doing this for NTPv4, its just that its proprietary.

A little reminder of history: an early draft of NTPv4 was based on a modular approach, but was later revised to be a monolithic complete solution.  The monolithic solution works very well for setting clocks in servers and routers for general IT purposes.  However, the collapse of the modular approach drove the telecom timing community out of the NTP working group and into the IEEE 1588, where we ended up defining unicast PTP.  Unicast PTP is incompatible with traditional multicast PTP and in some ways is more like NTP than it is like multicast PTP.  So we created a new find or PTP to fit an application which could have been served by a more flexible NTPv4.

Now the same thing is happening in data centers.  People are installing PTP.  Many of these people tell me they would prefer to install a high precision variant of NTP.  They keep asking me to change PTP to be more like NTP.  It might be better to make NTP a little more flexible than to create more incompatible flavors of PTP.

Doug

________________________________
From: ntp <[email-masked]> on behalf of Philip Prindeville <[email-masked]>
Sent: Tuesday, April 21, 2020 10:56 AM
To: Harlan Stenn <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

-Philip

---

From: Philip Prindeville <[email-masked]>
Date: Tue, 21 Apr 2020 15:05:12 -0600

At the very least, the protocol and the algorithms should be published as separate standards so one can be updated as needed and independently of the other.  If we get the protocol right (and we’ve certainly been discussing it for a while now), it might not change again for a long time.  But there’s no reason we can’t have more than one algorithm published as a reference for implementation.

It’s not unprecedented.

The domain name system was published as RFC-1065 (concepts, structure, and proposed management) and RFC-1066 the actual protocol and file formats.

-Philip

> On Apr 21, 2020, at 12:59 PM, Doug Arnold <[email-masked]> wrote:
>

---

From: Watson Ladd <[email-masked]>
Date: Tue, 21 Apr 2020 14:14:35 -0700

> I agree with Philip.  And I also like the terms protocol and policy.  In my mind the protocol is what has to be the same for implementations to communicate and the policy is what allows a specific…

On the matter of policy, are we ever going to unify posting formats? *ducks*
> There could be a generic NTPv5 which works for 90% or more of the users, but a modular architecture which allows for some alternative policies for niche uses.  People are already doing this for NTP…

I wouldn't call chrony's algorithm proprietary. We've seen widespread
deployment and the sky hasn't fallen.

> Now the same thing is happening in data centers.  People are installing PTP.  Many of these people tell me they would prefer to install a high precision variant of NTP.  They keep asking me to chan…

I definitely agree with this sentiment. There is considerable
administrative overhead involved with running an additional service,
ensuring it is up, fixing it when it is not, etc.  Having unified
implementations and permitting gradual improvement in intermediate
network hardware and configuration is far nicer to administrators.
That said in order to get there we need people who make high precision
timekeeping devices to express their opinion about NTPv5 and what will
or won't work for them.

Certainly for leaves in the timing hierarchy there isn't a problem
with alternative devices.

Sincerely,
Watson Ladd

-- 
"Man is born free, but everywhere he is in chains".
--Rousseau.

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 21 Apr 2020 16:23:35 -0700

I still disagree here, for whatever that's worth.

Splitting the documents is great *when it's appropriate*.  In this case,
it will be a *lot* more work to create correct RFCs, there will be way
more opportunities to confuse things or get them wrong, and the people
who specify conformance stuff will have a lot more work to do, too.

A case in point is 5905 and 5906 and the keyIDs: Miroslav saw that 5906
was optional and since he wasn't planning on implementing Autokey he
ignored it.  But 5906 had critical, required information in it that
*should* have been in 5905, and since the reference implementation did
both of them and that's what folks were looking at when the RFCs were
drafted folks didn't notice the fencepost problems.

When people pay attention to something, they often do not pay attention
to stuff they consider "out of scope".

NTP has been a carefully designed system, that includes the protocol,
its operating environments (conditions and constraints), and some
behavioral requirements.

Splitting the document up into different parts is an invitation for
things to be "missed".

H

On 4/21/2020 11:59 AM, Doug Arnold wrote:
> I agree with Philip.  And I also like the terms protocol and policy.  In
> my mind the protocol is what has to be the same for implementations to
> communicate and the policy is what allows a specific node to make use of
> information received via the protocol.  It is possible to have nodes in
> the same network using the same protocol, but with different policies.
> 
> There could be a generic NTPv5 which works for 90% or more of the users,
> but a modular architecture which allows for some alternative policies
> for niche uses.  People are already doing this for NTPv4, its just that
> its proprietary.  
> 
> A little reminder of history: an early draft of NTPv4 was based on a
> modular approach, but was later revised to be a monolithic complete
> solution.  The monolithic solution works very well for setting clocks in
> servers and routers for general IT purposes.  However, the collapse of
> the modular approach drove the telecom timing community out of the NTP
> working group and into the IEEE 1588, where we ended up defining unicast
> PTP.  Unicast PTP is incompatible with traditional multicast PTP and in
> some ways is more like NTP than it is like multicast PTP.  So we created
> a new find or PTP to fit an application which could have been served by
> a more flexible NTPv4.  
> 
> Now the same thing is happening in data centers.  People are installing
> PTP.  Many of these people tell me they would prefer to install a high
> precision variant of NTP.  They keep asking me to change PTP to be more
> like NTP.  It might be better to make NTP a little more flexible than to
> create more incompatible flavors of PTP.
> 
> Doug
> 
> ------------------------------------------------------------------------
> *From:* ntp <[email-masked]> on behalf of Philip Prindeville
> <[email-masked]>
> *Sent:* Tuesday, April 21, 2020 10:56 AM
> *To:* Harlan Stenn <[email-masked]>
> *Cc:* [email-masked] <[email-masked]>
> *Subject:* Re: [Ntp] The bump, or why NTP v5 must specify impulse response
>  
> 
> 
> Not to throw gasoline on the fire, but Postel told me decades ago,
> “don’t confuse protocol and policy” (he was [ahem] encouraging me to
> drop 3 paragraphs from the draft of RFC 1048).  It was good advice then
> and it holds.
> 
> A compromise might be to define requirements for the protocol
> parametrically, in terms of drift, jitter, etc. while not defining the
> algorithms to do that.
> 
> This opens the field up to more active research in ways to improve the
> protocol and the use of clocks without tying anyone’s hands.
> 
> The protocol could provide a “reference algorithm” that’s used as a
> baseline and as an example of a certain performance “envelope” under
> well-defined circumstances.
> 
> Given how long NTPv4 has been around (for both right and wrong reasons),
> it’s reasonable to expect NTPv5 to live at least half as long.  A lot of
> technology can potentially emerge during that time and we should have
> the flexibility to incorporate it as it develops.
>

---

From: "Ulrich Windl" <[email-masked]>
Date: Wed, 22 Apr 2020 01:44:18 +0200

Hi!

But maybe let's try to structure v5 better than the versions before.
I could imagine like specifying packet formats and packet flows for one major
part, while other parts treat specific algorithms. Maybe even there could be
more than one for each category.
And I think the algorithms should not just be given as code (specification by
implementation), but also as some high-level description. That alone could help
to find flaws in the code. I think in the past there was a lot of "magic math"
in the algorithms, claiming "don't ever change this, or the world will fall
apart" ;-)
I had several cases where the specs did not help me, so I had to read the code
of the implementation. Having to do that is a clear sign that the specification
is incomplete.
In a perfect world the specification says what is allowed, and everything that
is not allowed is forbidden. Mayb specification use both worlds: Some things
are allowed, some things are forbidden, and there is a gray zone that is to be
avoided ("neither allowed or forbidden").

Regards,
Ulrich

>>> Harlan Stenn 22.04.2020, 01:24 >>>

---

From: Watson Ladd <[email-masked]>
Date: Tue, 21 Apr 2020 19:30:53 -0700

> who specify conformance stuff will have a lot more work to do, too.

Are there particular examples of conformance stuff you want to point
to that would be substantially made harder by splitting the documents?

> drafted folks didn't notice the fencepost problems.

This is an argument *for* properly modularization. Autokey was, to use
the technical term, a mess. Embedding important information in the
midst of  this mess is a problem whether one or two documents.

> its operating environments (conditions and constraints), and some

The question is what it should look like in the future. I agree it's
been a very carefully designed program, but there are drawbacks to
that model particularly when other implementations want to coexist.

---

From: "Ulrich Windl" <[email-masked]>
Date: Wed, 22 Apr 2020 08:33:10 +0200

>>> Philip Prindeville <[email-masked]> schrieb am 21.04.2020
um
19:56 in Nachricht
<31673_1587492815_5E9F37CE_31673_735_1_2A6E880D-C467-4AC5-A8C2-F3C61B323A53@redf
sh-solutions.com>:

> Not to throw gasoline on the fire, but Postel told me decades ago, “don’t 
> confuse protocol and policy” (he was [ahem] encouraging me to drop 3 
> paragraphs from the draft of RFC 1048).  It was good advice then and it 
> holds.
> 

What speaks against that IMHO is correctness assumptions: If everybody can
define his own algorithms, but subsequent clients have to use those numbers for
their own correctness calculations, how can a client have any guarantee that
the time is within the bounds the packets (I'm not using the word "protocol")
claim? Even with the current NTP implementation it's quite a challenge to
assure that the time displayed is the actual time (the NTP performance numbers
may look nice, but still the time can be wrong).

> The protocol could provide a “reference algorithm” that’s used as a baseline

> and as an example of a certain performance “envelope” under well-defined 
> circumstances.
> 
> technology can potentially emerge during that time and we should have the 
> flexibility to incorporate it as it develops.
>

---

From: "Dieter Sibold" <[email-masked]>
Date: Wed, 22 Apr 2020 08:59:27 +0200

On 22 Apr 2020, at 8:33, Ulrich Windl wrote:

>> “don’t

The NTPv5 protocol specification has to define a set of metrics which 
each server instance at each stratum has to pass to its clients. This by 
the way also is necessary for achieve traceability. If these metrics are 
well defined, correctness assumption is a priori independent of the 
algorithms to discipline clocks.

>> the

---

From: "Ulrich Windl" <[email-masked]>
Date: Wed, 22 Apr 2020 09:14:53 +0200

>>> "Ulrich Windl" <[email-masked]> schrieb am 22.04.2020 um
> is not allowed is forbidden. Mayb specification use both worlds: Some things

s/Mayb specification/Many specifications/

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 22 Apr 2020 12:40:28 +0200

> this in the NTPv5 spec, since we've obviously been getting by okay so

That's a good question. I did some experiments to try to get an
answer. The code of the PLL/FLL that's provided in RFC 5905 seems to
be identical to the code in ntp-4.2.4. Here is a response to a 10ms
time step of three clients in a chain using the default minpoll and
maxpoll:

https://mlichvar.fedorapeople.org/ntpresponse/daemon.png

That looks ok to me.

However, that's not the loop which is actually used on most systems
running ntpd. By default it uses the kernel PLL/FLL (aka kernel
discipline), which has a different FLL. A response I saw in my test
was much worse:

https://mlichvar.fedorapeople.org/ntpresponse/kernel.png

There is a large overshoot and ringing, getting worse in the chain of
servers. If that's how ntp is expected to work, or at least users were
happy with so far, we probably don't need to worry about it too much.

I think it should be a responsibility of individual implementations to
have a reasonably well working control loop. Is it likely that two
different implementations would perform well in a homogeneous
environment, but fail horribly when mixed? If that happened, couldn't
they cooperate and make a fix on either side?

FWIW, the PTPv2 specification doesn't have any requirements on the
clock servos and different implementations seem to be able to
interoperate. As I understand it, the issue with long chains of clocks
(which is avoided with transparent clocks) is not specific to mixed
environments.

-- 
Miroslav Lichvar

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 22 Apr 2020 12:49:54 +0200

> in the algorithms, claiming "don't ever change this, or the world will fall

Yes. No code, just a general description.

What point is in including code or having overly restrictive
requirements on the algorithms when nobody actually implements them?
RFC 5905 is based on algorithms of a particular ntp version. ntp has
evolved since then. I'm not aware of any NTP implementation that would
now "fully conform" to the spec. 

-- 
Miroslav Lichvar

---

From: Danny Mayer <[email-masked]>
Date: Thu, 23 Apr 2020 10:59:41 -0400

> implementations, or in a separate document.

This makes no sense. You can't get performance out of a clock. The
things you have to rely on is the reliability of the clocks your NTP
server are referencing, it's stability, error estimates, jitter,
accuracy, and closeness to UTC (if that can be measured). You need to
remember that NTP is a set of loosely coupled networked clocks
interacting with each other. They cannot be easily separated as if you
are running a computer game.

Danny

---

From: Doug Arnold <[email-masked]>
Date: Thu, 23 Apr 2020 18:37:38 +0000

The accuracy of time at a client or a higher stratum number server has almost nothing to do with the stratum 1 server or servers from which the time is derived.  The accuracy is mostly due to the conditions in the network.  Also different applications require a different approach.  Consider the two different use cases for ntpv5:
User case A:
A Stratum 2 server get it's time from a pool of stratum 1 servers through the public internet and distributes it to clients in computers, servers and routers in a large enterprise network.  The clients need time to about 1 second for log file entries and security time out intervals.  Accuracy at the stratum 2 server and clients is almost completely determined by queuing noise from so many hops that the noise is almost Gaussian.  After removal of outliers the stratum 2 server and clients average clock corrections in a PLL based on a low pass filter.  Because accuracy requirements are loose, clients request updates only every few minutes.

Use case B:
A financial traders network needs microsecond level timing for regulatory requirements and network latency requirements in their enterprise network.  They have their own stratum 1 server inside their firewall, and just a few hops to the clients.  They can achieve their accuracy goals with a high rate of ntp packets, non-linear lucky packet filters, and by turning off encryption to make hardware timestamping easier.

Both of these use cases can use the same ntp packets and the same sequence (client -> server -> client).  But the servo loops are different, the packet rates are different, and the security requirements are different.  The use cases there for could use the same protocol (ntp) but different policies (servo, security settings, packet rate)

Now certainly the security is part of the protocol, but whether you turn it on or not, is policy.  Or whether it is encryption or just an authentication hash could be policy.

The easiest way to specify ntpv5 for both these use cases, and many others, is to make the policies separate from the over the wire protocol, so that there can be different policy subsystems which work with the same protocol.

Use case B is currently solved either with PTP, or with a non-standard variant of ntpv4.  It seems that if people in the field need to violate the ntpv4 standard to get their job done, then maybe ntpv5 should be more flexible.

Doug

________________________________
From: ntp <[email-masked]> on behalf of Danny Mayer <[email-masked]>
Sent: Thursday, April 23, 2020 7:59 AM
To: Kurt Roeckx <[email-masked]>; Harlan Stenn <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [Ntp] The bump, or why NTP v5 must specify impulse response

Danny

---

From: Philip Prindeville <[email-masked]>
Date: Thu, 23 Apr 2020 13:38:59 -0600

Not sure I agree…

> On Apr 23, 2020, at 12:37 PM, Doug Arnold <[email-masked]> wrote:
> 
> User case A:

As you get into 10GBE networking, events can happen in extremely small intervals.

Half a second of variance either way can completely obscure if two events are related, which happened first, what’s cause and what’s effect, etc.  I’d say that millisecond accuracy is the minimum useful precision in Enterprise and data center scenarios.

> A financial traders network needs microsecond level timing for regulatory requirements and network latency requirements in their enterprise network.  They have their own stratum 1 server inside the…

But encryption is (by necessity, or you open yourself to side-channel timing attacks) fixed delay.  And fixed delay is something that NTP handles very well (like propagation delay, which is speed-of-light constrained).  It’s the variable queuing delays that get you into trouble (those QoS markings and traffic shaping can mostly mitigate that).

> The easiest way to specify ntpv5 for both these use cases, and many others, is to make the policies separate from the over the wire protocol, so that there can be different policy subsystems which…

And here we find agreement.  “Protocol not Policy”, as Jon would say.

-Philip

---

From: Kurt Roeckx <[email-masked]>
Date: Thu, 23 Apr 2020 22:18:04 +0200

> Half a second of variance either way can completely obscure if two events are related, which happened first, what’s cause and what’s effect, etc.  I’d say that millisecond accuracy is the minimum u…

I think your requirements seem to fall under use case B, not A.

If you want millisecond accuracy, the stratum 1 servers should
probably be in the order of 10 millisecond away, and always
reachable, never congested, and you probably need an update rate
that's smaller than 1 minute. And I'm not sure that 10 millisecond
is actually good enough.

Kurt

---

From: "Ulrich Windl" <[email-masked]>
Date: Fri, 24 Apr 2020 10:29:07 +0200

>>> Kurt Roeckx <[email-masked]> schrieb am 23.04.2020 um 22:18 in Nachricht
<[email-masked]>:
>> > On Apr 23, 2020, at 12:37 PM, Doug Arnold <[email-masked]>

> wrote:
>> > 
>> > User case A:
>> > A Stratum 2 server get it's time from a pool of stratum 1 servers through

> the public internet and distributes it to clients in computers, servers and

> routers in a large enterprise network.  The clients need time to about 1 
> second for log file entries and security time out intervals.  Accuracy at
the 
> stratum 2 server and clients is almost completely determined by queuing
noise 
> from so many hops that the noise is almost Gaussian.  After removal of 
> outliers the stratum 2 server and clients average clock corrections in a PLL

> based on a low pass filter.  Because accuracy requirements are loose,
clients 
> request updates only every few minutes.
>> 
>> 
>> Half a second of variance either way can completely obscure if two events 
> are related, which happened first, what’s cause and what’s effect, etc.  I’d

> say that millisecond accuracy is the minimum useful precision in Enterprise

> and data center scenarios.
> 

I'm just wondering (stupid idea following):
Can you configure a PPS-signal alone (meaning: time comes from higher (lower
number) stratum servers)? If so does it affect the stratum that server would
have? I one had the idea of reusing old "thin-wire" Ethernet cables with
BNC-T-adapters for distributing a PPS signal, but I never actually did it...
That PPS thing always had been some part of the monolithic NTP algorithm, and
some parts of the protocol rely on it (like "kerninfo"), while others don't. I
think a future NTP spec could modularize PPS much more...

---

From: Hal Murray <[email-masked]>
Date: Fri, 24 Apr 2020 02:24:59 -0700

There are 2 parts to PPS.

The first is part of the normal refclock tangle.  The time accuracy of the 
serial text from a typical GPS is crappy.  Most OS-es provide a method of 
grabbing a time stamp at interrupt time from something like a PPS connected to 
a modem control signal.  That gets much better timing.  The complication in 
this area is that the PPS doesn't tell you which second it is associated with. 
 You need something like the typical NMEA sentences to tell you that.  So the 
PPS gives you the bits to the right of the decimal point and something else 
gives you the bits to the left of the decimal point.

The other part is that many kernels have an option to do-it-all in the kernel. 
 (It's probably not configured in the kernel shipped by your distro.  You have 
to build your own.)  ntpd (or whatever) sets a flag and the kernel does the 
rest.  Normally, ntpd would get the time close enough, turn on kernel PPS 
mode, then watch the serial stream and disable the kernel mode if the serial 
stream said that GPS stopped working.  This generally works much better than 
not using the kernel mode.

I'll come up with some graphs if people want real data.

I don't know why the kernel mode works better.  My guess is that the PLL 
parameters are tuned "better", but I don't see any reason why that code 
couldn't be move to user mode.  Trying that hasn't gotten to the top of my 
list.

------

To answer your question:  Yes, you should be able to distribute a PPS to 
multiple systems.  It's simpler if you distribute the serial signal with it.  
You can make a splitter.  It's just a couple of RS-232 level shifter chips.  I 
don't know of any commercial version.

Some GPS gear just sends the serial text every second once you get it 
configured correctly.  Some needs to be poked every second.  If you run the 
same software on 2 systems connected via a splitter, one of the pokes falls 
into the bit bucket.  That works fine as long as the system doing the poking 
is running but the non-poking system won't get any serial data if the other 
system crashes.

---

From: Doug Arnold <[email-masked]>
Date: Fri, 24 Apr 2020 14:51:14 +0000

The description of stratum in RFC 5905 is a little vague, so your case is open to interpretation.

I am assuming that the server receiving a PPS signal also receives ntp in order to know which second it is, and to know the health of its source of time.  The purpose of stratum numbers is to give a rough indication of the accuracy of the server.  If you calibrate the PPS cable delay, then the accuracy of your PPS receiving server should be nearly as good as the PPS sending one, and it would appropriate for the PPS receiving server to present itself as the same stratum as the PPS sending one.  It is not really different from an ntp server card in a modular time server which gets its time from the backplane of the chassis via pps and ntp.   You just have a long skinny chassis.

Doug

________________________________
From: Ulrich Windl <[email-masked]>
Sent: Friday, April 24, 2020 1:29 AM
To: [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Cc: [email-masked] <[email-masked]>; Doug Arnold <[email-masked]>; [email-masked] <[email-masked]>; [email-masked] <[email-masked]>
Subject: Antw: [EXT] Re: [Ntp] The bump, or why NTP v5 must specify impulse response

the
noise

clients