Subject: [ntpwg] Testing NTP performance

From: "STUART VENTERS" <[email-masked]>
Date: Thu, 1 May 2008 14:47:36 -0500

Greetings,

I'm puzzling over how to test embedded NTP implementations.

Given a client and server connected through some test network,
   how do you accurately measure the difference between the server and client clocks.

If there was a 1PPS test wire coming out of each implementation, it would be easy.
   But unless you can add code to the client, there is usually not a wire,

I need some way to peek into the client and server to see their internal clocks.

I'm thinking the server functionality embedded with the client might help.

Consider the following system with a third NTP box:

                        (C)
                       CLIENT
                    /         \
                  /             \
                /                 \
              /                     \
            /                         \
          /
      SERVER ---------- TEST  ------ CLIENT/SERVER
        (B)            NETWORK           (A)

We are trying to measure the effects of the test network on the client in A.

Server B provides a stable timing reference for the test.

Client/Server A is locked to Server B through the test network.

Client C is connected to servers A and B.

If the links connecting Client C are simple point to point links,
 then hopefully most of the errors in time transport 
   are due to the test network affecting A.

C provides an offset between B&C and also between A&C.

It seems like the difference between these offsets
  should be the offset between A&B.

This seems like a complicated plan, is there a better way?

Regards,

Stuart

ps: My apologies for an NTP 101 question.
     Perhaps someone could point me to where this has already been sorted out.
_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: Rob Seaman <[email-masked]>
Date: Thu, 1 May 2008 13:39:08 -0700

Not to get too philosophical, but isn't this a description of the  
precise problem that NTP itself is designed to solve?

Rob Seaman
NOAO
--

---

From: Danny Mayer <[email-masked]>
Date: Sat, 03 May 2008 18:54:31 -0400

> precise problem that NTP itself is designed to solve?

I think that the real questions that needs to be answered are:
1) How accurate is the estimated offset that the server is using at any 
point in time;
2) What is the error in the calculated frequency that the server has 
estimated;
3) What is the error of the jitter that the server has calculated?

There are two possibilities here:
1) The server itself calculates these values so it will have an estimate 
of the error;
2) An external estimate of these errors introduces you to quantum 
physics issues, the measurement affects the values being measured.

Does this restate the question better?

Danny

---

From: "TS Glassey" <[email-masked]>
Date: Sun, 4 May 2008 07:52:37 -0700

----- Original Message ----- 
From: "Danny Mayer" <[email-masked]>
To: "Rob Seaman" <[email-masked]>
Cc: <[email-masked]>
Sent: Saturday, May 03, 2008 3:54 PM
Subject: Re: [ntpwg] Testing NTP performance

> 1) How accurate is the estimated offset that the server is using at any

yes

> 2) What is the error in the calculated frequency that the server has

yes

> 3) What is the error of the jitter that the server has calculated?

yes

But there is a minimum availability and loading overhaed model needed to 
manage the latency/overhead added by the underlying platform activity.

> 1) The server itself calculates these values so it will have an estimate

yes - but the question of whether the main-thread of the server is handling 
this or there is some form of alternative or co-processor handling the 
overhead for the quantum non-deterministic values of the underlying platform 
dynamics.

The other possibility is to create a MACH or other uKernal type 
implementation to run the core NTP application suite on and rate performance 
relative to it as the reference system.

Todd

---

From: "David L. Mills" <[email-masked]>
Date: Sun, 04 May 2008 18:04:48 +0000

Danny,

With due respect, you guys are missing the point. The NTP design, 
specification and implementation have very specific performance 
objectives and metrics. The sources of error have been carefully 
modelled and justified, by analysis and experiment. You might argue 
about the specific error budget, but it is carefully exposed both in the 
literature and specification. Even the vulnerability to Byzantine 
attacks has been carefully considered and defended.

In your question below the measured offset is the maximum liklihood 
estimate of the client offset relative to the server, while the jitter 
represents the estimated error of the offset estimate itself. The 
synchronization distance represents the maximum error due all causes 
with the assumption that the maximum inherent clock frequency error is 
bounded. The assumed parameters for phase and frequency error estimate 
are represented by the limbs of the Allan deviation specific to each 
installation. There's not much more than can be said abount the 
probabilistics.

There is no quantum physics here, although there might be a dose of 
long-range depndency on oscillator frequency. You can of course argue 
that the specified performance metrics are flawed and incomplete. But, 
the basic metrics called out in the specifidation should in fact provide 
sufficient evidence for an informed judgement of performance.

Time and time again folks have requested absolute bounds on performance, 
such as required accuracy (sic) of one milliseoond within one minute 
after startup. I use as example an aircraft autopilot. Is it required to 
place the aircraft within one meter of the centerling of runway 27-right 
at Heathrow? Under all conditions? Even if the ILS is not operating? 
Does the landing gear survive an altitude error of one meter? Is NTP 
operating correctly if required to slew over one minute and the 
application reserves two passengers for the same seat or sells a stock 
before it is bought?

In short, there are no definitive answers to your questions, only 
statistics.

Dave

---

From: "TS Glassey" <[email-masked]>
Date: Sun, 4 May 2008 11:56:42 -0700

Dave the design wasn't the question - nor was the excellent and innovative 
work done by you and others in the advancement of asymmetric time management 
schema development. The question was what are good numbers for establishing 
minimum performance number's for the composite 'NTP service tool and the 
computer running it' to meet and how do we characterize the performance of 
the appliances or computer based NTP service systems available toay.

The intent is to be able to set specific performance and return-performance 
numbers and to publish them as part of the formal characterization 
specification.

Todd

---

From: Harlan Stenn <[email-masked]>
Date: Sun, 04 May 2008 19:42:32 +0000

Dave,

Please help me out here.

> installation. There's not much more than can be said abount the 

So if somebody wants the answer to ask the question "Does ntpd think the
time on the machine is OK and sync'd?" the first thing we all agree on
is that's not the best question to ask.  But we should be able to
determine:

Q: Does ntpd think it is sync'd?  To something "useful"?
A: Look at the (system) status word in the response to a mode-6 control
   message, and make sure:
   - LI is not 11
   - the value of "clock source" is something tolerable

Q: What time does ntpd think it is?
A: do "the dance of the 4 timestamps".

Q: How close can we expect the previous answer to be to the "correct"
   time?
A: Check out lots of error-related variables and pick the ones you think
   you want.  This includes root dispersion, jitter, and synchronization
   distance.  Remember that the answer is a probability.  (OK, we also
   need to be able to report that probability.)

Q: How is the previous answer affected by "slew-only" mode?
A: (Harlan does not currently know the answer to this one.)

Q: Are any of the above questions superfluous?  If so, please explain.

Q: Should we be asking any additional or different questions?

This general topic is a FAQ, and there is a need to have it covered in
sufficient detail.  Dave, while it may not be productive for *you* to
answer this (either once or rarely), I believe it is critical that we
can point people at a place to get an understanding of the issues that
is more than "read the spec and Dave's book and then study the problem".

Of course, the next thing that will happen is that people will tell us:

- We want to meet the following "performance" target at a given
  statistical level.
- Based on the spec, we can see the target above is reachable.
- We are monitoring the values (X), and we are seeing (Y) which seems to
  be out of spec.
- - how do we figure out what the problem is?
- - how do we fix it?

-- 
Harlan Stenn <[email-masked]>
http://ntpforum.isc.org  - be a member!

---

From: "David L. Mills" <[email-masked]>
Date: Sun, 04 May 2008 21:21:27 +0000

Todd,

Performance criteria are no more appropriate in a formal NTP 
specification than they would be in a TCP formal specification. 
Performance crteria are not in the PTP specification nor in the IEEE 
specifications. What you get from the NTP specification metrics are a 
set of averages and upper bounds. In many cases you can extrapolate 
these values to future expections and estimate future behavior, but that 
is not a matter for specification.

There may be merit in subjecting an implementation to synthetic sources 
that faithfully emulate real phase noise and frequency noise, then 
measure the response statistics. I have done this in several papers and 
the book. The dominant determinant is the Allan intercept. You could say 
at a given Allan intercept the statistics should be such and so. I 
wouldn't bet that every conforming implementation would behave the same way.

You can project nominal behavior by class of application and produce 
cumulative statistics and time series, as I did in the book. These 
statistics typically have long tails, so it might be that the median or 
mean error is mumbleseconds, but that hides the fact that occasional 
measurements may be many times that. I cite a published book called "How 
to Lie With Statistics".

You are free to characterize some applications as I did or concoct your 
own. In any case you must include a disclaimer that the characteristics 
are based on averages with ill defined upper bounds.

Dave

TS Glassey wrote:

> Dave the design wasn't the question - nor was the excellent and 
> innovative work done by you and others in the advancement of 
> asymmetric time management schema development. The question was what 
> are good numbers for establishing minimum performance number's for the 
> composite 'NTP service tool and the computer running it' to meet and 
> how do we characterize the performance of the appliances or computer 
> based NTP service systems available toay.
>
> The intent is to be able to set specific performance and 
> return-performance numbers and to publish them as part of the formal 
> characterization specification.
>
> Todd
> ----- Original Message ----- From: "David L. Mills" <[email-masked]>

---

From: "David L. Mills" <[email-masked]>
Date: Mon, 05 May 2008 01:45:28 +0000

Harlan,

It's hard with my limited eyesight to respond inline.

There are two themes running through your questions, whether the beast 
is hungry and being fed on one hand and how rich are the calories on the 
other. There has been a god deal of recent effor on the first theme in 
the revamped monitoring statistics, in particular the new protostats. 
These are the quickest way to assess whether the beast is working 
correctly now and in the recent past. Take a look on these at pogo 
and/or rackety. These record all significant events that might occur in 
a specification-conformant implementation.

Counting calories is harder, especially about your questions about 
question answers. This is a statistically noisy place and full, it sems, 
of urban legends. The short answers are in the specification. Read the 
definitions carefully and most of what you need pops right out - peer 
offset, delay, jitter and dispersion plus their root derivatives. Folks 
might like to know aout nominal expectations are in the wild; well, a 
few minutes with ntpq can be most entertaining. As for the rest, well, 
there many briefings on the project page and support site; there is a 
faq and, well, even a book.

 From my perception as a teacher, my undergraduate students hate to go 
to briefing documents and even less to a textbook. I read textbooks for 
fun (so thaaats how it works...). Everybody wants to google for 
everything, even when a carefully prepared table of contents and list of 
index terms are available. (I just got 212,000 hits on "autokey 
protocol"; all I checked were for the real thimg.)

Dave

---

From: "TS Glassey" <[email-masked]>
Date: Sun, 4 May 2008 19:52:12 -0700

----- Original Message ----- 
From: "David L. Mills" <[email-masked]>
Cc: <[email-masked]>
Sent: Sunday, May 04, 2008 2:21 PM
Subject: Re: [ntpwg] Testing NTP performance

> specification than they would be in a TCP formal specification.

Duh... but they are very relevant to trhe use of NTP in production 
timekeeping environment's, and that was my point. The protocol as you refer 
to it is a model for exchanging time data but what happens after that is 
what we are talking about. The Jitter of the local system has little to do 
with the protocol either in this case.

Todd

---

From: Danny Mayer <[email-masked]>
Date: Sun, 04 May 2008 22:54:40 -0400

Dave,

I'm not missing the point at all. NTP does it's best to calculate these 
values and it's best estimates of error. If there were a way to get a 
better estimate NTP would use it. Noone has yet come up with a better 
solution. Installing a refclock on the server will give a better idea of 
how good the calculated values are but then you might as well just use 
the refclock.

Danny

---

From: Danny Mayer <[email-masked]>
Date: Sat, 10 May 2008 19:00:41 -0400

> with the protocol either in this case.

All of which tells us that you don't understand NTP at all. Otherwise 
you would know that all the answers are already there in the statistics.

Danny

---

From: "TS Glassey" <[email-masked]>
Date: Sat, 10 May 2008 19:40:30 -0700

----- Original Message ----- 
From: "Danny Mayer" <[email-masked]>
To: "TS Glassey" <[email-masked]>
Cc: "David L. Mills" <[email-masked]>; <[email-masked]>
Sent: Saturday, May 10, 2008 4:00 PM
Subject: Re: [ntpwg] Testing NTP performance

>> timekeeping environment's, and that was my point. The protocol as you 
>> refer to it is a model for exchanging time data but what happens after 
>> that is what we are talking about. The Jitter of the local system has 
>> little to do with the protocol either in this case.
>>
>> Todd
>
> All of which tells us that you don't understand NTP at all. Otherwise you 
> would know that all the answers are already there in the statistics.

You might want to try cutting down on the caffein and maybe get some more 
sleep.