Subject: Re: [ntpwg] Testing NTP performance

From: "STUART VENTERS" <[email-masked]>
Date: Fri, 2 May 2008 09:52:20 -0500

>>Not to get too philosophical, but isn't this a description of the  
>>precise problem that NTP itself is designed to solve?

>>Rob Seaman
>>NOAO
>>--

Rob,

Guilty as charged, there is circular logic here.
   I'm using NTP's ability to transport time to measure same.

But perhaps I can get out on a technicality.
   I'm proposing to use the time transport ability over easy links
       to measure the time transport ability over a more difficult link.

I'd be happy to say that the only unit under test is A
  and so B and C can be 'Golden units'.

I need to test down to sub-millisecond accuracy.
  There are servers available for B with 100nS accuracy.
  There are packet bert testers that could pinch hit as a client C with similar accuracy.
  These should be sufficient to get us to uS accuracy.

Again, this seems like a complicated plan, there ought to be an easier way.

Asked another way, folks here have claimed that NTP runs
 with particular accuracies much tighter
   than a human can check by printing out the time on a console.

How do they know?

Regards,

Stuart

> Greetings,
>
> I'm puzzling over how to test embedded NTP implementations.
>
> Given a client and server connected through some test network,
>   how do you accurately measure the difference between the server  
> and client clocks.
>
> If there was a 1PPS test wire coming out of each implementation, it  
> would be easy.
>   But unless you can add code to the client, there is usually not a  
> wire,
>
> I need some way to peek into the client and server to see their  
> internal clocks.
>
> I'm thinking the server functionality embedded with the client might  
> help.
>
>
> Consider the following system with a third NTP box:
>
>
>                        (C)
>                       CLIENT
>                    /         \
>                  /             \
>                /                 \
>              /                     \
>            /                         \
>          /
>      SERVER ---------- TEST  ------ CLIENT/SERVER
>        (B)            NETWORK           (A)
>
>
>
> We are trying to measure the effects of the test network on the  
> client in A.
>
> Server B provides a stable timing reference for the test.
>
> Client/Server A is locked to Server B through the test network.
>
> Client C is connected to servers A and B.
>
>
> If the links connecting Client C are simple point to point links,
> then hopefully most of the errors in time transport
>   are due to the test network affecting A.
>
> C provides an offset between B&C and also between A&C.
>
> It seems like the difference between these offsets
>  should be the offset between A&B.
>
> This seems like a complicated plan, is there a better way?
>
>
> Regards,
>
> Stuart
>
>
> ps: My apologies for an NTP 101 question.
>     Perhaps someone could point me to where this has already been  
> sorted out.

_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 02 May 2008 16:08:01 +0100

Stuart
>    than a human can check by printing out the time on a console.
My guess is that they make a hardware measurement, it is certainly the 
most accurate way to do it.

In the 1588 bakeoffs I have seen the most popular technique was to look 
at the eye pattern, but I am fairly sure that there is commercial 
equipment that takes the pulses and  measures  mtie, tdev etc, although 
there is concern as to whether these are the most appropriate metric in 
a packet as oppsoed to a synchronous environment.

- Stewart

---

From: "Greg Dowd" <[email-masked]>
Date: Fri, 2 May 2008 08:31:41 -0700

I think the testing methodology is quite dependent on what you are
trying to observe.  Are you looking for the absolute accuracy of a
client with respect to UTC, accuracy wrt server, relative accuracy
between 2 clients, or a pool of clients, etc.  Or, are you testing a
server's ability to put time on a network?  Are you looking for p-p
error, convergence time, rms.  Are you testing some application which
would add its own error on top?

For my testing, I'm usually testing the egress port of a server for
accuracy with respect to the server clock and jitter.  For that, I use
custom designed ntp "probes".  These probes are pseudo clients,
basically boxes with hardware timestamping synchronized to gps or house
pps and then pointed at a single server and querying at 1 packet per
second.  The probes record the statistics using peerstats and rawstats
which I can then load into a custom statistics package we have called
TimeMonitor Analyzer.  Phase plots, histograms, ADEV, TDEV, minTDEV,
MTIE at your fingertips.  After getting a baseline, I can start loading
up ntp traffic, network traffic, etc and continue to sample.

I am not a big fan of just testing pps signals as they don't tell all
the story.  They give you the end to end performance but don't let you
break down the error into discrete components to analyze.  But, it can
be useful.  It isn't too hard to hack linux these days to capture
timestamps or toggle serial lines.

I have also tested the standard performance of linux clients running
ntpd against our servers.  In that case, one great way is to plug a gps
receiver into the slave, with appropriate refclock, and run the query
against the server notrust.  Log the stats and analyze.

If you can't do that, put a known good ntp server right on the switch
port next to your box and then test the server under test with notrust.

If you are testing the client, the easy way is to turn on stats if
possible.  Not absolutely accurate, but they give good data.  

When testing networks, another thing I like to use are Ethernet taps.
These guys let you "sample" the flow at any point when you are testing
loaded networks (a la g.8261).  Then you can map the packet delay
variation, and its subsequent impact on the client servo.  

I have not been overly impressed with the current generation of network
testers (e.g., spirent or ixia) for their ability to generate packets
with timestamps embedded or to manipulate timestamp fields.  But I think
that is getting better.  You really, really have to watch out for how
the testers are configured when setting up test nets.  It is very easy
to get standing wave patterns in your network that trash the test.
Really easy.  Too easy.  And very time-consuming :-)

Greg Dowd
gdowd at symmetricom dot com (antispam format)
Symmetricom, Inc.
www.symmetricom.com
"Everything should be made as simple as possible, but no simpler" Albert
Einstein

> -----Original Message-----
> From: ntpwg-bounces+gdowd=[email-masked] 
> [mailto:ntpwg-bounces+gdowd=[email-masked]] On 
> Behalf Of STUART VENTERS
> Sent: Friday, May 02, 2008 7:52 AM
> To: [email-masked]
> Subject: Re: [ntpwg] Testing NTP performance
> 
> 
> >   how do you accurately measure the difference between the 
> server and 
> > client clocks.
> >
> > We are trying to measure the effects of the test network on 
> the client 
> > in A.
> >
> > If the links connecting Client C are simple point to point 
> links, then 
> > hopefully most of the errors in time transport
> > It seems like the difference between these offsets  should be the 
> > offset between A&B.
> >

---

From: Steve Kostecke <[email-masked]>
Date: Fri, 02 May 2008 11:58:03 -0400

"Greg Dowd" said:

>against the server notrust.

Depending on what version of the NTP Reference Implementation you are
using that may not work the way you think it will.

The behavior of notrust changed between versions 4.1 and 4.2.

* In 4.1 (and earlier) notrust meant "Don't trust this host/subnet for
  time".

* In 4.2 (and later) notrust means "Ignore all NTP packets that are not
  cryptographically authenticated." This forces remote time servers to
  authenticate themselves to your (client) ntpd.

For NTP 4.2.x (and later) use 'noselect' instead of 'notrust' if you
just want to log statistics from a remote time server.

-- 
Steve Kostecke <[email-masked]>
NTP Public Services Project http://support.ntp.org/
Public Key at http://support.ntp.org/Users/SteveKostecke
_______________________________________________
ntpwg mailing list
[email-masked]
https://lists.ntp.org/mailman/listinfo/ntpwg

---

From: "TS Glassey" <[email-masked]>
Date: Fri, 2 May 2008 08:56:07 -0700

Greg -

----- Original Message ----- 
From: "Greg Dowd" <[email-masked]>
To: "STUART VENTERS" <[email-masked]>; <[email-masked]>
Sent: Friday, May 02, 2008 8:31 AM
Subject: Re: [ntpwg] Testing NTP performance

> would add its own error on top?

Remember that NTP is a SW product as it is distributed today from the land 
of Oz and that SW is by definition 'an emulation of something' - an idea we 
seem to have forgotten across our culture today.

> second.

Greg does Symmetricom publish the results of this testing?

> The probes record the statistics using peerstats and rawstats

Yes but this is only true if the local network is set up for portspanning 
between the TAP Port and the TEST Port. Most people today use switches for 
their timing service networks and that means that the ports isolate 
same-subnet traffic to the specific ports it is flowing to/from on that 
switch. So unless portspanning is used to turn the listener port on properly 
or a Hub is used instead of a switch, this wont necessaryily work.

---

From: "Greg Dowd" <[email-masked]>
Date: Fri, 2 May 2008 09:16:50 -0700

You're absolutely correct.  I'm showing my ntp age :-)

Greg Dowd

> Behalf Of Steve Kostecke
> Sent: Friday, May 02, 2008 8:58 AM
> To: [email-masked]
> Depending on what version of the NTP Reference Implementation 
> you are using that may not work the way you think it will.
> 
> For NTP 4.2.x (and later) use 'noselect' instead of 'notrust' 
> if you just want to log statistics from a remote time server.
> 
> --
> NTP Public Services Project http://support.ntp.org/ Public 
> Key at http://support.ntp.org/Users/SteveKostecke

---

From: "Greg Dowd" <[email-masked]>
Date: Fri, 2 May 2008 09:24:43 -0700

Greg Dowd

> From: TS Glassey [mailto:[email-masked]] 
> Sent: Friday, May 02, 2008 8:56 AM
> To: Greg Dowd; STUART VENTERS; [email-masked]
NTP is a protocol, not software.  It is defined by RFC1305 or 4330.  The
implementation is something different to me.  Closely aligned but not
the same.  ntpd is software and I agree with you that we often confuse
the two.
> Greg does Symmetricom publish the results of this testing?
Yes we do publish the results.  That's more a marketing function but the
results show up in datasheets, white papers, presentations, etc.  We do
joint papers with various partners as well.  But, as I mentioned, we're
mostly testing our egress performance so it is not germane to most
people.

> or a Hub is used instead of a switch, this wont necessaryily work.
Most of my stuff has moved into the Gigabit domain now and, while the
standard supports it, I don't think anyone makes a GBE hub :-)  I'm
talking about a standalone tap (www.networktaps.com).  They work
anywhere.  Handy things to have around the lab.

---

From: "TS Glassey" <[email-masked]>
Date: Fri, 2 May 2008 09:57:03 -0700

Gee Greg - I didn't think you spoke to me anymore...

NTP is BOTH a package and a physical data-exchange protocol but I thought 
that was self-evident to anyone on this list. The NTP transaction process 
protocol - designed by Mills and his teams originally was implemented in a 
SW model to prove its functionality but the term NTP is generally used to 
refer to the NTP Daemon and the protocol it runs.

My assertion here is that NTP has become sorta like the term XEROX which 
pertains to a process and the actual copy itself. Notice that when any other 
part of the package is discussed (SNTP or NTPDATE for instance) these are 
clearly enumerated.

As to the testing you talk about its good its in papers but those need to 
come with the product itself as part of the performance specifications for 
the unit and not as a part of a whitepaper per se IMHO.

Todd

----- Original Message ----- 
From: "Greg Dowd" <[email-masked]>
To: "TS Glassey" <[email-masked]>; "STUART VENTERS" 
<[email-masked]>; <[email-masked]>
Sent: Friday, May 02, 2008 9:24 AM
Subject: RE: [ntpwg] Testing NTP performance