Subject: [Ntp] Antw: [EXT] Minutes from IETF 108 NTP WG session

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 03 Aug 2020 08:17:08 +0200

Hi!

As I read claims like the following from time to time:
"It turns out that for trades a nanosecond resolution matters."

I wonder what's really behind that claim: Reading the current system time on a
current Linux server yields in about 1µs precision (independent from how the
clock is synchronized). Asuming that reading the time is much faster than any
database transaction, I wonder whether any trade transaction can be completed
in less than 1 nanosecond. Or maybe they are using the nanoseconds just for
serializing seemingly parallel operations. If so the nanoseconds actually would
be "cheats".
(OpenLDAP with multi-master replication uses and incremented microsecond for
event ordering; obviously as even OpenLDAP cannot handle a database
modification is less than a few microseconds)

Example for such "timestamps" from OpenLDAP:
20200722085045.000018Z
20200722085045.000019Z

So before claiming that financial industry needs sub-nanosecond resolution,
I'd like to see hard facts and details.
The other thing is: How small would the stability of the clock have to be to
make any sub-nanosecond resolution be realistic? 0.001 PPM?
Real "normal" servers will probably have 1 PPM, and a "certified chronograph"
will allow around 50 PPM (AFAIK)...

Regards,
Ulrich

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 3 Aug 2020 12:36:18 +0200

On Mon, Aug 03, 2020 at 08:17:08AM +0200, Ulrich Windl wrote:
> current Linux server yields in about 1Âµs precision (independent from how the
> clock is synchronized).

On modern computers running Linux, at least for the last decade, the
time it takes to read the system clock is usually less than 50
nanoseconds. I have not seen a sub-10ns clock yet. That's still an
order of magnitude to get into the sub-nanosecond range. Even if the
CPU was so fast that it could read and convert the TSC counter in less
than a nanosecond, there is no POSIX API to return the timestamp in a
sub-nanosecond resolution.

I guess for hardware clocks (e.g. NICs that support hardware
timestamping) a sub-nanosecond resolution might be needed sooner than
for the system clock.

NTP has a 1/4ns resolution. For the White Rabbit that might not be
good enough. For synchronizing ordinary computers (in financial
industry or other) I think that will be sufficient for quite some
time. In any case, a correction extension field (which will need to be
supported in switches/routers in order to get a sub-nanosecond
accuracy) can have a better resolution than the timestamps in the NTP
header.

> I'd like to see hard facts and details.

FWIW, I have not heard anyone asking for a sub-nanosecond resolution
yet. The accuracy of the clock is a bigger issue.

> will allow around 50 PPM (AFAIK)...

For sub-nanosecond stability with an ordinary computer clock, the
update interval would need to very short, e.g. few milliseconds.

In a test I did with two directly connected idle computers and a
polling interval of -6 (using the interleaved mode), the reported
stability was just a few nanoseconds. With a shorter interval it might
be possible to get it under 1 nanosecond, but I'm not sure what would
be the point when the precision and accuracy of the clock is in tens
nanoseconds at best.

-- 
Miroslav Lichvar

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 03 Aug 2020 13:01:02 +0200

>>> Miroslav Lichvar <[email-masked]> schrieb am 03.08.2020 um 12:36 in
Nachricht <20200803103618.GK762467@localhost>:
>> I wonder what's really behind that claim: Reading the current system time
on 
> a
> nanoseconds. I have not seen a sub‑10ns clock yet. That's still an

Hi!

I'm surprised.

I wrote an algorithm like this to "measure":

typedef struct timespec ts_t;           /* short-cut for struct timespec */
static  long    get_res(ts_t *tsp)
{
        int             result;
        ts_t            samples[CLOCK_SAMPLES];
        unsigned        u;

        if ( (result = clock_getres(CLOCK_MONOTONIC, tsp)) != 0 )
                goto leave;
        assert(tsp->tv_sec == 0);
        for ( u = 0; u < CLOCK_SAMPLES; ++u )
        {
                get_time(samples + u);
        }
        for ( u = 1; u < CLOCK_SAMPLES; ++u )
        {
                samples[u - 1] = subtract_time(&samples[u], &samples[u - 1]);
        }
        qsort(samples, CLOCK_SAMPLES - 1, sizeof(samples[0]), cmp_time);
        if ( samples[0].tv_nsec > tsp->tv_nsec )
                tsp->tv_nsec = samples[0].tv_nsec;
leave:  return(result);
}

On one machine I got these debugging messages:
(D)get_res: resolution is 0.000000001 ## from POSIX call
(D)get_res: smallest delta is 0.000001514
(D)get_res: largest delta is 0.000001695
(I)set_resolution: ts 0.000001514, digits 6, nsecs 1000

So your machines must be way faster... OK, I tried the latest generation we
have:
(D)get_res: resolution is 0.000000001
(D)get_res: smallest delta is 0.000000030
(D)get_res: largest delta is 0.000000100
(D)set_resolution: ts 0.000000030, digits 8, nsecs 10

> order of magnitude to get into the sub‑nanosecond range. Even if the
> sub‑nanosecond resolution.
> 
> timestamping) a sub‑nanosecond resolution might be needed sooner than
> for the system clock.
> 
> supported in switches/routers in order to get a sub‑nanosecond
>> So before claiming that financial industry needs sub‑nanosecond
resolution,
> FWIW, I have not heard anyone asking for a sub‑nanosecond resolution
>> make any sub‑nanosecond resolution be realistic? 0.001 PPM?
> For sub‑nanosecond stability with an ordinary computer clock, the
> polling interval of ‑6 (using the interleaved mode), the reported

My guess is that your oscillator was much better than standard, TCXO and OCXO,
or you have a very stable temperature around.

Regards,
Ulrich

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 3 Aug 2020 13:21:30 +0200

> (I)set_resolution: ts 0.000001514, digits 6, nsecs 1000

This suggests the system is not using the TSC clocksource, but
something slower like HPET. Check the following files:

/sys/devices/system/clocksource/clocksource0/current_clocksource
/sys/devices/system/clocksource/clocksource0/available_clocksource

> So your machines must be way faster... OK, I tried the latest generation we

No, they are not faster, just using the TSC clocksource. On some
machines it's unreliable, so the kernel may need to switch to a slower
clocksource. IIRC even the first Intel Core and AMD Athlon 64 CPUs
from the 2000s were about 50ns or better. Make sure the CPU is running
at full speed (e.g. disabled ondemand governor).

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Mon, 03 Aug 2020 05:12:06 -0700

> (D)get_res: smallest delta is 0.000000030

The first looks a little slow to me.  The second is typical of Linux on a 
reasonably modern Intel CPU.

There is a mechanism for reading the clock without doing a syscall.  man vdso 
for more info.

There is a "clocks" program in ntpsec/attic/

On recent Intel:
      res   avg      min  dups  CLOCK
        1    23       22        CLOCK_REALTIME
  1000000     7  1000012   -11  CLOCK_REALTIME_COARSE
        1    20       21        CLOCK_MONOTONIC
        1    22       27        CLOCK_MONOTONIC_RAW
        1    20       23        CLOCK_BOOTTIME

The -11 in the dups column means that all but 3 samples got the same time as 
the previous sample.

Histogram: CLOCK_REALTIME, 1 ns per bucket, 1000000 samples.
        ns      hits
        19    177480
        20    360015
        21    423042
        22     37224
        23        76
        24        46
        25        42
        26        45
        27        50
        28        60
1920 samples were bigger than 28.

Fedora on a Pi 3 in 64 bit mode.
      res   avg      min  dups  CLOCK
        1    77       52        CLOCK_REALTIME
 10000000    42  9999942    -3  CLOCK_REALTIME_COARSE
        1    69       52        CLOCK_MONOTONIC
        1    75       52        CLOCK_MONOTONIC_RAW
        1    69       52        CLOCK_BOOTTIME

Histogram: CLOCK_REALTIME, 1 ns per bucket, 1000000 samples.
        ns      hits
        52    647881
        53     58686
       104    244716
       105     48709
       157         2
       208         1
       209         2
3 samples were bigger than 2551.

Note the histogram takes big jumps.  The hardware clock ticks at 52 ns.  
Actually, a bit slower since a fraction of the ticks report 53 rather than 52.

Fedora on a Pi 3, 32 bit mode:
No vdso, so it's pretty slow.
      res   avg      min  dups  CLOCK
        1   891      833        CLOCK_REALTIME
  1000000   749  1000001  -717  CLOCK_REALTIME_COARSE
        1   906      833        CLOCK_MONOTONIC
        1   896      833        CLOCK_MONOTONIC_RAW
        1  1229     1145        CLOCK_BOOTTIME

Histogram: CLOCK_REALTIME, 1 ns per bucket, 1000000 samples.
        ns      hits
       833    289745
       834    145467
       885    326491
       886    235960
       937       294
       938       280
       989        60
       990        84
      1041        55
      1042       100
1464 samples were bigger than 1042.

Again, the steps are all multiples of 52 (or a bt over).

-- 
These are my opinions.  I hate spam.

---

From: Hal Murray <[email-masked]>
Date: Mon, 03 Aug 2020 05:12:41 -0700

> This suggests the system is not using the TSC clocksource, but something
> slower like HPET. Check the following files: 

Maybe, but I'll bet a bigger contribution is doing a syscall rather than using 
vsdo.

Onc thing to check for is a delta of 0.  If the hardware clock is slow enough 
and you can read it fast enough, you can read 2 consecutive samples before the 
clock ticks.

----------

I have a pair of machines with Intel Atom 230 and 330 CPUs.  The 330 has 2 
cores.  Same CPU speed and such.

Atom 230:
      res   avg      min  dups  CLOCK
        1   107      105        CLOCK_REALTIME

Atom 330:
        1  1013      558        CLOCK_REALTIME

The 330 logs the following at boot time:
  kernel: Measured 156 cycles TSC warp between CPUs, turning off TSC clock.
  kernel: Unstable clock detected, switching default tracing clock to 
"global"#012If you want to keep using the local clock, then add:#012  
"trace_clock=local"#012on the kernel command line

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 3 Aug 2020 15:11:02 +0200

> Maybe, but I'll bet a bigger contribution is doing a syscall rather than using 

That depends on hardware and kernel configuration. Looking at some of
my old tests, the overhead of the syscall was much smaller then
reading of the HPET itself. However, on Intel CPUs some of the
security mitigations (KPTI) made the syscall about 3x slower, so I
guess it could be now comparable or even larger than the HPET reading.

-- 
Miroslav Lichvar