Subject: Re: [art] Predictable Internet Time

From: Philip Homburg <[email-masked]>
Date: Tue, 28 Mar 2017 16:05:37 +0200

In your letter dated Mon, 27 Mar 2017 12:34:10 -0700 you wrote:
>I've submitted the time frame discussion intended to resolve this issue,
>which also recently arouse on another mailing list. Further discussion
>on this draft will occur on the ART mailing list ([email-masked]).

A few random comments. I think a document like this is useful, though it
might hard to strike the right balance between providing enough and too much
detail.

- There are two types of solar time. The actual solar time where the sun is 
at the highest point at noon, and mean solar time. Section 4 does not make
that distinction, but it is mentioned in a later section. After far as I know,
actual solar time is extremely rare (only sundials). Solar time is almost
always mean solar time. I.e UT0 is already mean solar time.

- UTC (and TAI) doesn't really exist. They are realized at various labs all
  over the world. The real value of UTC is coordinated only after the fact.
  This makes a statement that GPS time differs 25 ns from TAI a bit weird.
  Maybe this is too much detail. 

- It is not clear to me why NTP would differ 100ms from TAI.

- The epoch for NTP is 1900-01-01.

- It is not clear to me what 'Unix time' is in this context. Typically POSIX
time is linked to UTC, i.e. every value of time_t corresponds to a specific
UTC timestamp. The inverse is not true, leapseconds cannot be represented in
time_t.

- I can't find any reference where the epoch of TAI is defined as 1977-01-01.

- Section 5.1.3 says 'Similarly, the earth's orbit around the sun varies and
is slowing over time, resulting in an increasing stretching of a solar second.'
It is the rotation of the earth around it's own axis that causes the
solar second to become longer over time, not the rotation of the earth 
around the sun.

- One thing I'd like to add is 'uptime' or more general, monotonic time.
Many network protocols need timers and timeout. There is however no need 
to reference these to UTC, or TAI, etc. So computing time intervals using
uptime is in many cases better because that also works if an external time
reference doesn't become available until long after booting.

- Another thing worth mentioning explictly is that for the distant future,
there is no way to convert between TAI and UTC because future leapsecond are
not yet defined. So TAI is good for computing intervals, except if it involves
timestamps more than a few months in the future.

---

From: Tony Finch <[email-masked]>
Date: Tue, 28 Mar 2017 16:13:49 +0100

Philip Homburg <[email-masked]> wrote:
> A few random comments.

Good comments :-)

> at the highest point at noon, and mean solar time.

The usual term for the former is "apparent solar time". If you are talking
about both then it's worth saying the difference between apparent and mean
solar time is called the equation of time.

> - One thing I'd like to add is 'uptime' or more general, monotonic time.

An interesting discussion re. compatible APIs for elapsed time:
https://github.com/golang/proposal/blob/master/design/12914-monotonic.md

> timestamps more than a few months in the future.

I think "distant future" isn't a very good description for "a few months" :-)

The actual horizon is about 5 - 11 months, just before and just after
bulletin C is published in January or July.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
North Hebrides, Bailey, Fair Isle, Faeroes, Southeast Iceland: Easterly 5 to
7, occasionally gale 8 in western Southeast Iceland. Moderate or rough.
Occasional drizzle. Good, occasionally poor.

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2017 11:11:59 -0700

Hi, Philip,

> might hard to strike the right balance between providing enough and too much
Yeah - I struggled with that. I don't think everyone will be happy with
any particular boundary...

> always mean solar time. I.e UT0 is already mean solar time.
Section 3 states that in the definition of a solar day. Section 4.2
explains that UT0 is one of many different "means", and that UT0 is the
most common in current use.

>   over the world. The real value of UTC is coordinated only after the fact.
Although that's strictly true, there are also UTC and TAI sources that
provide the current time as close to UTC and TAI as known. I can add that.

>   Maybe this is too much detail. 

See above - even in retrospect, GPS can differ from TAI by 25 ns (it's
in the spec for GPS). GPS isn't one of the clocks averaged into TAI
(AFAICT); it's a separate source that's sync'd to TAI to ensure the 25
ns max delta.

> - It is not clear to me why NTP would differ 100ms from TAI.
It's in the spec.

> - The epoch for NTP is 1900-01-01.
Yup - my error. WIll fix.

> time is linked to UTC, 

That is incorrect. POSIX time is defined as 1/86400 of a day, which does
not take into account leap seconds at all (UTC does) and "day" is not
defined as related to SI units (UTC is). AFAICT, a POSIX "day" is at
best a rough approximation of UT0.

> i.e. every value of time_t corresponds to a specific
See above, I think.

> - I can't find any reference where the epoch of TAI is defined as 1977-01-01.
It's from the updated definition of TAI, the one that started to take
account for time dilation and gravitational influences.
I'll see if I can find a citable ref for that.

> solar second to become longer over time, not the rotation of the earth 
I'll fix that...

> - One thing I'd like to add is 'uptime' or more general, monotonic time.

Uptime is just a printout of the delta of the POSIX time when a computer
started and the current POSIX time.

It might be useful to indicate that some computers just start their
clocks at boot as zero.

> reference doesn't become available until long after booting.

How/when to sync a "clock" to a time scale is important to note, but the
details are implementation issues.

"uptime" isn't used for timers; internal clocks are, e.g., the POSIX
time scale.

> timestamps more than a few months in the future.
The doc mentions in various places that updated leapsecond information
is required to maintain UTC or derive TAI from UTC. I can mention that
we don't *expect* leapseconds to be added without at least a few months'
notice, and that this affects indicating future dates accurately.

Joe

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2017 11:21:02 -0700

> solar time is called the equation of time.

Will do.

> https://github.com/golang/proposal/blob/master/design/12914-monotonic.md
This is another example of the confusion that results when the time
scale is not clearly indicated or defined. Having Go read the "system
wall clock" is the source of the problem, and this doc explains a
variety of solutions.

I can add a new "monotonic clock" definition that uses an arbitrary
"tick" that is guaranteed to increase between reads and across reboots,
but is not comparable across machines.

> bulletin C is published in January or July
Agreed, but not sure we need that level of detail.

Joe

---

From: Michael Thornburgh <[email-masked]>
Date: Tue, 28 Mar 2017 18:41:12 +0000

> From: art [[email-masked]] on behalf of Philip Homburg [[email-masked]]
> Sent: Tuesday, March 28, 2017 7:05 AM

[...]
> UTC timestamp. The inverse is not true, leapseconds cannot be represented in
[...]

while this has been the case so far, this is not true in general.  there can be negative leap
seconds, for which there will be values of time_t with no corresponding UTC time.
fortunately there have not been any negative leap seconds yet.

as i mentioned on a different mailing list [0], unix/posix time is a (deficient) time representation form,
but people use it as a time keeping form (as if it's actually the "uptime" since the unix
epoch).  i'm sure it was initially envisioned as both, even though that's wrong.

really unfortunately (or at least annoyingly) for timekeeping, "1970-01-01 00:00:00 UTC" isn't
a whole number of seconds offset from TAI.  UTC's ticks weren't synchronized to TAI ticks until
1972-01-01 00:00:10 TAI / 1972-01-01 00:00:00 UTC.  so having 1970-01-01 00:00:00 UTC
as a timekeeping epoch is the wrong choice anyway.  as i indicated in [0], i think the
right choice is 1970-01-01 00:00:10 TAI as the timekeeping epoch.

because solar time is variable, the standardized second is useful, and it is useful for
civil time to be in close agreement with solar time, there will need to be leap seconds or
at least periodic civil time steps.  therefore, the representation of civil/human time in something
like a time_t or a struct timeval is wrong, especially for future times.  the correct choice should
represent civil/human time in something like a struct tm or ISO 8601 or something.

[0] https://lists.w3.org/Archives/Public/ietf-http-wg/2017JanMar/0424.html

---

From: Nico Williams <[email-masked]>
Date: Tue, 28 Mar 2017 13:43:02 -0500

> wall clock" is the source of the problem, and this doc explains a

Better data types -and strong typing- are needed:

 - Unix-like TAI (seconds since epoch, admitting not leap seconds)
 - Unix-like UTC (seconds since epoch, with leap seconds)
    - both with variations like real versus integer, or seconds +
      microseconds, etc..
 - broken-down time without leap seconds
 - broken-down time with    leap seconds
 - formatted time (with a zone indicator to indicate whether UTC or TAI)
 - perhaps other forms

Then there's intervals.  Having variations on intervals based on whether
or not they admit leaps seems silly though, so intervals are a bit
easier.

Nico
--

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2017 11:47:20 -0700

Hi, Nico,

> Better data types -and strong typing- are needed:
Unix defines only POSIX time right now, FWIW.

AFAICT, wouldn't specing Unix interfaces to the different times below be
out of scope for the IETF?

---

From: Nico Williams <[email-masked]>
Date: Tue, 28 Mar 2017 14:00:46 -0500

> Unix defines only POSIX time right now, FWIW.

POSIX defines POSIX time.  Unix and Unix-like systems use it.

(Nothing stops any Unix-like OS from including extensions.)

> out of scope for the IETF?

Yes and no.

First, the IETF has and does specify some APIs.  (BSD sockets bindings
for IPv6, GSS-API, and various others.)

Second, the IETF could define abstract APIs but refrain from specifying
C bindings for APIs and recommend to the Open Group and others that they
standardize those based on the abstract APIs.  (GSS-API, for example,
defines both, abstract and programming-language-specific bindings.)

Regardless, how could we address our time issues without specifying what
amounts to.. type conversions?  That's an API by whatever name you wish
to call it.  Best call it an API, define "types" and "type conversions"
or "functions" that do the same.

Obviously, for leap second smearing, besides an API there is also
behavior (smearing).

There's some difficulty with typing smeared time in that a smeared time
would be closer to UTC than POSIX time, but *pretending to be POSIX
time*.  Which makes it difficult to have strong typing (which you get
becomes a run-time matter!), but makes it easy to inject the new type in
existing object code.

But it may still be necessary to convert smeared time (when you know
that's what it is) to other time forms, and there has to be a canonical,
reproduceable, non-host-dependent way to do it, else you end up with a
synchronization problem.

Nico
--

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2017 12:07:38 -0700

Hi, Nico,

On 3/28/2017 12:00 PM, Nico Williams wrote:
> ...
Agreed.
> for IPv6, GSS-API, and various others.)

Only network APIs, AFAICT>

> amounts to.. type conversions? 
*our* time issues are mostly that we need to know the time frame being
used, and to appreciate that there's no single solution that can be used
to calculate intervals AND correspond to civil time that avoids the need
for constantly-updated external information. No closed system is
sufficient for that.

> ...

I sincerely hope we can stop referring to this nonstandard technique. We
certainly cannot even consider standardizing a conversion (even if that
were our purview) between time scales that include smearing until there
are smearing standards.

A key point in this doc is that smearing makes things horribly worse,
not better in any way, except to delude users into thinking otherwise.

Joe

---

From: Nico Williams <[email-masked]>
Date: Tue, 28 Mar 2017 14:19:15 -0500

> Only network APIs, AFAICT>

Network protocols often depend on or relate to time, thus there's a
nexus.

> for constantly-updated external information. No closed system is

If you can keep track of who needs time in what form...

And if you can't... then what's the point of adding PIT?

Agreed.

> not better in any way, except to delude users into thinking otherwise.

Agreed!

Nico
--

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2017 12:41:42 -0700

> Network protocols often depend on or relate to time, thus there's a
Mostly they depend on continuous time, which is already pointed out.

> And if you can't... then what's the point of adding PIT?

I don't think there is. Managing time across systems requires some sort
of coordination, either to unify time units or entire time scales. Once
you recognize that, it's sufficient to use UTC + locale.

Joe

---

From: Tony Finch <[email-masked]>
Date: Tue, 28 Mar 2017 21:12:04 +0100

> most common in current use.

No, UT1 is the most common form of mean solar time. (UT0 is basically
mean solar time at a particular observatory; UT1 is corrected for polar
motion so it is globally consistent.)

> GPS isn't one of the clocks averaged into TAI (AFAICT); it's a separate
> source that's sync'd to TAI to ensure the 25 ns max delta.

Well, GPS is an ensemble clock (each sat has its own clock) which is
steered with reference to UTC(USNO). UTC(USNO) feeds into the BIPM paper
clocks.

The USNO says that GPS's maximum deviation from UTC(USNO) is 1us though
in practice it is within a few hundred nanoseconds. The NAV message has
additional information that allows GPS receivers to get the UTC(USNO) time
to within 40ns.

http://www.usno.navy.mil/USNO/time/gps/usno-gps-time-transfer

> > - It is not clear to me why NTP would differ 100ms from TAI.

Where in which spec?

> started and the current POSIX time.

CLOCK_MONOTONIC cannot be reset, so it is more reliable than that.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--zr8h punycode
Trafalgar: In southeast, variable 3 or 4. In northwest, southerly 5 to 7. In
southeast, slight or moderate. in northwest, moderate or rough. In southeast,
fair. In northwest, rain or showers. Good, occasionally poor.

---

From: "Manger, James" <[email-masked]>
Date: Tue, 28 Mar 2017 22:56:12 +0000

>> leap second smearing

> We certainly cannot even consider standardizing a conversion (even if that

It looks like the most valuable IETF contribution would be standardizing one smearing scheme (& getting Google, Java, ... to agree to it).

> not better in any way, except to delude users into thinking otherwise.

Given that a well-defined smearing scheme allows a simple and precise mapping to UTC, it is hard to see how that can be "horribly worse".

Smearing moves the complexity of solar-vs-atomic-vs-civic time and the resultant leap seconds to a place where a huge number of systems can correctly handle that complexity by "doing nothing", while systems that care if time intervals can be 0.001% wrong (if 24hr is the smear standard) can still be more precise by making well-defined adjustments.

--
James Manger

---

From: Joe Touch <[email-masked]>
Date: Tue, 28 Mar 2017 16:11:27 -0700

> It looks like the most valuable IETF contribution would be standardizing one smearing scheme (& getting Google, Java, ... to agree to it).

Ick. IMO, the most valuable contribution is to recommend the use of TAI
or UTC, IMO.

> Given that a well-defined smearing scheme allows a simple and precise mapping to UTC, it is hard to see how that can be "horribly worse".
Even if we say that all interval calculations and conversions are
equally messy or not, why introduce yet another standard when we already
have TAI and UTC?
> Smearing moves the complexity of solar-vs-atomic-vs-civic time and the resultant leap seconds to a place where a huge number of systems can correctly handle that complexity by "doing nothing", whil…

Doing nothing means smeared times are always wrong. When that smeared
time corresponds to a day or year change in another time zone, you've
really messed things up. That means doing nothing either means your
wrong or you're really wrong. The only solution is to convert out of the
smear back to UTC, but then why bother smearing at all?

Joe

---

From: Philip Homburg <[email-masked]>
Date: Wed, 29 Mar 2017 13:36:49 +0200

[Just responding to text in many different e-mails]

>most common in current use.

Very confusing to have two completely different definitions of Solar day in one
item. Later you use solar time sometimes as mean, sometimes as 'apparent'.

>(AFAICT); it's a separate source that's sync'd to TAI to ensure the 25

Given the complexities of dealing with time at the nanosecond level, maybe
it is better to ignore those issues in this document. I.e, GPS and Galilelo
are synced to TAI (with some fixed offset). GLONASS tracks UTC. And leave it
at that.

>> - It is not clear to me why NTP would differ 100ms from TAI.

Can you be more specific.

>best a rough approximation of UT0.

I was going to propose that for this document we treat POSIX time as
NTP time minus 2208988800. Unfortunately, the latest NTP RFC (5905) doesn't
seem to actually define NTP time. So that doesn't help.

>But as others have pointed out, that is either a lie or an error: POSIX
>simply does not and cannot account for leap seconds in a numeric type
>representing seconds since an epoch.  Perhaps we can say that the
>reference to UTC was meant to be to TAI.  Either way the definition of
>seconds can only plausibly be SI.

The way I read it, the following formula

tm_sec + tm_min*60 + tm_hour*3600 + tm_yday*86400 +
    (tm_year-70)*31536000 + ((tm_year-69)/4)*86400 -
    ((tm_year-1)/100)*86400 + ((tm_year+299)/400)*86400

converts between a time_t value and a broken down UTC time. Just about
every piece of software assume this to be true. I.e, if you run ls then
the times shown by ls are based on this formula.

Of course this implies that time_t is horribly broken in many different ways,
but we already know that.

So if you subtract two time_t values, you don't get the real number of
seconds between the two points of time they represent because leapseconds
are ignored. If tm_sec == 60, i.e. during a leap second, you get the same
time_t value as the first second of the next minute.

>It might be useful to indicate that some computers just start their

The main thing is that monotonic time can never go backward. Ntpd will hapily
step the (POSIX) time backwards. So you need a separate clock, which POSIX 
calls CLOCK_MONOTONIC (see clock_settime).

>but is not comparable across machines.

Note that monotonic time is typically not maintained across reboots.

> - Unix-like TAI (seconds since epoch, admitting not leap seconds)

If we can get the opengroup to add CLOCK_TAI then we can have this. One
problem is that there doesn't seem to be a sensible epoch for TAI.

>    - both with variations like real versus integer, or seconds +

I'd like to treat time_t as something that is synchronized with UTC and behaves
rather weird around a leap second.

> - broken-down time without leap seconds

Not sure why that would be desirable, but that's what have now.

> - broken-down time with    leap seconds

A 'tai_gmtime' could do that.

>or not they admit leaps seems silly though, so intervals are a bit

If you want to set a timer based on an interval, then it may make sense to
know if leap seconds need to be counted or not.

>The obvious argument against smeared time is that we already have _two_
>time standards, and we don't need a third.  All we need is to carefully
>pick one or the other in each specification, and preferably TAI time in
>all cases.

There is (right now) no meaningful way to convert 2020-01-01 00:00:00 UTC 
to a TAI time value.

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Mar 2017 08:45:48 -0700

> motion so it is globally consistent.)
Yes - mistyped (the doc is correct on that - it says UT1 is the
particular mean that is in common use).

> http://www.usno.navy.mil/USNO/time/gps/usno-gps-time-transfer

The GPS spec guarantees that its max deviation is 25 ns AFAICT (see the
cited ref in the doc).

>>> - It is not clear to me why NTP would differ 100ms from TAI.

It's in the definition of NTP strata. Stratum 0 is the a reference clock
itself; stratum 1 is connected to the clock directly, stratum 2 is
connected to stratum 1 over the net. I'll dig up a specific citation for
that...

> CLOCK_MONOTONIC cannot be reset, so it is more reliable than that.
It is guaranteed monotonic only while the system is running. I don't
know what "reliable" means, but it isn't guaranteed to be preserved
across a boot.

http://pubs.opengroup.org/onlinepubs/9699919799/functions/clock_getres.html

Joe

---

From: Joe Touch <[email-masked]>
Date: Wed, 29 Mar 2017 09:24:52 -0700

> item. Later you use solar time sometimes as mean, sometimes as 'apparent'.

That's on my list to fix (I'm typing that summary to post to the list in
a minute or two)...
> Can you be more specific.

I'm tracking that down - it's because of the strata.

> seem to actually define NTP time. So that doesn't help.

I think I see the issue here.

Unix time is a count since epoch, but never defines second or day. It
emulates TAI, but there's no definition of how it drifts (or not) from TAI.

POSIX *time* is this Unix time.

The POSIX time API allows access to two different clocks: a local
realtime clock and a local monotonic clock. The API *approximates* UTC
by trying to convert its internal time to what it thinks is UTC. When
that realtime clock is updated by NTP, the API can present an accurate
UTC value. When not, it likely presents something that is TAI+offset,
because it doesn't track leaps.

It's important to differentiate these; I've been talking almost
exclusively about POSIX *time*, not the POSIX time API.

> converts between a time_t value and a broken down UTC time. 
See above. This is just an *approximation* that tries to convert Unix
time to something close to UTC, but because it doesn't track leaps, it
really emulates TAI.

> Just about
If time_t is an integer (i.e., you're not referring to the struct tm or
some other API), then it is supposed to be the Unix time...

> are ignored. 
You do (or should) get the real seconds.

You can't subtract two struct tm's that way, though.

> The main thing is that monotonic time can never go backward. 

It can on reboot.

> Ntpd will hapily
We really need to differentiate between the POSIX time scale and the
POSIX time API. You're talking about the POSIX time API. POSIX time
itself should never go backwards...

> Note that monotonic time is typically not maintained across reboots.
I'll point that out too..

> problem is that there doesn't seem to be a sensible epoch for TAI.
TAI defines its epoch as 1977-01-01

When you say "sensible", is the problem in representing times before
that epoch? They can be denoted as negatives...

> rather weird around a leap second.

time_t should be the Unix time, which can't sync to a time scale with
leaps like UTC

the POSIX time API does have a struct tm, which might derive a UTC
approximation.

> know if leap seconds need to be counted or not.
That would depend IMO on whether the timer is supposed to trigger on an
exact date or at the end of an exact interval. That's worth noting.

> There is (right now) no meaningful way to convert 2020-01-01 00:00:00 UTC 

True, but that's also because that UTC date is indeterminate.  Until you
know the leaps between now and then, you don't know when that date will
actually occur.

Joe