Subject: Re: [Ntp] Leap second draft

From: Watson Ladd <[email-masked]>
Date: Thu, 23 Jan 2020 15:03:45 -0800

On Wed, Mar 27, 2019, 9:47 AM Daniel Franke <[email-masked]> wrote:

> I'm writing up an I-D clarifying how NTP implementations are to behave
> in proximity to a leap second and also introducing an extension field
> that provides better leap-second-related information than is
> expressible using header fields alone, including giving the TAI-UTC
> offset.
>
> Karen, you mentioned a possible CfA for
> draft-stenn-ntp-extended-information if Harlan revises it to comply
> with RFC 7822 (-04 does not appear to comply). Our two drafts are
> partly redundant in purpose since they both provide TAI-UTC offset and
> we won't want to adopt both. Of course I think mine is the one we
> should adopt, but I'll strive to get a -00 submitted promptly so that
> they can be considered side-by-side without unduly delaying Harlan's
> CfA.
>

Apologies for the necromancy but someone at work just asked me which second
is repeated in an NTP timestamp and to make sure all our sources use the
right one.

This draft is needed.

>
> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp
>

---

From: Warner Losh <[email-masked]>
Date: Thu, 23 Jan 2020 16:09:26 -0700

On Thu, Jan 23, 2020 at 4:04 PM Watson Ladd <[email-masked]> wrote:

> Apologies for the necromancy but someone at work just asked me which
> second is repeated in an NTP timestamp and to make sure all our sources use
> the right one.
>

NTP timestamp repeats the last second of the day. Once without the leap bit
set and then again with it set.

POSIX doesn't specify, and so systems choose a number of different ways of
coping (repeat the last second of the day, repeat the first second of the
next day, freeze time during leap second, do some crazy smear to pretend
reality matches the POSIX spec, kernel panic, etc).

Warner

---

From: Warner Losh <[email-masked]>
Date: Thu, 23 Jan 2020 16:36:06 -0700

On Thu, Jan 23, 2020 at 4:09 PM Warner Losh <[email-masked]> wrote:

> NTP timestamp repeats the last second of the day. Once without the leap
> bit set and then again with it set.
>

Ah, the standard doesn't say this...  It just says that the values can be
subtracted, so forget I said anything about NTP

> reality matches the POSIX spec, kernel panic, etc).

This part is true. It's the biggest flaw with time_t: It doesn't
acknowledge that leap seconds exist, so what to do during that thing which
the standard says doesn't exist is up for grabs :(

Warner

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 24 Jan 2020 07:51:24 +0100

On 24 Jan 2020, at 0:36, Warner Losh wrote:

>> POSIX doesn't specify, and so systems choose a number of different ways of coping (repeat the last second of the day, repeat the first second of the next day, freeze time during leap second, do some crazy smear to pretend reality matches the POSIX spec, kernel panic, etc).
>
>
>
> This part is true. It's the biggest flaw with time_t: It doesn't acknowledge that leap seconds exist, so what to do during that thing which the standard says doesn't exist is up for grabs :(

As the POSIX definition of "number of seconds since the epoch" is "number of days since the epoch plus number of seconds within the current day" the last second of one day and the first on the next have the same number.

The fix is of course to be made at POSIX and if it was done, tons of issues would just go poof as time then always did increase.

There would still be an issue with "what time is it a gazillion seconds from now" as one do not know the number of leap seconds to be added in the future.

But lets not mix up the issues.

Anyway, I have seen proposals in POSIX on fixing this which have not been moved forward. Anyone being involved in POSIX that knows anything about that status?

   Patrik

---

From: Warner Losh <[email-masked]>
Date: Fri, 24 Jan 2020 07:12:40 -0700

On Thu, Jan 23, 2020, 11:51 PM Patrik Fältström <[email-masked]> wrote:

> >> POSIX doesn't specify, and so systems choose a number of different ways
> of coping (repeat the last second of the day, repeat the first second of
> the next day, freeze time during leap second, do some crazy smear to
> pretend reality matches the POSIX spec, kernel panic, etc).
> >
> >
> >
> As the POSIX definition of "number of seconds since the epoch" is "number
> of days since the epoch plus number of seconds within the current day" the
> last second of one day and the first on the next have the same number.
>

No. That is wrong. The numbering of the leap second is undefined. This is
one way to resolve the issue, but it is not the only way. You will not find
this specified, except indirectly via math on the struct tm, which isn't
definitive about what to do during the leap.

The fix is of course to be made at POSIX and if it was done, tons of issues
> would just go poof as time then always did increase.
>

The fix turns out to be hard. Fix one thing, break another. There are about
a dozen fixes, each breaks something different.

There would still be an issue with "what time is it a gazillion seconds
> from now" as one do not know the number of leap seconds to be added in the
> future.
>
> Anyway, I have seen proposals in POSIX on fixing this which have not been
> moved forward. Anyone being involved in POSIX that knows anything about
> that status?
>

I'd like to see how they can do that. Short of mandating TAI, the problem
can't be fixed.

Warner

   Patrik

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 24 Jan 2020 15:45:08 +0100

On 24 Jan 2020, at 15:12, Warner Losh wrote:

> No. That is wrong. The numbering of the leap second is undefined. This is one way to resolve the issue, but it is not the only way. You will not find this specified, except indirectly via math on the struct tm, which isn't definitive about what to do during the leap.

What I see are two things. First the math:

> If the year is <1970 or the value is negative, the relationship is undefined. If the year is >=1970 and the value is non-negative, the value is related to a Coordinated Universal Time name according to the C-language expression, where tm_sec, tm_min, tm_hour, tm_yday, and tm_year are all integer types:
>
> tm_sec + tm_min*60 + tm_hour*3600 + tm_yday*86400 +
>     (tm_year-70)*31536000 + ((tm_year-69)/4)*86400 -
>     ((tm_year-1)/100)*86400 + ((tm_year+299)/400)*86400

Then the definition of tm_sec which can have values [0,61].

So far, so good, and the number of seconds in a day with an added leap second will then be 86401, and the first second of the following day (the following second) will have the same value.

Where I agree with you that the leap second is undefined is the plain text which says:

> As represented in seconds since the Epoch, each and every day shall be accounted for by exactly 86400 seconds.

This implies that even if the math gives 86401 seconds (including the fact tm_sec can have a value of 61), the value 86401 seconds in a day is not valid.

   Patrik

---

From: Watson Ladd <[email-masked]>
Date: Fri, 24 Jan 2020 07:04:54 -0800

On Thu, Jan 23, 2020, 10:51 PM Patrik Fältström <[email-masked]> wrote:

> The fix is of course to be made at POSIX and if it was done, tons of
> issues would just go poof as time then always did increase.
>

Why does POSIX matter for NTP timestamps? We can always clarify the
language independently.

---

From: Martin Burnicki <[email-masked]>
Date: Fri, 24 Jan 2020 16:09:36 +0100

Watson Ladd wrote:
> On Wed, Mar 27, 2019, 9:47 AM Daniel Franke <[email-masked]
> <mailto:[email-masked]>> wrote:
> 
> second is repeated in an NTP timestamp and to make sure all our sources
> use the right one. 

Strictly speaking, it depends on the operating system.

A process like ntpd notifies the kernel that a leap second is coming up
at the end of the day, and the kernel handles the leap second. So it
depends on the implementation in the kernel whether the last or the
first second is repeated to insert a leap second.

ntpd just picks up the current system time from the kernel and sends it
to its clients.

It only detects that the leap second event has passed when it queries
the kernel time stat the next time. If clients retrieved the time from
ntpd after the leap second has finished but before ntpd has updated the
kernel time status the leap bits can be wrong.

This is one of the reasons why ntpd sends leap bits "unsynchronized" for
a short time around the leap second. This doesn't hurt or confuse the
client.

Martin
-- 
Martin Burnicki

Senior Software Engineer

MEINBERG Funkuhren GmbH & Co. KG
Email: [email-masked]
Phone: +49 5281 9309-414
Linkedin: https://www.linkedin.com/in/martinburnicki/

Lange Wand 9, 31812 Bad Pyrmont, Germany
Amtsgericht Hannover 17HRA 100322
Geschäftsführer/Managing Directors: Günter Meinberg, Werner Meinberg,
Andre Hartmann, Heiko Gerstung
Websites: https://www.meinberg.de  https://www.meinbergglobal.com
Training: https://www.meinberg.academy

---

From: Warner Losh <[email-masked]>
Date: Fri, 24 Jan 2020 08:42:18 -0700

On Fri, Jan 24, 2020 at 7:45 AM Patrik Fältström <[email-masked]> wrote:

> > No. That is wrong. The numbering of the leap second is undefined. This
> is one way to resolve the issue, but it is not the only way. You will not
> find this specified, except indirectly via math on the struct tm, which
> isn't definitive about what to do during the leap.
>
> > If the year is <1970 or the value is negative, the relationship is
> undefined. If the year is >=1970 and the value is non-negative, the value
> is related to a Coordinated Universal Time name according to the C-language
> expression, where tm_sec, tm_min, tm_hour, tm_yday, and tm_year are all
> integer types:
> >
> So far, so good, and the number of seconds in a day with an added leap
> second will then be 86401, and the first second of the following day (the
> following second) will have the same value.
>
> This implies that even if the math gives 86401 seconds (including the fact
> tm_sec can have a value of 61), the value 86401 seconds in a day is not
> valid.
>

Yes. It's ambiguous. A POSIX time_t is not defined at the leap second. You
could use your math to get one answer. However, that answer, back
translated to a struct tm gives the wrong day, so is also incorrect. In
POSIX land, leap seconds don't exist, and any definitive statement about
the 'proper' time_t encoding for a leap second cannot be made. It's
undefined.

Warner

---

From: Watson Ladd <[email-masked]>
Date: Fri, 24 Jan 2020 08:29:55 -0800

> a short time around the leap second. This doesn't hurt or confuse the

Is this mandated by RFC 5905?

If not, should it be/should the behavior be more precise?

I realize there may be a bug around this when using a separate server
process: my gut says one measurement off by a second is insignificant since
the maximum adjustment in response, but there may be other issues I'm not
thinking about.

---

From: Warner Losh <[email-masked]>
Date: Fri, 24 Jan 2020 10:15:54 -0700

On Fri, Jan 24, 2020 at 9:31 AM Watson Ladd <[email-masked]> wrote:

> thinking about

Carefully reading RFC 5905 shows this to be unspecified. One can make a
logical argument that the unsynchronized is a sound approach (I'm giving
you data that I know I can't properly encode because ntp timestamps can't
encode leap seconds), but it's not mandated specifically that I could find.
And it's a tricky problem absent some kind of quiet period around a leap
second because the data is ambiguous. It's made more so because the only
well defined operations on the timestamps, according to the RFC, is to
subtract them to get a time phase delta measurement.

Warner

>> <https://www..meinbergglobal.com>

---

From: "Patrik Fältström" <[email-masked]>
Date: Fri, 24 Jan 2020 19:59:15 +0100

On 24 Jan 2020, at 16:04, Watson Ladd wrote:

> Why does POSIX matter for NTP timestamps? We can always clarify the language independently.

I completely agree, but so much discussions about NTP today are built upon the POSIX definitions. For example by having Google (and possibly others) having NTP servers that do smear.

<https://developers.google.com/time/smear>

Personally, I find the fact the leap second itself is undefined is of no problem for NTP or implementations. The problem is when the implementations have to push the response from NTP into a POSIX based environment that do not take leap seconds into account.

   Patrik

---

From: Warner Losh <[email-masked]>
Date: Fri, 24 Jan 2020 14:58:51 -0700

> > Why does POSIX matter for NTP timestamps? We can always clarify the

It matters because NTP's timestamps and POSIX's time_t have the same issue:
there is no clear, unambiguous value to encode a timestamps during a
leapsecond.

> I completely agree, but so much discussions about NTP today are built upon
> the POSIX definitions. For example by having Google (and possibly others)
> having NTP servers that do smear.
>
> Personally, I find the fact the leap second itself is undefined is of no
> problem for NTP or implementations. The problem is when the implementations
> have to push the response from NTP into a POSIX based environment that do
> not take leap seconds into account.
>

How then do I encode a leap second timestamp into an NTP timestamp. Please
cite the relevant text from the RFC. I found no formula for the 'seconds'
value, just a table of values that implies it's 'simple math' and would
disregard leapseconds.

Warner

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 27 Jan 2020 08:07:43 +0100

>>> Watson Ladd <[email-masked]> schrieb am 24.01.2020 um 16:04 in
Nachricht
> Why does POSIX matter for NTP timestamps? We can always clarify the

I guess ist because operating systems keep time as POSIX time, and after
starting NTP, it's the only local time it has to start with.  Any additional
time scale could be maintained within NTP only (i.e. is lost when NTP stops).

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 27 Jan 2020 08:13:49 +0100

>>> Martin Burnicki <[email-masked]> schrieb am 24.01.2020 um
16:09 in
Nachricht <[email-masked]>:

...
> a short time around the leap second. This doesn't hurt or confuse the

Given the typical polling intervals that UNSYNC status may be valid
significantly longer than the actual leap second. That is why I think it's
wrong to set UNSYNC during leap second. It's also against expectations: Why
should an NTP server that is stable be UNSYNC or jump to stratum 16 suddenly?

This simply feels wrong.

Regards,
Ulrich

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 27 Jan 2020 08:37:30 +0100

>>> Warner Losh <[email-masked]> schrieb am 24.01.2020 um 16:42 in Nachricht
> the 'proper' time_t encoding for a leap second cannot be made. It's

I think that is the major bug in POSIX leap seconds: While time_t is the
internal representation of time, struct tm is only the human-readable
representation. Leap seconds were based on the human representation (61st
second in minute), while most software works with the internal representation.
So converting between both representations is not unique.

---

From: "Ulrich Windl" <[email-masked]>
Date: Mon, 27 Jan 2020 08:46:53 +0100

>>> Warner Losh <[email-masked]> schrieb am 24.01.2020 um 18:15 in Nachricht
> subtract them to get a time phase delta measurement.

I was just thinking: (assuming the long batle with designing proper extension
fields is over)

When a (new) server would add an "leap second in progress" extension field,
the (new) clients could simply ignore the timestamp if they feel like that.
For regular leap second processing that extension field would be set just
during the actual leap second processing, while for "leap smear server" it
could be set until leap smearing is done.

Or maybe have two different "leap second in progress" indicators.

BTW: How about adding an extension field that specifies the time and type of
the next leap second to occur?

Regards,
Ulrich

---

From: Martin Burnicki <[email-masked]>
Date: Mon, 27 Jan 2020 09:37:37 +0100

> Then the definition of tm_sec which can have values [0,61].

Yes, you can set tm_sec to 61, but how should the conversion function of
the runtime library know that it shoudld return second 61 of the current
day instead of second 0 of the next day? In POSIX there is no way I'm
aware of that can be used to tell the conversion function that a
specific second is an inserted leap second.

Martin
-- 
Martin Burnicki

Senior Software Engineer

---

From: Martin Burnicki <[email-masked]>
Date: Mon, 27 Jan 2020 09:51:57 +0100

> should an NTP server that is stable be UNSYNC or jump to stratum 16 suddenly?

Sorry, I disagree. Please see my other post with some explanation.

Martin
-- 
Martin Burnicki

Senior Software Engineer

---

From: Watson Ladd <[email-masked]>
Date: Mon, 27 Jan 2020 07:56:45 -0800

> specific second is an inserted leap second.

Operating systems with the ntp_adjtimex system call do return leapsecond
info. This includes linux and the *BSDs.

System call interfaces can be extended if we have needs here.