Subject: [Ntp] Guidelines for Defining Packet Timestamps

From: Tal Mizrahi <[email-masked]>
Date: Sun, 24 Sep 2017 16:37:31 +0300

Hi,

We have revised the draft based on the comments received in IETF 99, and
based on the comments from Yaakov (thanks Yaakov).

https://datatracker.ietf.org/doc/draft-mizrahi-intarea-packet-timestamps/

The main changes compared to the previous version of the draft:
- We have extended the discussion about the factors that may affect the
choice of the timestamp format.
- A new section has been added, called "Timestamp Use Cases".
- The sychronization aspects have been separated from the timestamp format,
allowing the timestamp format to be independent of how time is synchronized.

Any further comments are welcome.

Cheers,
Tal, Joachim, and Al.

---

From: Warner Losh <[email-masked]>
Date: Sun, 24 Sep 2017 09:06:44 -0600

On Sun, Sep 24, 2017 at 7:37 AM, Tal Mizrahi <[email-masked]>
wrote:

> - The sychronization aspects have been separated from the timestamp
> format, allowing the timestamp format to be independent of how time is
> synchronized.
>

Section 4.1 says "Further considerations may be discussed in this section,
such as required accuracy, or leap second handling." Yet none of the
formats make any allowance for leap seconds. In NTP headers, leap second
time stamps are marked with an out-of-band bit. In addition, all NTP time
stamps are adjusted by the cumulative number of leap seconds, yet that
relevant information is not included.PTP timestamps, on the other hand, are
a count of TAI seconds since 1970, which have no adjustments.

When I was doing time and frequency recovery, these differences weren't
documented and many arguments ensued to get to the understanding of what's
actually on the wire, what the auxiliary bits mean, etc. I'd rather see
this document be explicit about such things rather than some vague hand
wave that will be interpreted in many different ways leading to even more
issues with leap seconds than we already have.

Warner

---

From: Tal Mizrahi <[email-masked]>
Date: Mon, 25 Sep 2017 09:37:47 +0300

Hi Warner,

Thanks for the feedback.

>Section 4.1 says "Further considerations may be discussed in
>this section, such as required accuracy, or leap second handling."
>Yet none of the formats make any allowance for leap seconds.

Right. As I pointed out below, we have separated the *formats* from the
*synchronization aspects*. The advantage of this approach is that a given
timestamp format does not necessarily define the time standard (UTC / TAI),
and specifically does not necessarily define the leap second handling.

That means that a specification that uses a recommended timestamp format
should define the synchronization aspects, including leap second
considerations if such exist. See the following text in the draft:

   A specification that uses one of the recommended timestamp formats
   should also include a section on Synchronization Aspects...
   ...such as required accuracy, or leap second handling.

With that said, your point is well taken. In the next draft we will add
some more text that clarifies this point, including an example in Section 5.

Regards,
Tal.

On Sun, Sep 24, 2017 at 6:06 PM, Warner Losh <[email-masked]> wrote:

> Section 4.1 says "Further considerations may be discussed in this
> section, such as required accuracy, or leap second handling." Yet none of
> the formats make any allowance for leap seconds. In NTP headers, leap
> second time stamps are marked with an out-of-band bit. In addition, all NTP
> time stamps are adjusted by the cumulative number of leap seconds, yet that

---

From: Warner Losh <[email-masked]>
Date: Mon, 25 Sep 2017 07:17:16 -0600

> and specifically does not necessarily define the leap second handling.

That's not an advantage by any means. If you don't know how to interpret
the data, it's useless. And leap seconds are part of the data, as much as
people want to ignore them. It's a flaw in the specification. You go to
great lengths to say that the PPT is seconds since a time in TAI. How you
count the seconds since the epoch is quite relevant. It's not just a
counter, but a corrected counter in NTP's case. And a counter and a
corrected counter are two very different things, since the math you can
safely do on them, and the meta data you need for some operations with them
differ. You can't escape that in defining the format because that's
critical to its use. And a corrected counter has ambiguities around the
leap second that are impossible to avoid without other metadata (NTP puts
that metadata into another bit of the NTP packet so you know, for example).
The details about the metadata likely can remain unspecified, but whether
or not leap seconds are swizzled in should not.

> some more text that clarifies this point, including an example in Section 5.

As someone who has fought with formats being under specified, I can tell
right now that this will lead to trouble. You can't run away from the
Synchronization Aspects, at least not all of them. Having dealt with this
stuff for almost two decades in high precision time applications, and
having tried to separate out the two concepts to make things "cleaner" I
can attest from first hand experience that this is a mistake.

Warner

>> section, such as required accuracy, or leap second handling." Yet none
>> of the formats make any allowance for leap seconds. In NTP headers, leap

---

From: Tal Mizrahi <[email-masked]>
Date: Mon, 25 Sep 2017 18:19:41 +0300

Hi Rodney,

Thanks for the feedback.
We will certainly take these notes into consideration in the next version.

Can you please elaborate which protocols use the 80-bit timestamp format
(other than PTP, obviously)?

Thanks,
Tal.

On Mon, Sep 25, 2017 at 6:01 PM, Rodney Cummings <[email-masked]>
wrote:

> Hi Tal,
>
> This is an excellent draft. I have a couple of suggestions.
>
> 1. Add the PTP 80-bit Timestamp, since there are protocols using that as
> well, some with hardware support.
>
> 2. In section 4, add informative text to explain one of the benefits of
> specifying a small set of fixed format specifications: hardware support.
> The lower down in the interface stack that timestamping occurs, the more
> accurate and precise the timestamp. This is well known for PTP, for which
> timestamping is often done at the MAC or PHY level. Hardware support is
> potentially a reason to avoid the control field mentioned in section 7,
> because hardware typically cannot support "any" format... ideally this RFC
> will narrow it down to 2-3 at most.
>
> Rodney
>
> From: TICTOC [mailto:[email-masked]] On Behalf Of Tal Mizrahi
> Sent: Sunday, September 24, 2017 8:38 AM
> To: [email-masked]; [email-masked]
> Subject: [TICTOC] Guidelines for Defining Packet Timestamps
>
> Hi,
>
> https://urldefense.proofpoint.com/v2/url?u=https-3A__
> datatracker.ietf.org_doc_draft-2Dmizrahi-2Dintarea-
> 2Dpacket-2Dtimestamps_&d=DwMFaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZ
> uIPelcSjixA&r=WA71sf2o7Dw7CbYhFt24DPjt3lJuupswWYdnboKbZ8k&m=
> yUZsj6L0ib9J4wu3kPNa1OQJK4MJPBl2vMca6Rpvx_Q&s=xdFj3xTP-9EZqma_
> 84T6eWQ0p0VlwMrdXil6FFU3FW0&e=
>

---

From: Tal Mizrahi <[email-masked]>
Date: Mon, 25 Sep 2017 18:31:39 +0300

Hi Warner,

We are in full agreement that any considerations relating to the leap
second should be defined.

I believe the main question here is whether leap second considerations
should be defined in the recommended timestamp formats, or in the protocols
that use them.
One thing to remember is that this draft is intended to be generic, so that
timestamp formats will be applicable to many different protocols such as
OWAMP, TWAMP, TRILL loss/delay, MPLS loss/delay, and of course NTP.
Therefore, there may be an advantage to having a common timestamp format
that is sometimes used with respect to TAI (no leap seconds), and sometimes
used with respect to UTC (leap second considerations are required).

Whether leap second considerations are included in the timestamp format, or
in the protocol that uses it, either way I fully agree that such
considerations should be defined.

Since there are advantages on either way, I would be happy to hear from
other people on the list what they believe would be the best approach here.

Cheers,
Tal.

On Mon, Sep 25, 2017 at 4:17 PM, Warner Losh <[email-masked]> wrote:

>>>> We have revised the draft based on the comments received in IETF 99,
>>>> and based on the comments from Yaakov (thanks Yaakov).
>>>>

---

From: Rodney Cummings <[email-masked]>
Date: Mon, 25 Sep 2017 15:01:37 +0000

Hi Tal,

1. Add the PTP 80-bit Timestamp, since there are protocols using that as well, some with hardware support.

2. In section 4, add informative text to explain one of the benefits of specifying a small set of fixed format specifications: hardware support. The lower down in the interface stack that timestamping occurs, the more accurate and precise the timestamp. This is well known for PTP, for which timestamping is often done at the MAC or PHY level. Hardware support is potentially a reason to avoid the control field mentioned in section 7, because hardware typically cannot support "any" format... ideally this RFC will narrow it down to 2-3 at most.

Rodney

Hi, 

We have revised the draft based on the comments received in IETF 99, and based on the comments from Yaakov (thanks Yaakov).

https://urldefense.proofpoint.com/v2/url?u=https-3A__datatracker.ietf.org_doc_draft-2Dmizrahi-2Dintarea-2Dpacket-2Dtimestamps_&d=DwMFaQ&c=I_0YwoKy7z5LMTVdyO6YCiE2uzI1jjZZuIPelcSjixA&r=WA71sf2o7Dw7CbYhFt24DPjt3lJuupswWYdnboKbZ8k&m=yUZsj6L0ib9J4wu3kPNa1OQJK4MJPBl2vMca6Rpvx_Q&s=xdFj3xTP-9EZqma_84T6eWQ0p0VlwMrdXil6FFU3FW0&e=

The main changes compared to the previous version of the draft:
- We have extended the discussion about the factors that may affect the choice of the timestamp format.
- A new section has been added, called "Timestamp Use Cases".
- The sychronization aspects have been separated from the timestamp format, allowing the timestamp format to be independent of how time is synchronized.

Cheers,
Tal, Joachim, and Al.

---

From: Rodney Cummings <[email-masked]>
Date: Mon, 25 Sep 2017 17:21:39 +0000

Thanks Tal... good question.

One protocol I had in mind is in the industrial space, in early draft form, and not in IETF (or IEEE). Nevertheless, I took a look, and it turns out that it uses the lower 64-bits (i.e. PTP truncated timestamp). The rationale is that they want to leverage PTP hardware, and they didn't see the upper 16 bits as essential.

Thinking about it more... I tend to think that the PTP truncated timestamp will be sufficient.

I withdraw my comment #1 below. Let's just keep with 64-bits and see where that goes.

Rodney

From: Tal Mizrahi [mailto:[email-masked]] 
Sent: Monday, September 25, 2017 10:20 AM
To: Rodney Cummings <[email-masked]>
Cc: [email-masked]; [email-masked]
Subject: Re: [TICTOC] Guidelines for Defining Packet Timestamps

Hi Rodney,

Thanks for the feedback. 

Can you please elaborate which protocols use the 80-bit timestamp format (other than PTP, obviously)?

Thanks,
Tal.

On Mon, Sep 25, 2017 at 6:01 PM, Rodney Cummings <mailto:[email-masked]> wrote:
Hi Tal,

Rodney

From: TICTOC [mailto:mailto:[email-masked]] On Behalf Of Tal Mizrahi
Sent: Sunday, September 24, 2017 8:38 AM
To: mailto:[email-masked]; mailto:[email-masked]
Subject: [TICTOC] Guidelines for Defining Packet Timestamps

Hi, 

Cheers,
Tal, Joachim, and Al.

---

From: Joachim Fabini <[email-masked]>
Date: Mon, 25 Sep 2017 23:00:41 +0200

Hi Warner,

thanks for your input, it triggered some additional ideas. As co-author
of the draft I'm adding some thoughts to this discussion - please find
the comments to your statements inserted inline.

On 25.09.2017 15:17, Warner Losh wrote:
> 
> 
> On Mon, Sep 25, 2017 at 12:37 AM, Tal Mizrahi <[email-masked]
> <mailto:[email-masked]>> wrote:
>
>     Right. As I pointed out below, we have separated the *formats* from
>     the *synchronization aspects*. The advantage of this approach is
>     that a given timestamp format does not necessarily define the time
>     standard (UTC / TAI), and specifically does not necessarily define
>     the leap second handling.
> 
> 
> the data, it's useless. 

I agree that timestamp interpretation (semantics) must be made explicit.
But I argue that interpretation needs not be explicit part of _any_
timestamp field.
Adding (a) interpretation by extending _any_ timestamp protocol field is
 one option. But there are many other ways to convey semantics. Examples
include, but are not limited to b) preliminary (one-time) agreement
between communicating parties (semantics negotiated once at connection
establishment time and valid throughout the session) or even c)
well-defined semantics in a textual standard or protocol specification.

Variants (a)-(c) are all realistic use cases that have their pros and
cons. A generic timestamp draft must support all of these use cases (and
likely some more).

> And leap seconds are part of the data, as much
> as people want to ignore them. It's a flaw in the specification. You go
> to great lengths to say that the PPT is seconds since a time in TAI. How
> you count the seconds since the epoch is quite relevant. It's not just a
> safely do on them, and the meta data you need for some operations with
> them differ. You can't escape that in defining the format because that's
> leap second that are impossible to avoid without other metadata (NTP
> puts that metadata into another bit of the NTP packet so you know, for
> example). The details about the metadata likely can remain unspecified,
> but whether or not leap seconds are swizzled in should not.
> 

Becoming more technical: the greatest common factor that current
protocols share wrt timestamps is a set of timestamp (data) fields of a
specific size. This is what the timestamp draft already covers in
detail. Interpretation - i.e., semantics - is currently up to the
protocols that use these timestamp data fields. Some protocols define
control fields on this purpose, but these are proprietary.

This draft will eventually contribute exactly what is currently missing
(and only present as a placeholder): a standardized control field that
_can_ be used by protocols to extend timestamp data fields with
semantics. Protocols that are willing to use the control field (and
accept its overhead) can add semantics to their timestamps. In some
cases it may even be used separated from timestamp data - for instance
when negotiating a connection in the case (b) mentioned earlier.
Prerequisite is that these semantics persist over connection lifetime,
which may or may not be true and applicable.

In other cases it may be meaningful for protocols to augment any single
timestamp with semantics. A timestamp control field could, for instance,
convey information on (estimated) local time synchronization accuracy
(like IEEE C37.118/2011 does). Or on leap seconds. And there are many
more cases.

Btw, while writing these lines I realize that it may even be worth to
differentiate between static timestamp semantics (valid over an
association lifetime) and dynamical semantics (that are valid for the
instant in time when the timestamp was generated).
 
> 
>     That means that a specification that uses a recommended timestamp
>     format should define the synchronization aspects, including leap
>     second considerations if such exist. See the following text in the
>     draft:
> 
>     With that said, your point is well taken. In the next draft we will
>     add some more text that clarifies this point, including an example
>     in Section 5.
> 
> 
> Synchronization Aspects, at least not all of them. Having dealt with
> this stuff for almost two decades in high precision time applications,
> and having tried to separate out the two concepts to make things
> "cleaner" I can attest from first hand experience that this is a mistake.

Imho no timestamp format can ever prevent protocol designers from
under-specifying their timestamps. But the timestamp draft will support
them by providing guidance and eventually improve the status quo.

Your practical expertise and the one of other experts in ntp is an
invaluable help in figuring out the semantics that timestamps really
need. Finding the right trade-off between features, flexibility,
representation (size) and simplicity will be a major challenge. Likewise
we must find solutions for conflicting requirements (for instance Rodney
raised the topic of hardware support and the use of a control field).

So there's plenty of work ahead but imo it's worth spending the effort.
The discussion shows that there is a need for this draft and that
ntp/tictoc is the right group to host it. This, again, supports the
request for adoption as WG item.

thanks again,
Joachim