Subject: [Ntp] NTP over PTP

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 24 Jun 2021 13:32:46 +0200

The NTP WG was asked to secure the PTP unicast mode considering it has
experience with securing NTP. The problem is that the PTP unicast mode
is very different from the NTP client/server mode, the only mode which
I think we can claim to be well secured (by symmetric key and NTS).
The closest thing to the PTP unicast mode would be the NTP control
mode (6), which is not secured (e.g. against replay attacks).

I suspect there is nothing here to be reused to secure the unicast
PTP. There might be some practical advantages of sharing NTS-KE with
NTP, but I don't see any security benefits. The two drafts that were
discussed here just need TLS and might actually be simpler if they
used it directly.

With this WG still having NTP in its name, there is the uncomfortable
question that has to be asked. Why can't you just use the NTP
client/server mode instead of the PTP unicast mode?

A point was made that there is still a lot of hardware that supports
only plain PTP. It doesn't support PTP secured by IPSec, MACSec, DTLS,
etc, either. This was apparently one of the reasons for writing the
NTS4UPTP draft.

That has a simple solution. Use PTP as a transport for NTP messages.
The hardware doesn't care what is inside as long as it can recognize
the PTP header to trigger the timestamping.

I submit this idea as a (very short) draft:
https://datatracker.ietf.org/doc/html/draft-mlichvar-ntp-over-ptp-00

I understand some people won't like this as an answer to the question
of securing the unicast PTP. It would be useful in other cases too.

But is it worth trying to invent a PTP-specific solution to secure its
unicast mode, when the result cannot possibly be as good as what NTP
already has?

As I understand it, the unicast mode of PTP was invented mainly for
some of the telco profiles, which for the most part seem to emulate
NTP. What exactly was missing in NTP when those profiles were written?
IIRC someone mentioned it was discussed with the NTP WG at the time,
but I don't have any references.

-- 
Miroslav Lichvar

---

From: Heiko Gerstung <[email-masked]>
Date: Fri, 25 Jun 2021 15:00:25 +0200

If I follow your argument that unicast PTP is like the NTP control mode then yes, both mode 6 and unicast PTP are not secured against replay and amplification attacks. The NTS4UPTP draft is exactly addressing this for unicast PTP and provides the required protection. I outlined in the draft that the proposed approach achieves most of the stated objectives/goals of NTS4NTP, which is well secured (we certainly agree on that). 

In the interim meeting I believe there have been multiple participants that agreed that 
a) PTP should be secured (it is worth it)
b) the NTP WG is the right place for this

Arguing that the two drafts on the table might just need TLS instead of NTS-KE is something I cannot understand, too. PTP already has a security mechanism that is standardized and, with some extensions (provided by the submitted drafts), can be used to significantly enhance security of the protocol. What is needed is a key exchange mechanism and some additional features to close the security holes (i.e. no protection for the source IP address) - again, exactly what the two drafts provide. 

Regarding "NTP over PTP": trying to trick the hardware timestamping engines of a large variety of vendors into timestamping a PTP packet is bound to fail IMHO. There is no standard for how these timestamping engines work, some may even look at sizes of packets (they should not do that, as PTP messages can theoretically carry TLVs, but sometimes hardware vendors take shortcuts). Some implementations will forward packets to a management CPU when the forwarding plane detects a PTP frame. The CPU will not recognize that this is NTP and will probably throw away the packet. Again, most likely not the right way to do it, but alas ... 

Why did the telcos choose to go for unicast PTP instead of NTP? Well, one reason was that PTP already provided hardware timestamping capabilities at that time, to dramatically increase the accuracy to a sub-microsecond level that is required for modern telecommunication networks. I know of some hardware vendors in that application space that implemented hardware timestamping for NTP, but there are not a lot of them (Meinberg offers a NTP module which has hw timestamping for NTP, providing 10ns accuracy in the timestamp of the outgoing NTP packets). And, maybe the main reason: the first unicast PTP telco profile ITU-T G8265.1 was designed for frequency-only applications. Unicast PTP as a protocol allows that you do not measure the delay from the client to the server, something that is not required for frequnecy only mode. Using unicast PTP was a lot more efficient that NTP, because the PTP client only has to ask for packets once every minute for example, and gets a constant stream of highly accuracy and hardware timestamped packets from the server for that 60 seconds without having to send a request for each of the packets. 

Well, this is all history now. PTP is here to stay and unicast PTP is probably not going to take over the world, but will for sure be used in some new applications in the future. 

Regards,
  Heiko

-- 
Heiko Gerstung 
Managing Director 
 
MEINBERG® Funkuhren GmbH & Co. KG 
Lange Wand 9 
D-31812 Bad Pyrmont, Germany 
Phone: +49 (0)5281 9309-404 
Fax: +49 (0)5281 9309-9404 
 
Amtsgericht Hannover 17HRA 100322 
Geschäftsführer/Management: Günter Meinberg, Werner Meinberg, Andre Hartmann, Heiko Gerstung 
 
Email: 
[email-masked]
Web: 
Deutsch https://www.meinberg.de
English https://www.meinbergglobal.com
 
Do not miss our Time Synchronization Blog: 
https://blog.meinbergglobal.com
 
Connect via LinkedIn: 
https://www.linkedin.com/in/heikogerstung

Am 24.06.21, 13:33 schrieb "ntp im Auftrag von Miroslav Lichvar" <[email-masked] im Auftrag von [email-masked]>:

    -- 
    Miroslav Lichvar

    _______________________________________________
    ntp mailing list
    [email-masked]
    https://www.ietf.org/mailman/listinfo/ntp

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 28 Jun 2021 15:45:51 +0200

> If I follow your argument that unicast PTP is like the NTP control mode then yes, both mode 6 and unicast PTP are not secured against replay and amplification attacks. The NTS4UPTP draft is exactly…

If I understand it correctly, your draft is addressing the
amplification issue by requiring client authentication. That's
probably the best one can do, but it doesn't make it comparable to NTP
or NTS4NTP. The operator will still have to trust the clients to not
abuse their credentials and also trust them to not get compromised.
That's nothing like NTS4NTP where you can run a public server without
any worry about someone exploiting your server for amplification.

> a) PTP should be secured (it is worth it)

At least for the normal non-unicast mode, I agree.

The unicast mode seems to be intended for networks with partial
on-path hardware support, where requirements on accuracy are less
strict, and I think this might already be better supported by NTP.

> Arguing that the two drafts on the table might just need TLS instead of NTS-KE is something I cannot understand, too. PTP already has a security mechanism that is standardized and, with some extens…

Ok, but to me it seems it would be simpler if you have skipped NTS-KE
and went with TLS directly. Anyway, for adopting the document that
shouldn't matter.

> Regarding "NTP over PTP": trying to trick the hardware timestamping engines of a large variety of vendors into timestamping a PTP packet is bound to fail IMHO. There is no standard for how these ti…

If there is some hardware that can timestamp only fixed-length PTP
messages with no TLVs, then that will not work with any form of
NTS4UPTP either, right? There has to be some TLV to authenticate the
message and the hardware needs to accept that.

The same applies to one-step clocks. If implemented in silicon, that
will not work in any case.

-- 
Miroslav Lichvar

---

From: Heiko Gerstung <[email-masked]>
Date: Mon, 28 Jun 2021 17:02:43 +0200

> Am 28.06.21, 15:46 schrieb "ntp im Auftrag von Miroslav Lichvar"
> <[email-masked] im Auftrag von [email-masked]>:
> 
>> If I follow your argument that unicast PTP is like the NTP control mode
>> then yes, both mode 6 and unicast PTP are not secured against replay and
>> amplification attacks. The NTS4UPTP draft is exactly addressing this for
>> unicast PTP and provides the required protection. I outlined in the draft that
>> the proposed approach achieves most of the stated objectives/goals of NTS4NTP,
>> which is well secured (we certainly agree on that).
> 
That is true and it is a result of the way the protocol is designed. 

> strict, and I think this might already be better supported by NTP.
They might be less strict but that does not mean they are worse/equal compared to NTP. With HW timestamping on the server and client, you can remove the impact of the multitasking OS network stack in the end devices and "partial" on path support means that at least some of the switches/routers can have PTP support. Plus there are use cases where you can have full on-path support because you are running a private network of a certain size (for example a whole full blown hyperscale DC). In those cases (and probably many more), NTP is not an alternative.

>> Arguing that the two drafts on the table might just need TLS instead of
>> NTS-KE is something I cannot understand, too. PTP already has a security
>> mechanism that is standardized and, with some extensions (provided by the
>> submitted drafts), can be used to significantly enhance security of the
>> protocol. What is needed is a key exchange mechanism and some additional
>> features to close the security holes (i.e. no protection for the source IP
>> address) - again, exactly what the two drafts provide.

> and went with TLS directly. Anyway, for adopting the document that

OK. 

>> Regarding "NTP over PTP": trying to trick the hardware timestamping
>> engines of a large variety of vendors into timestamping a PTP packet is bound
>> to fail IMHO. There is no standard for how these timestamping engines work,
>> some may even look at sizes of packets (they should not do that, as PTP
>> messages can theoretically carry TLVs, but sometimes hardware vendors take
>> shortcuts). Some implementations will forward packets to a management CPU when
>> the forwarding plane detects a PTP frame. The CPU will not recognize that this
> is NTP and will probably throw away the packet. Again, most likely not the
>> right way to do it, but alas ...
> 

No, that's not correct. The hardware timestamping is only required during phase 3 (packet transmission), which does not require an NTS TLV as it relies completely on the AUTHENTICATION_TLV and the integrated PTP security. See 3.3 of my draft. The NTS TLV is required in phase 2 where client and server only exchange messages to negotiate the transmission, and those are not hw timestamped. 

> will not work in any case.

Again, depends on the implementation and the choices made by the designers. I do not want to say that such a flawed implementation is the rule, and I also do not want to imply that the whole concept cannot work, but I am sure that it is not the solution to the problem we wanted to solve with NTS4UPTP. 

Best Regards,
  Heiko

-- 
Heiko Gerstung 
Managing Director 

 
Email: 

Connect via LinkedIn:

---

From: Doug Arnold <[email-masked]>
Date: Mon, 28 Jun 2021 16:55:06 +0000

The telecom industry is definitely going to use unicast PTP whether NTP is better or not.  They have invested a lot in PTP and it is all over in their standards.  The telecom industry won’t even go to the bathroom unless there is a standard for it.  People in the power industry sometimes use unicast PTP between substations because their substations are connected using technology barrowed the telecom industry.  Some people in finance also use unicast PTP.  Unicast PTP is here for foreseeable future.  We need to make it as robust as we can.

Doug

From: ntp <[email-masked]> on behalf of Miroslav Lichvar <[email-masked]>
Date: Monday, June 28, 2021 at 9:46 AM
To: Heiko Gerstung <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [Ntp] NTP over PTP

--
Miroslav Lichvar

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 29 Jun 2021 08:52:11 +0200

On Mon, Jun 28, 2021 at 05:02:43PM +0200, Heiko Gerstung wrote:
> They might be less strict but that does not mean they are worse/equal compared to NTP.

How so?

You said unicast transparent clocks don't really exist. That would be
the only advantage of unicast PTP. Boundary clocks could fully support
hardware-timestamped NTP if someone was actually interested in
implementing that.

Without full on-path support NTP should generally perform better than
PTP as it doesn't assume network has a constant delay.

In the context of the drafts we are discussing here, I think it might
be easier for an existing PTP implementation to add support for
NTP+NTS than add NTS4UPTP.

> No, that's not correct. The hardware timestamping is only required during phase 3 (packet transmission), which does not require an NTS TLV as it relies completely on the AUTHENTICATION_TLV and the…

I don't see how that is relevant to hardware not timestamping PTP
messages with TLVs. If a TLV present in an event message breaks the
timestamping, the hardware is unusable for any security mechanism
discussed here, except the Daniel's idea to protect unauthenticated
PTP by authenticated NTP.

-- 
Miroslav Lichvar

---

From: Heiko Gerstung <[email-masked]>
Date: Tue, 29 Jun 2021 09:15:22 +0200

>> They might be less strict but that does not mean they are worse/equal
Because of the removal of jitter/delay variation in the network stack and OS layer (kernel) of the server and the client. 

> hardware-timestamped NTP if someone was actually interested in
I was wrong, or better: my knowledge was outdated. Arista for example supports unicast in their switches now.

> PTP as it doesn't assume network has a constant delay.
Why do you think PTP assumes a constant network delay? PTP is measuring the delay constantly in both directions and calculates the round trip. Its delay measurement mechanism is definitely inferior to the one NTP uses (in regards to the concept) but it is good enough to allow (unicast) PTP to reliably achieve sub microsecond accuracy when used in networks with partial or full on-path support. That would not be possible in a packet switched network without proper delay compensation.

> NTP+NTS than add NTS4UPTP.

We are maintaining three different PTP implementations here at Meinberg and I can tell you that this is not true. 

>> No, that's not correct. The hardware timestamping is only required during
>> phase 3 (packet transmission), which does not require an NTS TLV as it relies
>> completely on the AUTHENTICATION_TLV and the integrated PTP security. See 3.3
>> of my draft. The NTS TLV is required in phase 2 where client and server only
>> exchange messages to negotiate the transmission, and those are not hw
>> timestamped.
> 

I did not say that hardware is not timestamping PTP event messages that carry a TLV. I just pointed out that some of the hardware timestampers might look at the length of a packet and do not timestamp anything that is longer then X bytes. A sync message is 44 bytes plus maybe 26 bytes for the AUTENTICATION_TLV.  If you remove the requirement for the AUTHENTICATION_TLV because you want to run NTS over PTP, your PTP "header" is 44 bytes + 48 bytes for the NTP header + anything that NTS adds (Unique Identifier EF, NTS Authentication and Encrypted Extension Fields EF, at least one NTS Cookie EF). It is a bit complicated (for me) to calculate the total maximum size of such an NTS over PTP packet, but I am guessing that you will end up with more than 128 bytes, which might be a limit packet matching algorithms have for quickly identifying if this can be a PTP event message. 

The idea might still be worth to discuss and investigate. But it is not the answer to the question how we can secure unicast PTP IMHO. 

Regards,
   Heiko

-- 
Heiko Gerstung 
Managing Director 

 
Email: 

Connect via LinkedIn: 

Am 29.06.21, 08:52 schrieb "Miroslav Lichvar" <[email-masked]>:

    How so?

    -- 
    Miroslav Lichvar

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 29 Jun 2021 11:39:38 +0200

> Because of the removal of jitter/delay variation in the network stack and OS layer (kernel) of the server and the client. 

That jitter/delay has no impact on PTP or NTP measurements when using
hardware timestamping. Even with software timestamps they can be
eliminated if the timestamp can be captured in the network driver
right before passing the packet to hardware, as most drivers on Linux
can do.

> I was wrong, or better: my knowledge was outdated. Arista for example supports unicast in their switches now.

Ok, I could extend the NTP-over-PTP draft to take advantage of that
if the support is expected to spread.

> Why do you think PTP assumes a constant network delay? PTP is measuring the delay constantly in both directions and calculates the round trip.

Yes, it does, but it is separate from the offset calculation.

The calculation is described in section 11.2 of 1588-2019. It uses
<meanDelay> and there is only the TX and RX timestamp of the sync
message like in the NTP broadcast mode. If the distribution of the
actual delay is not symmetric, as is common without full on-path
support, the average error of the measurements will not even get close
to zero. PTP relies on full hardware support. Without that, it
generally cannot perform as well as NTP.

Another issue with using PTP in network without PTP support is RX
timestamping fixed to the beginning of the message. If the server is
on a 1Gb/s link and the PTP client is on a faster link, there will be
an asymmetry of hundreds of nanoseconds due to the asymmetric delay
in forwarding of messages between different link speeds.

> We are maintaining three different PTP implementations here at Meinberg and I can tell you that this is not true. 

I don't know what implementations are those, but at least for the
well-known open-source implementations I think it would be.

> I did not say that hardware is not timestamping PTP event messages that carry a TLV. I just pointed out that some of the hardware timestampers might look at the length of a packet and do not timest…

Ok, so it's not about the message having a fixed length, but a maximum
length. That looks like a very odd quirk. Do you have a specific
example of such a hardware?

-- 
Miroslav Lichvar

---

From: Heiko Gerstung <[email-masked]>
Date: Tue, 29 Jun 2021 13:17:45 +0200

>> Because of the removal of jitter/delay variation in the network stack and OS
>> layer (kernel) of the server and the client.
> 
Even a kernel generated SO_TIMESTAMP is subject to jitter and latency on a multitasking OS. For egress timestamps you might be able to control that to a certain extent, but for ingress timestamps the network driver will read out the rx queue of the chip and timestamp packets/frames whenever it gets to it. 

But I agree that once you use hw timestamping for NTP, you can get results which are as good as PTP on a host. 

> if the support is expected to spread.

You definitely should do that. 

>> Why do you think PTP assumes a constant network delay? PTP is measuring the
>> delay constantly in both directions and calculates the round trip.
> 
Which is not the same as claiming that PTP assumes the "network has a constant delay". 

> generally cannot perform as well as NTP.
Wrong, even without full on-path support unicast PTP uses delay requests/responses to take the client-to-server delay into consideration as well. See IEEE1588-2019 Subsection 11.3 for a description of how this works. ITU-T G.8265.1 (Telecom Profile)  and its frequency-only approach is not using the delay req/resp exchange as it does not need to compensate the delay (it is only used to synthonize the client, which is typically a 2G/3G/4G FDD base station ). All other unicast PTP use-cases that I am aware of are using a round trip delay calculation to compensate the delay, including the newer telecom profiles that have been created to allow phase synchronization for TDD. 

> in forwarding of messages between different link speeds.
Yes, there are implementations which take that into account by applying static correction values to compensate for link speed asymmetry. I believe this also affects NTP, but in most cases hundreds of nanoseconds are not a problem for applications relying on NTP synchronization. 
 
> well-known open-source implementations I think it would be.

I am not so sure about this, but ultimately for those implementations the maintainers should probably speak. To me it looks like it would be easier to add NTS-over-PTP support to existing NTS4NTP implementations (you mentioned 7 LoC for chrony earlier) than having to add a full NTS-for-NTP implementation to an existing PTP stack. 

>> I did not say that hardware is not timestamping PTP event messages that carry
>> a TLV. I just pointed out that some of the hardware timestampers might look at
>> the length of a packet and do not timestamp anything that is longer then X
>> bytes. A sync message is 44 bytes plus maybe 26 bytes for the
>> AUTENTICATION_TLV.  If you remove the requirement for the AUTHENTICATION_TLV
>> because you want to run NTS over PTP, your PTP "header" is 44 bytes + 48 bytes
>> for the NTP header + anything that NTS adds (Unique Identifier EF, NTS
>> Authentication and Encrypted Extension Fields EF, at least one NTS Cookie EF).
>> It is a bit complicated (for me) to calculate the total maximum size of such an
>> NTS over PTP packet, but I am guessing that you will end up with more than 128
>> bytes, which might be a limit packet matching algorithms have for quickly
>> identifying if this can be a PTP event message.
> 

Over the years I have come to see a lot of odd things in regards to PTP implementations (latest highlight: a software-only implementation of a transparent clock in an industrial switch), therefore I would not be surprised if there are some implementations out there who will break or simply not work when being faced with a NTS-over-PTP packet. For the one I have in mind: this is not a product / implementation I am responsible plus we have an NDA in place with that vendor, so I cannot name names here. 

There are more challenges I see for NTS-over-PTP. You need to synchronize the clock of the hardware timestamper itself, i.e. getting the time into the silicon that creates the timestamp. PTP timestamps are TAI (not UTC), which itself is not a problem as long as you know the TAI-UTC offset. On a server (PTP Grandmaster) this is typically done by using some form of hardware sync for the timestamper engine, e.g. setting the ToD to the upcoming TAI second and then use the PPS to zeroize the fractions. In reality the solution is typically more sophisticated as you do not want to see micro timesteps at the start of every second. 

On a client you have to synchronize your system time with the time of the hw timestamper (e.g. the NIC). That time is synchronized by the hardware itself to the PTP server. PTP4L uses phy2sys for this, but I am not sure about the accuracy with which you can read out the PHC clock and correct the OS clock with it. There is a delay when accessing a NIC over the PCI(e) bus, but this is affecting PTP in the same way. So for the client, you should be on par with PTP in this regard. But for a server you have to find a NIC that supports feeding the PPS of your GNSS receiver (for example) to it, not impossible but also not an easy task for someone who is responsible for maintaining highly accurate synchronization for an entire corporate network. 

The next challenge is on the server, which for unicast PTP requires a certain timestamp queue size to support a usable number of clients. A lot of NICs that claim they have IEEE1588 hardware support have small to tiny ts queue sizes, one common exampe is 4 timestamps. That means you have to be able to read out the hardware timestamps very quickly and you will not really have a chance on high speed links with hundreds and thousands of incoming NTS-over-PTP requests per second.  Those hardware timestamping engines have been designed to be used for PTP clients only, and even then not for the high packet rates that PTP supports (and sometimes requires to improve accuracy over partial on-path-support networks). They cannot be used for servers expecting to handle a high packet rate. 

Finally, I am not sure if IEEE1588 would be happy about an IETF standard "hijacking" one of their protocols, but most probably they cannot do anything about it. Personally I think it is a hack and should not be standardized, but that's just me. I would rather like to see some standard way of flagging an Ethernet frame that I send out to trigger a hardware timestamping engine to timestamp that frame. Such a universal approach could be used by NTP, PTP and other protocols and applications as well (not only time sync protocols), for example to measure network propagation delays etc. It is incredibly hard to get support for this into the silicon of companies like Intel or Broadcom etc., but if it would be universal enough, the chances are higher that it will make its way into products eventually. 

Again, most of the hardware timestampers would probably work and if you want to pursue this approach and move it forward, please do so. It just does not address the problem of securing PTP, therefore I believe it is neither an alternative to any of the submitted NTS for PTP documents.

   Heiko

-- 
Heiko Gerstung 
Managing Director 

 
Email: 

Connect via LinkedIn:

---

From: Doug Arnold <[email-masked]>
Date: Tue, 29 Jun 2021 17:10:02 +0000

PTP with embedded NTP messages would not be accepted by the equipment vendors who make PTP capable devices, and not be accepted by network operators who deploy PTP.  Even when the PTP ATHENTICATION TLV was defined people were concerned about making PTP messages longer.  The NTP messages would be redundant information that adds no value, accept for the attached ICV.  So why not just add the ICV and skip the NTP message.

I’ve spoken previously about replacing unicast PTP with NTP.  Technically that should work as NTP and unicast PTP basically do the same thing in a slightly different way.  Large data centers operators, who are fond of leap smears, will probably want to stay with NTP, even as their timing agreement requirements are trending down to the 10 microsecond range.  Finance network operators are flexible and will use whatever works, so there will probably be some PTP and some NTP deployments, both of which should be secure.

However some industries are wedded to unicast PTP.  Telecom is heavily invested in PTP for wireless back haul timing.  The investment is in their standards, in product development and network deployment. Their contributions to the definition of NTPv4 were rejected,  but IEEE 1588 made significant changes to PTP to support them. So this community is not likely to move from PTP to NTP.  Ditto for the power industry, who sometimes use unicast PTP between substations.  Grid operators are used to following the lead of telecom for inter substation communications.  Note that the IEC republished the IEEE 1588 standard as IEC 61588.

The broadcast, Industrial automation, automotive, and test and measurement industries are also heavily invested in PTP, but use multicast with on path support. So they will want a group key management protocol such as GDOI, or something like NTS4PTP.

Doug

From: ntp <[email-masked]> on behalf of Heiko Gerstung <heiko.gerstung=[email-masked]>
Date: Tuesday, June 29, 2021 at 7:18 AM
To: Miroslav Lichvar <[email-masked]>
Cc: [email-masked] <[email-masked]>
Subject: Re: [Ntp] NTP over PTP

   Heiko

--
Heiko Gerstung
Managing Director

Email:

Connect via LinkedIn:

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 30 Jun 2021 12:06:37 +0200

> Which is not the same as claiming that PTP assumes the "network has a constant delay". 

Ok, let's try again. It is assumed to be constant in the interval
between its measurement and the offset measurement. The delay is
measured periodically in order to adapt to changes in network
configuration and topology. It's not assumed to change frequently.

> Wrong, even without full on-path support unicast PTP uses delay requests/responses to take the client-to-server delay into consideration as well. See IEEE1588-2019 Subsection 11.3 for a description of how this works.

Well, yes, but the delay measurement is separate from the offset
measurement. If you can log and plot the offsets measured by PTP and
NTP in a network without PTP support, you will see how the
distribution is different due to the different calculation.

> Yes, there are implementations which take that into account by applying static correction values to compensate for link speed asymmetry. I believe this also affects NTP, but in most cases hundreds…

NTP is much less impacted as it timestamps the end of the reception.
A software timestamp is captured after the packet is received.
Hardware timestamps are transposed as described in this document:
https://www.eecis.udel.edu/~mills/stamp.html

In my testing with several different switches the ideal point of
the transposition was around beginning of the FCS or a couple octets
before it, instead of the end. I think the explanation is that it
compensates for the preamble. In either case the error was much
smaller than if it was not transposing at all.

> There are more challenges I see for NTS-over-PTP. You need to synchronize the clock of the hardware timestamper itself, i.e. getting the time into the silicon that creates the timestamp. PTP timest…

The same approach can be used with an NTP implementation. The hardware
can keep time in TAI as long as the TAI-UTC offset is known.

> On a client you have to synchronize your system time with the time of the hw timestamper (e.g. the NIC). That time is synchronized by the hardware itself to the PTP server. PTP4L uses phy2sys for this, but I am not sure about the accuracy with which you can read out the PHC clock and correct the OS clock with it. There is a delay when accessing a NIC over the PCI(e) bus, but this is affecting PTP in the same way. So for the client, you should be on par with PTP in this regard.

I don't see a difference between PTP and NTP in this aspect. You can
use the protocol to synchronize the NIC or the system clock, directly
or indirectly. The PCIe latency is an issue for the system clock
either way. Some NICs support PTM, which is an NTP-like protocol for
PCIe with hardware timestamping, which can be used to avoid the error
due to asymmetric PCIe latency.

> But for a server you have to find a NIC that supports feeding the PPS of your GNSS receiver (for example) to it, not impossible but also not an easy task for someone who is responsible for maintaining highly accurate synchronization for an entire corporate network. 

Same applies to both NTP and PTP. The Intel I210 is a popular NIC for
these use cases.

> The next challenge is on the server, which for unicast PTP requires a certain timestamp queue size to support a usable number of clients. A lot of NICs that claim they have IEEE1588 hardware suppor…

Isn't that an issue for both NTP and PTP unicast using high rate sync
and delay requests?
> Finally, I am not sure if IEEE1588 would be happy about an IETF standard "hijacking" one of their protocols, but most probably they cannot do anything about it. Personally I think it is a hack and…
I think the best approach is for the hardware to timestamp all packets
as many NICs already do. The problem is with existing hardware that
cannot do that. I agree it doesn't look great when you have to run NTP
over PTP, but that seems to be the only way to get the timestamping
working on this hardware.

-- 
Miroslav Lichvar

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 30 Jun 2021 12:18:58 +0200

On Tue, Jun 29, 2021 at 05:10:02PM +0000, Doug Arnold wrote:
> PTP with embedded NTP messages would not be accepted by the equipment vendors who make PTP capable devices, and not be accepted by network operators who deploy PTP.

Do you speak for all vendors and operators?

> Even when the PTP ATHENTICATION TLV was defined people were concerned about making PTP messages longer.  The NTP messages would be redundant information that adds no value, accept for the attached ICV.  So why not just add the ICV and skip the NTP message.

Yes, you can do that, but you will not get all of the security of
NTS4NTP, which some people here seem to be interested in. You can
think of the NTP TLV as an authentication TLV that provides you with
that. It has to be longer as it contains the keys.

> However some industries are wedded to unicast PTP.  Telecom is heavily invested in PTP for wireless back haul timing.  The investment is in their standards, in product development and network deployment.

I'm somewhat familiar with the ongoing efforts in the telecom
industry. From what I understand, they are not going to use unicast
PTP in the new-generation networks. It's only for the older
generations as it cannot meet the new requirements. They need full
on-path support (i.e. G.8275.1) and in some cases even SyncE. Please
correct me if I'm wrong.

> Their contributions to the definition of NTPv4 were rejected, 

This is the interesting part for me. Is the discussion archived
anywhere?

-- 
Miroslav Lichvar

---

From: Heiko Gerstung <[email-masked]>
Date: Wed, 30 Jun 2021 15:42:53 +0200

> configuration and topology. It's not assumed to change frequently.

I understand your argument that the delay is assumed to be constant in between two delay measurements, and I agree with that. PTP allows the client to define the interval of those delay measurements. This means a client can decide how often it wants to measure the client-to-server part of the roundtrip. In a network where the assumption that this does not change frequently is made, the delay measurements in the client-server direction can happen in longer intervals. However, if a client does not know how stable the network delay is or in cases when it expects it to be highly dynamic, it can send delay requests with the same rate/interval as the sync messages (which are used to measure the server-to-client delay and calculate the offset). 

So PTP simply supports both cases: a network environment where delay changes are assumed to happen infrequently (in this case you choose a longer interval for the delay measurements and therefore save traffic and processing time) or a highly dynamic network environment where you need to do the delay measurements more often. 

As I said earlier, this is conceptually inferior to the NTP delay measurement mechanism which happens at the right time and ensures that the deltas of t4-t1 and t3-t2 are each based on the same clock (client and server), which is not true in PTP. But it is still working great ;-) ...
 
>> Wrong, even without full on-path support unicast PTP uses delay
>> requests/responses to take the client-to-server delay into consideration as
>> well. See IEEE1588-2019 Subsection 11.3 for a description of how this works.

> distribution is different due to the different calculation.

This depends, as pointed out earlier, on the network itself. If you have highly dynamic network paths where traffic patterns change quickly and therefore result in wildly changing queueing and forwarding delays, you will see slightly different delay distribution in NTP and PTP logs. In less dynamic environments, you will probably not notice something like that at all. And in networks with partial or full on-path support, you will see that PTP is performing great with its inferior delay measurement approach. 
 
>> Yes, there are implementations which take that into account by applying
>> static correction values to compensate for link speed asymmetry. I believe this
>> also affects NTP, but in most cases hundreds of nanoseconds are not a problem
>> for applications relying on NTP synchronization.
> 

That is a very confusing argument. 

We want to measure the delay of a server-to-client message (sync packets in PTP, NTP responses in NTP) in order to be able to compensate for the delay. There are three contributors to the error budget that you want to eliminate:

a) the latency of the packet in the server, i.e. from the point the software daemon wrote the timestamp into the packet until it actually goes out on the wire via the NIC port

b) the latency of the network, i.e. the time it took the packet from traveling from the server's network port to the network port of the client (passing through an arbitrary number of switches and routers which all contribute to the delay in addition to the mostly static delay of the cable)

c) the latency of the packet in the client, i.e. from the point it arrived at the network port of the client until it reaches the software client.

Both (a) and (c) depend on a lot of factors, a few of them pretty static (CPU power, kernel + driver and network stack code) and some highly dynamic (current workload of the system, activity on unrelated hardware, e.g. disk access). Using hardware time stamping right at the network port eliminates the highly dynamic part of (a) and (c). In addition, commercial PTP products also calibrate phy delays for both TX and RX path and use the measured values to correct for those delay (this also addresses phy delay asymmetry issues). The reason why VMs are not easily synchronized is mostly (a) and (b) and the fact that for a VM there are zero static factors.

If (b) would only be a direct cable between the server and the client, it would be *very* static. If it consists of multiple hops and switches/routers, it can be highly dynamic too. The PTP concept of a transparent clock tries to significantly eliminate the dynamic part of (b) by measuring the delta between a packet arriving at the router and leaving it on another interface. That delta is added to the correction field by every switch/router that the packet passes and the client in the end can deduct the resulting total value from its calculated delay. That way PTP eliminates most of the variable part of the network. 

In a network without any PTP support, hardware time stamping would still deal with (a) and (c) and would improve the accuracy. 

Please note that PTP typically uses the hardware clock implemented in the PTP network interface instead of the software clock maintained by the OS. This way you entirely remove (a) and (c) from the error budget. 

When we connect one of our grandmaster clocks (PTP server) to one of our PTP client devices over a direct crossover cable or fiber link, we can measure an offset of less than 20ns between the PPS generated by the GNSS synchronized master clock and the client. 

> smaller than if it was not transposing at all.
The good thing here is that for a given link speed the delay from octet 1 to octet n in one received frame is pretty static. That means, you can timestamp wherever you want (as soon as you identified that is is a packet that needs to be timestamped) if you calibrate your timestamper engine to add that static delay. Some timestamper engines also simply take a timestamp for every packet at the very beginning and decide whether they want to store that timestamp in their TS queue once they identified that this is a PTP event message. 
 
>> There are more challenges I see for NTS-over-PTP. You need to synchronize the
> clock of the hardware timestamper itself, i.e. getting the time into the
> silicon that creates the timestamp. PTP timestamps are TAI (not UTC), which
> itself is not a problem as long as you know the TAI-UTC offset. On a server
> (PTP Grandmaster) this is typically done by using some form of hardware sync
> for the timestamper engine, e.g. setting the ToD to the upcoming TAI second and
> then use the PPS to zeroize the fractions. In reality the solution is typically
> more sophisticated as you do not want to see micro timesteps at the start of
> every second.
> 

Definitely yes, it needs to be implemented but is certainly solvable. 

>> On a client you have to synchronize your system time with the time of the hw
>> timestamper (e.g. the NIC). That time is synchronized by the hardware itself to
>> the PTP server. PTP4L uses phy2sys for this, but I am not sure about the
>> accuracy with which you can read out the PHC clock and correct the OS clock
>> with it. There is a delay when accessing a NIC over the PCI(e) bus, but this is
>> affecting PTP in the same way. So for the client, you should be on par with PTP
>> in this regard.
> 

Yes, I agree. Same for NTP and PTP. 
 
>> But for a server you have to find a NIC that supports feeding the PPS of your
>> GNSS receiver (for example) to it, not impossible but also not an easy task for
>> someone who is responsible for maintaining highly accurate synchronization for
>> an entire corporate network.
> 

Yes, for sure. Requires a lot of tinkering but it is solvable, too. 

>> The next challenge is on the server, which for unicast PTP requires a certain
>> timestamp queue size to support a usable number of clients. A lot of NICs that
>> claim they have IEEE1588 hardware support have small to tiny ts queue sizes,
>> one common exampe is 4 timestamps. That means you have to be able to read out
>> the hardware timestamps very quickly and you will not really have a chance on
>> high speed links with hundreds and thousands of incoming NTS-over-PTP requests
>> per second.  Those hardware timestamping engines have been designed to be used
>> for PTP clients only, and even then not for the high packet rates that PTP
>> supports (and sometimes requires to improve accuracy over partial
>> on-path-support networks). They cannot be used for servers expecting to handle
>> a high packet rate.
> 

Yes, but unicast PTP servers with hardware time stamping are typically commercial products with dedicated network hardware to support the large number of timestamps that have to be stored and processed. The i210 has a time stamping queue length of 4, i.e. it stores a maximum of 4 timestamps in a ring buffer. A unicast PTP server with 10 clients running at 128 sync and delay pkt/s will generate 1,280 time stamps per second. Even on some serious hardware there will be a significant number of timestamps that are lost due to queue overruns.  

Our devices support 2,048 PTP clients on every Gigabit port, each client running at 128 sync pkt/s and 128 delay req pkt/s. This is only possible by designing your own hardware time stamping engine. What I wanted to say is that it will be very hard to build an NTS-over-PTP server which supports a reasonable number of NTS clients. Not impossible, of course. But a vendor will try to calculate whether the effort is worth it or not. There are solutions in the field that support hardware time stamping for NTP (our own products can do that, no need to put the NTP packets into a unicast PTP packet), and there are also solutions for hardware NTS (the Netnod implementation for example). 

The choice seems to be to either try and convince switch vendors and NIC manufacturers to add hw time stamping support for NTP(+NTS) or to convince PTP server vendors to implement NTS-over-PTP to make use of the PTP hw time stamping capabilities of the NICs and maybe also the switches etc. The latter seems to be the easier task, but without succeeding here, the approach might be a dead end. 

>> Finally, I am not sure if IEEE1588 would be happy about an IETF standard
>> "hijacking" one of their protocols, but most probably they cannot do anything
>> about it. Personally I think it is a hack and should not be standardized, but
>> that's just me. I would rather like to see some standard way of flagging an
>> Ethernet frame that I send out to trigger a hardware timestamping engine to
>> timestamp that frame. Such a universal approach could be used by NTP, PTP and
>> other protocols and applications as well (not only time sync protocols), for
>> example to measure network propagation delays etc. It is incredibly hard to get
>> support for this into the silicon of companies like Intel or Broadcom etc., but
>> if it would be universal enough, the chances are higher that it will make its
>> way into products eventually.

I believe there should be a standard that adds a hardware timestamp to the end of every Ethernet frame. It requires a NIC vendor to implement a hardware clock and a time stamping engine into their silicon. The bandwidth between the NIC and the OS layer (driver) has to accommodate the extra data (but you could reduce the MTU of course). 

I know from personal experience how long it took to convince the big NIC manufacturers to add something like that to their products and I will not hold my breath for it. I do not know how long it takes IEEE 802.3 to standardize something like this, but before that has been done, those manufacturers will definitely not do it IMHO. Bottom line: I will not hold my breath ...

Regards,
   Heiko

-- 
Heiko Gerstung 
Managing Director 

 
Email: 

Connect via LinkedIn:

---

From: Doug Arnold <[email-masked]>
Date: Wed, 30 Jun 2021 16:00:02 +0000

Hello Miroslav

RE: acceptance of NTP over PTP
Obviously I do not speak for everyone, and I could be mistaken. But I talk to a lot of vendors and network operators and this is my take on where they are coming from.  I also believe that I have a good sense of the PTP standards world and doubt that anyone will include NTP over PTP in a PTP Profile.  NTP over PTP is a clever hack, and might be done quickly, but I think people will want something cleaner with fewer redundant message fields.  People in standards meetings frequently talk about the importance of keeping PTP messages small.

RE: use of unicast PTP in telecom:
Yes, telecom will use full on path support as soon as they can, and switch to G.8275.1 with SyncE.  But that is probably a decade out for some carriers.  Telecom grade switches are just too expensive to replace because they don’t support PTP. They replace them on a schedule that is planned for years in advance.  An active area of discussion in the ITU these days in defining the “interworking function.”  This is basically a boundary clock that translates between two PTP profiles, mainly to facilitate the gradual transition from G.8275.2 to G.8275.1.

RE: History of the ITU, NTP WG and IEEE 1588
About twenty years ago, after the telecom industry changed to Ethernet and packet switched networks, they knew that they needed packet based time transfer.  All of the telecom equipment vendors started presenting lab research papers at the annual T1 X1 workshops (now called WSTS) on what they called “carrier grade NTP”.  By this they meant NTP packets, but with high message rates, and different algorithms for clock steering and server selection.  Some people in the NTP WG proposed separating the over the wire protocol from the algorithms, just as you have done in your draft NTPv5.  Others in the working group wanted to keep NTP as a coherent complete solution for general IT use cases, as it traditionally had been.  The complete solution camp prevailed.  I can understand that.  NTP is spectacularly successful at meeting the general IT needs.

I went to an IEEE 1588 meeting in the early days of version two.  We were surprised when twice as many people showed up as had attended previously.  All of the new people worked for telecom equipment vendors, and were members of the ITU-T.  They explained to us that they had given up on carrier grade NTP and hoped that IEEE 1588 would help them.  That was about 2005.  After a few fierce arguments we agreed to make changes to PTP that they wanted.  Among the concessions was unicast PTP.  So the telecom folks got their carrier grade NTP after all.

If you want another take on this ask anyone who was in these groups 15 years ago: the NTP WG, IEEE 1588, ITU-T Q13/15.

RE: NTPv5 and protocol algorithm split
Some people in the finance industry have recently reinvented carrier grade NTP and prefer it to PTP.  They like that clients getting time from multiple servers is the norm, and the clients figure out which servers to believe, rather than running a BMCA as in PTP.

That is why I actively support the separating of protocol and algorithms in NTPv5, including supporting your proposal.  I may be the chair of IEEE 1588, but I’m not interested in turf wars.  I want the network operators to have everything they need, including a more flexible NTP, and a more secure PTP.

Doug

From: Miroslav Lichvar <[email-masked]>
Date: Wednesday, June 30, 2021 at 6:19 AM
To: Doug Arnold <[email-masked]>
Cc: Heiko Gerstung <heiko.gerstung=[email-masked]>, [email-masked] <[email-masked]>
Subject: Re: [Ntp] NTP over PTP

--
Miroslav Lichvar