Subject: Re: [Ntp] draft-ietf-ntp-mac, extensions

From: Miroslav Lichvar <[email-masked]>
Date: Thu, 1 Mar 2018 09:44:14 +0100

On Wed, Feb 28, 2018 at 11:06:36PM -0800, Hal Murray wrote:
> I'm interested in the simple case of shared/symmetric key authentication.

> Why is AES-CMAC more interesting than one of the digests supported by OpenSSL?

As I understand it, the main advantage is better performance (when
implemented properly). For instance, on a server configured with a
SHA256 key I'm observing a 40% drop in performance when compared to no
authentication. With AES-CMAC the impact should be much smaller. I've
not tried it yet. However, I'm not sure how much this really matters
as (public) servers typically don't handle a large number of packets
authenticated with symmetric keys.

> The current code and extension RFCs and proposals only supports digest 
> lengths of 16 and 20 bytes.  I was expecting an extension code for shared 
> keys so we could handle any length, but I don't see one on the IANA list.  Is 
> anybody else interested in that case?

The trouble is with parsing the packets. The strict rules for lengths
of extension fields and MACs are needed to prevent ambiguity.
Otherwise, a long MAC could be incorrectly interpreted as an extension
field, or a short extension field could be interpreted as a MAC.

This was a frequent topic on this list, so I'd suggest to go through
the archives if you are interested in more details.

Please note that latest versions of ntp truncate long digests in MACs
to 160 bits, so the authentication should work with any hash function
supported by OpenSSL.

-- 
Miroslav Lichvar

---

From: Harlan Stenn <[email-masked]>
Date: Thu, 1 Mar 2018 00:54:30 -0800

> field, or a short extension field could be interpreted as a MAC.

There is no difficulty in parsing new EFs with legacy MACs.  I've been
doing this for nearly a year now.

-- 
Harlan Stenn <[email-masked]>
http://networktimefoundation.org - be a member!

---

From: Hal Murray <[email-masked]>
Date: Thu, 01 Mar 2018 14:36:45 -0800

Thanks.

> implemented properly). For instance, on a server configured with a SHA256
> key I'm observing a 40% drop in performance when compared to no
> authentication. With AES-CMAC the impact should be much smaller. I've not
> tried it yet.

Intel has hardware support for AES on some chips - AES-NI.  On Linux boxes, 
it will show up as an "aes" in the flags in /proc/cpuinfo
  grep aes /proc/cpuinfo

How are you measuring performance?

My measurements are that the CMAC part is 696 ns in software and 499 ns with 
AES-NI.  That was on a 2+ GHz cloud server.  (None of my handy boxes have 
AES-NI.)  SHA512 was 594 with 16 or 20 byte key and 624/614 with 24/32 byte 
keys.  That's just the crypto part to compute the digest.  X2 for transmit 
and receive and round up for getting there and deciding which key to use and 
copy or compare and such.

On a PI-2B, AES-CMAC is 10242 ns.  SHA512 digest is 13500 ns.

-- 
These are my opinions.  I hate spam.

---

From: Matthew Van Gundy <[email-masked]>
Date: Thu, 1 Mar 2018 18:40:45 -0500

> Why is AES-CMAC more interesting than one of the digests supported by OpenSSL?

The keyed-hash construction that NTP uses is not a secure message
authentication code, it is vulnerable to the classic length extension
attack [1] when used with common hash functions, such as MD5, SHA-1,
and the SHA-2 family of hash functions.  This allows an attacker to
append additional forged NTPv4 extension fields on to an authenticated
NTPv4 packet.  draft-ietf-ntp-mac-03 refers to this attack through its
reference to the BCK paper in section 2 [2] even though it does not
explicitly call out length extension attacks in the body of the draft.

In current usage, this rarely poses a problem.  In the NTP reference
implementation extension fields are only used by Autokey.  If you have
Autokey disabled, it does not give an attacker any additional power.
However, if NTPv4 extension fields catch on for uses other than
Autokey, this will become problematic.

AES-CMAC is a secure message authentication code and, therefore, is
not vulnerable to this kind of attack.

[1] https://en.wikipedia.org/wiki/Length_extension_attack
[2] https://tools.ietf.org/html/draft-ietf-ntp-mac-03#section-2

Cheers,
Matt

--
Matthew Van Gundy, Technical Leader
Advanced Security Initiatives Group
Cisco Systems, Inc.

---

From: Miroslav Lichvar <[email-masked]>
Date: Fri, 2 Mar 2018 10:11:45 +0100

On Thu, Mar 01, 2018 at 06:40:45PM -0500, Matthew Van Gundy wrote:
> NTPv4 packet.

Does this actually apply to NTPv4? Do you have any examples how could
an attacker append a valid extension field to an NTPv4 packet?

My understanding is that the NTPv4 packet format (with or without
RFC7822) prevents the length-extension attack as extension fields
cannot have a length of zero, which would come from the padding
(always starting with 0x80000000). Even if the length field could be
zero, there is no EF type 0x8000 assigned yet and it could be avoided
in future in case people are for some reason still using the old MAC.

-- 
Miroslav Lichvar

---

From: Miroslav Lichvar <[email-masked]>
Date: Fri, 2 Mar 2018 10:33:16 +0100

> How are you measuring performance?

I was measuring the maximum rate of requests the NTP server could
handle, so that includes all overhead of receiving and sending
packets in the OS, etc.

> keys.  That's just the crypto part to compute the digest.

Does that include the time spent in allocation of the CMAC context and
the initialization with the key? I think an optimized implementation
could avoid both. (With the old MAC the key has to be hashed with each
message).

Anyway, older versions of the ntp-mac draft included some measurements
of latency and throughput, which showed that with AES-NI the CMAC is
significantly faster than MD5, which should be significantly faster
than SHA512.

https://www.ietf.org/archive/id/draft-aanchal4-ntp-mac-03.txt

I think the only downside of AES-CMAC is that it is not widely
supported in crypto libraries yet. OpenSSL seems to be an exception.

-- 
Miroslav Lichvar

---

From: Matthew Van Gundy <[email-masked]>
Date: Fri, 2 Mar 2018 10:08:37 -0500

Hi Miroslav,

> in future in case people are for some reason still using the old MAC.

I was speaking in general terms. I have not actually proved out an
attack against an NTPv4 implementation.  I believe you are correct:
since most hash functions in common use pad the last block with a
binary string beginning with 0x80000000 or 0x00000000, an NTPv4
implementation would parse a length extended packet as an extension of
length zero, and would probably discard it (or fail to process
additional extensions in a meaningful way).

However, there is no requirement that hash functions pad the last
block with any specific value.  Though it may be unlikely, this is a
brittle assumption that could be invalidated by future hash functions.
Using secure message authentication codes such as CMAC or HMAC avoids
this potential failure case.

Cheers,
Matt

---

From: Hal Murray <[email-masked]>
Date: Tue, 06 Mar 2018 16:34:06 -0800

[email-masked] said:
> Does that include the time spent in allocation of the CMAC context and the
> initialization with the key? I think an optimized implementation could avoid
> both. (With the old MAC the key has to be hashed with each message). 

I pre allocated the context but I didn't preload the key.  I don't have any 
man pages for the CMAC routines. ??  I see CMAC_CTX_copy.  I assume  that's 
what you are referring to.  I'll add it to my list.  Thanks for the 
suggestion.

Looks like there is a EVP_MD_CTX_copy.  I assume it can be used in the same 
way for digests.

> Anyway, older versions of the ntp-mac draft included some measurements of
> latency and throughput, which showed that with AES-NI the CMAC is
> significantly faster than MD5, which should be significantly faster than
> SHA512.

Thanks.

(The link actually goes to a file that calls itself 
draft-aanchal4-ntp-mac-02, Oct 2016.  It does have lots of performance data.)

---

From: Harlan Stenn <[email-masked]>
Date: Mon, 26 Mar 2018 13:10:20 -0700

I like that it's on the Experimental track.

It's clear that this proposal is geared to being implemented
"internally", on a LAN, as opposed to "externally", on a WAN.

Am I correct that the proposal augments the receive and transmit
corrections to *add* 16 more bits to the fractional portions of the
receive and transmit timestamps, taking them from 32-bit values (about a
quarter of a nanosecond) down to the femtosecond range?

Similarly, this proposal gives origin and delay corrections that can
represent an 18 hour correction down to the femtosecond?

I'll hold further comment on these items until I'm sure I understand it
correctly.

How can the Origin ID and Path ID fields be used in practice?

We should earmark an EF type for this proposal.  0x0010 is the next
unused number.

What is the relationship between the ID numbers of the incoming and
outgoing ports and the Origin and Path IDs?  OK, having read even more
this is just the Path ID.

On the delay correction, if the transmission starts before the reception
has completed, the resulting correction will still be net-positive,
correct?  Please tell me how this process will result in a negative
value being added to the delay.

Also, is the integer portion of the delay signed, while the fractional
portion is unsigned?  If so, we're now talking about +/- 9 hours on the
correction fields, right?

Paragraph 3 of Section 4 says:

   If the server's clock has a better precision than resolution of the
   64-bit NTP timestamp format, the server SHOULD save the additional
   bits in the receive and transmit correction fields and set the
   precision field to the corresponding number, which is smaller than
   -32.  Otherwise, the receive and transmit correction fields SHOULD be
   zero.

I'm curious about these server clocks that have precision better than a
quarter of a nanosecond.  And the precision field that SHOULD/MUST be
updated in the base packet to be better than -32 - this must happen
before any MAC is calculated, right?

Paragraph 6 of Section 4 is the first time it became clear to me that
the receiving system takes the incoming Path ID and copies it to the
Origin ID.

What good are any of the extended fractional components if the devices
along the path do not have at least that level of precision?  These
devices are using difference clocks, not absolute clocks, right?

To what extent is it useful to know what the timing precision is of each
device that updates the correction field?

Would you make any changes to this document if either/both:

 draft-stenn-ntp-extension-fields

 draft-stenn-ntp-mac-last-ef

were adopted?

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 27 Mar 2018 10:47:06 +0200

> "internally", on a LAN, as opposed to "externally", on a WAN.

Yes, LAN would be the typical use case, but I think there shouldn't be
anything which would prevent people from using it (intentionally or
not) in WAN.

Unlike with PTP, which expects all switches/routers between the clocks
to support PTP (as a boundary clock or transparent clock), the NTP
correction field could be implemented gradually in a network,
improving the accuracy one step at a time. It would be perfectly ok to
mix devices that support the correction field with devices that don't.

> quarter of a nanosecond) down to the femtosecond range?

Yes.

> represent an 18 hour correction down to the femtosecond?

Yes, except it's a signed number so the interval is about [-9, 9] hours.

> How can the Origin ID and Path ID fields be used in practice?

The idea is that they will help with detecting asymmetric paths, which
seems to be a common problem when trying to get a highly accurate
synchronization in a network where multiple paths exist.

> value being added to the delay.

In some cases (switching between different link speeds or changing
length of the packet) the devices are required to timestamp the end of
the reception and beginning of the transmission. If the transmission
started at X and the reception ended at X+100 nanoseconds, the
correction field should be updated with -100 nanoseconds.

> quarter of a nanosecond.

Have a look at the White Rabbit project. They are working with clocks
that have a precision in tens of picoseconds (and accuracy in hundreds
of picoseconds). If someone wanted to do something similar with NTP
instead of PTP, the correction field provides NTP with sufficient
resolution.

> And the precision field that SHOULD/MUST be

Right. I'd expect the packet to have the <-32 precision since its
beginning, there is no need for updates.

> along the path do not have at least that level of precision?

Then they just won't contribute to improvement of the accuracy.
Nothing bad should happen. Are you suggesting to add a new field to
the EF that would be updated with the worst precision on the path?

> These

I guess that would be an implementation detail.

> device that updates the correction field?

That's a good question. I'm not sure.

>  draft-stenn-ntp-mac-last-ef

I suspect some change to significantly reduce the probability of the
corruption would be needed if the ambiguity in parsing was allowed.

Thanks for the comments.

-- 
Miroslav Lichvar

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 27 Mar 2018 03:01:52 -0700

> Yes, except it's a signed number so the interval is about [-9, 9] hours.

OK.  I wonder what the need is for these extra fractional bits in
"normal" networks.  It certainly is expensive in terms of packet length.
 But if you're trying to find a use for a 28 octet EF...  I think there
may be better ways to go than this, including ways to track the delay
added at each hop.

> synchronization in a network where multiple paths exist.

I suspect asymmetric paths and bufferbloat are bigger problems.

> correction field should be updated with -100 nanoseconds.

This still doesn't make any sense to me.  I suspect a bit more altitude
would help.  Are you describing:

 t0: incoming packet starts arriving
 t1: outgoing packet starts transmitting
 t2: incoming packet finishes arriving
 t3: outgoing packet finishes transmitting

and it's possible that t2-t0 is larger than t3-t1.  It's also possible
that the incoming packet finishes arriving before the outgoing packet
starts transmitting.

But the delay thru the device is still t3-t0, which is a positive value.
 Perhaps a better value for the delay is (t3-t1)+(t1-t0).

> instead of PTP, the correction field provides NTP with sufficient

OK, and given the sizes of stuff we're talking about, and the possible
change to allow separate corrections at each hop, there may well be a
significant benefit to having both "appropriately sized" correction
fields, and per-hop corrections (and per-hop precision).

> beginning, there is no need for updates.

But what if the devices along the path have significantly better or
worse precision?

> the EF that would be updated with the worst precision on the path?

I'd rather see the precision and correction for each path along the way.
 But I also haven't thought about this very thoroughly.

> corruption would be needed if the ambiguity in parsing was allowed.

I don't really understand what you're saying.  draft-stenn-ntp-extension
field discusses 3 ways to go.  Two of them have no parsing ambiguity.
The third describes how to address things in the face of a parsing
ambiguity in the case where a legacy MAC is provided.

For this proposal, there CANNOT be a legacy MAC followed by an EF, and
since this proposal requires a MAC-EF before the correction EF in the
case where the first part of the packet is covered by a MAC, there
obviously cannot be any parsing ambiguity when this EF is used.

If this proposal had a case where its EF length was less than 28 octets,
it *still* wouldn't cause a problem if a LAST-EF came at the end of the
packet.

Note that another solution for this proposal would use 2 EF subcodes.

The first would be sent with room for a path count and an origin count:

 0x0110	Correction: count request
 0x8119 Correction: count response

and this would give the originator the count of the number of devices
along the path that can offer a correction computation.  Once this was
done, the next exchange could be:

 0x0210 Correction: calculation request (with N empty data slots in it)
 0x1210 Correction: calculation response (with N data slots filled in)

Note that in the 0x0210 message, each path along the way could put a '1'
in the next available slot, to make sure that there are enough slots to
hold the corrections along the path from the requestor to the responder.

There could/should be space in there for "how many devices along the
path did not have an available slot to hold a correction", and this
value could be checked similarly to the Origin ID and Path ID, to make
sure there was enough room for all of the corrections.

Note that with this mechanism, the packet stays pretty much the same
size all along the path, in each direction.

I also note that if we pursue this path, we can continue using the same
EF type, and make future changes with additional EF codes.

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 27 Mar 2018 12:33:53 +0200

> may be better ways to go than this, including ways to track the delay

If we could avoid ambiguous parsing with a shorter extension field,
then yes, we could probably save few octets, but certainly not enough
to add a second pair of the origin/delay correction fields.

> I suspect asymmetric paths and bufferbloat are bigger problems.

Well, the IDs are supposed to detect an asymmetric path and the delay
corrections are supposed to fix the effect of bufferbloat on
the NTP measurements.

>  Perhaps a better value for the delay is (t3-t1)+(t1-t0).

The delay should be updated with t1-t2 (which is negative in your
example), not t3-t0.

> fields, and per-hop corrections (and per-hop precision).

How many per-hop corrections can you fit in a short constant extension
field? And what exactly would you do with that information?

> obviously cannot be any parsing ambiguity when this EF is used.

The problem is in the other direction. A packet with a long legacy MAC
could be misinterpreted by the switch as one or more extension fields,
including the Correction Extension field. I hope you agree that a
packet following the NTP specification should not be corrupted in the
network.

> hold the corrections along the path from the requestor to the responder.

This could work, but I'm not sure if it's worth the extra complexity
and longer packets.

-- 
Miroslav Lichvar

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 27 Mar 2018 04:13:51 -0700

> to add a second pair of the origin/delay correction fields.

Where is the ambiguous parsing here?  In my proposal you could trivially
avoid the ambiguous parsings.   Adding more correction fields is easy if
they're not so big.  It might even be useful to go to scientific
notation and appropriate significant digits.

Also, if the receiving system has a precision limit of, say, -18, of
what possible use is knowing the correction to better than even -19?

> The delay should be updated with t1-t2 (which is negative in your

I'll re-read, because I don't see that t1-t2 is a useful value.

> field? And what exactly would you do with that information?

It's not a short constant extension field.  Depending on how many bits
are needed for this, it's possible that 16 bits are adequate for the
correction field for each hop.

Is only one instance of the origin correction needed?  The number of
bits needed for that is also limited by the precision of each endpoint,
right?  Assuming for the moment that 16 bits in a floating point
representation is adequate, at least for this thumbnail calculation,
then you can track 11 hops with a 28-octet EF for this.

> packet following the NTP specification should not be corrupted in the

I thought you CANNOT uses a legacy MAC with this proposal, as the legacy
MAC MUST be at the end of the packet, and these corrections MUST come
after the MAC, which means "use MAC-EF or NTS MAC, not legacy MAC."

>>  0x8119 Correction: count response

    0x8110 is what I meant above.

> This could work, but I'm not sure if it's worth the extra complexity

You might be right that a single delay correction is all that's needed.
And you're looking at an experimental proposal here, so you could also
implement your original proposal using 0x0010 for the outgoing format,
and 0x8010 on the response.  If there's value in having per-path
updates, what I suggested might be useful.  The good news is that it
will be pretty easy to figure these things out.  These options and more
can be easily tested like this, without burning thru EF types.  With
this, the EF length can be changed as-needed.

---

From: Hal Murray <[email-masked]>
Date: Tue, 27 Mar 2018 04:17:34 -0700

[email-masked] said:
> The problem is in the other direction. A packet with a long legacy MAC could
> be misinterpreted by the switch as one or more extension fields, including
> the Correction Extension field. I hope you agree that a packet following the
> NTP specification should not be corrupted in the network. 

How does a switch decide that it should modify a packet?  Is it going to 
modify all NTP packets?  Is there something at a lower level header that says 
"modify this packet"?

Is this a good time to start getting rid of legacy MACs?  If a switch is 
going to update a new EF, the ends are going to need new software to process 
that EF.  Why not just arrange that there is some way for that new software 
to not use legacy MACs?

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 27 Mar 2018 13:34:01 +0200

> notation and appropriate significant digits.

Well, it's obvious I'm not able to explain it to you. Maybe someone
else will have more luck.

> what possible use is knowing the correction to better than even -19?

No use. It's the same as with the transmit and receive timestamps in
normal NTP packets. Server responds with the best precision it has,
even if it's ntpdate saying it has only -6.

> correction field for each hop.

Even if we stick to the 1/4ns resolution of NTP, 16 bits is not enough
to describe all delays of a switch/router. It can easily be in tens of
milliseconds.

> Is only one instance of the origin correction needed?

I don't think it would make sense to have only one origin correction
and multiple delay corrections. If a client had a use for this information
it would probably need it in both directions.

> after the MAC, which means "use MAC-EF or NTS MAC, not legacy MAC."

The packets with MAC would be sent by clients that don't use or care
about the Correction Field.

-- 
Miroslav Lichvar

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 27 Mar 2018 13:38:52 +0200

> modify all NTP packets?  Is there something at a lower level header that says 

It's the presence of the Correction Field. If the switch sees a packet
with the field, it may modify it. But here we have a problem that if
the length of MACs is no longer restricted, the switch could interpret
a (part of a) random MAC as the Correction Field and break the MAC.

> that EF.  Why not just arrange that there is some way for that new software 

In NTPv5 we will hopefully get rid of it. For NTPv4 I don't think that
is possible.

-- 
Miroslav Lichvar

---

From: Hal Murray <[email-masked]>
Date: Tue, 27 Mar 2018 05:03:46 -0700

[email-masked] said:
> Even if we stick to the 1/4ns resolution of NTP, 16 bits is not enough to
> describe all delays of a switch/router. It can easily be in tens of
> milliseconds. 

I've seen bufferbloat on a DSL line with delays of almost 4 seconds.

---

From: "Ulrich Windl" <[email-masked]>
Date: Tue, 27 Mar 2018 14:09:41 +0200

>>> Miroslav Lichvar <[email-masked]> schrieb am 27.03.2018 um 13:34 in
Nachricht <20180327113401.GG26324@localhost>:

[...]
> to describe all delays of a switch/router. It can easily be in tens of
[...]
FYI: With virtual machines it's even possible to have more than one second PING delay in a LAN 8-(

Description: ping test succeeded [response time 1.560 s]
Description: ping test succeeded [response time 2.035 s]

Regards,
Ulrich

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 27 Mar 2018 14:05:32 -0700

> I've seen bufferbloat on a DSL line with delays of almost 4 seconds.

I'm still liking a floating-point representation for this.

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 27 Mar 2018 14:07:41 -0700

> Description: ping test succeeded [response time 2.035 s]

Floating point representation.

So how well does NTP work in this environment?

What is the clock precision?

What is the value in going from 32 to 48 fractional bits in this situation?

---

From: Hal Murray <[email-masked]>
Date: Tue, 27 Mar 2018 14:33:35 -0700

> So how well does NTP work in this environment?

My nasty case is horrible bufferbloat on a DSL line.  It works reasonably 
well if you turn on huffpuff.

Without huffpuff, the asymmetry is nasty enough that it will get clock jumps 
during big downloads, and jumps back when the download ends.

With huffpuff, it goes through the jumps if you restart ntpd while a download 
is in progress.

---

From: Harlan Stenn <[email-masked]>
Date: Tue, 27 Mar 2018 16:33:35 -0700

Hal, do you have any expectation that you’ll ever be able to get a DSL router that will implement this capability?

I’m just curious. It would be nice if this could be turned in to something that was both useful and available. 

Sent from my iPhone. Please excuse brevity and typos.

---

From: Hal Murray <[email-masked]>
Date: Tue, 27 Mar 2018 21:11:14 -0700

[email-masked] said:
> Hal, do you have any expectation that you’ll ever be able to get a DSL
> router that will implement this capability? 

Seems unlikely.  I think the major vendors would start by "fixing" 
bufferbloat, then declare victory and ignore any NTP issues.

Note that the hardware on DSL links in not symmetrical.  There is some 
competition for the modem/router box at the customer end.  Wifi is often 
included.  At least with some ISPs, you have a choice.  I expect there is 
some open source work in that area, but I don't know any details.

I think the box at the other end is called a DSLAM.  I doubt if the customer 
gets any choice.

I've seen comments about working on bufferbloat by interjecting an 
ethernet-ethernet box between DSL modem and LAN.  The idea is to do rate 
limiting on that box at slightly below the DSL link speed so there is no 
queueing at the DSL uplink and the downlink queue is under your control.  It 
would be easy to add code in that environment to update NTP packets.

I could easily magine research/hacker experiments that might get deployed 
outside the initial designer's lab.  Consider drones using Wifi, cell phones, 
or low tech ISM radio links.  Somebody might get interested in NTP 
corrections.

In any case, I think we should have enough bits for a few seconds or few 10s 
of seconds, rather than just 10s of ms.  You can do that by allocating lots 
of bits, or use something like a floating point format.  You could move the 
floating point part to the EF type - one type for fast links and another for 
slow links with the time slot scaled differently.  That doesn't scale to many 
bits of exponent, but it works for 1.

---

From: Miroslav Lichvar <[email-masked]>
Date: Wed, 28 Mar 2018 09:35:09 +0200

> I'm still liking a floating-point representation for this.

Why exactly?

If we assume the device may have a clock accurate to 1ppb, we would
still need about 32 bits for mantissa to represent those 4 seconds in
sufficient resolution.

Please keep in my mind that this needs to be as simple as possible.
Requiring the devices to implement a floating-point unit (operating at
a very high speed) would not be a good idea.

Realistically, if we don't allow sub-1/4ns resolution, the delay
correction fields should have at least 40 bits. With 1ps resolution
that would be 48 bits. That would save 32+16 bits in the extension
field.

We could use 16 of those bits to extend the path IDs and 32 to add new
correction fields for the root delay and root dispersion (the 32-bit
fixed-point format is too short in some applications).

-- 
Miroslav Lichvar

---

From: Harlan Stenn <[email-masked]>
Date: Wed, 28 Mar 2018 01:24:17 -0700

> fixed-point format is too short in some applications).

I too would love to have 40 fractional bits.

But in talking about path delays, for anything other than pure
experimental labs I'm not seeing the value in additional fractional bits.

The reason I like floating point is that with an IEEE 754 32-bit float,
we get 8 bits of exponent (which is huge for what we're looking at) and
about 24 bits of significand, which gives 6-9 decimal digits. Let's call
it 7 digits.  That's about a million, which is not quite enough.  But it
might be enough for many on-path devices when calculating delay.  If we
have 32 bits of significand, that's more than 9 decimal digits, which
gets us to a billion.

It might be that we could use 3 bits to tell us the size of the
significand, leaving us 5 bits for the exponent, then 32, 40, or 48 bits
of significand.  Just one choice.  I haven't wondered about 4 bits for
the size of the significand and 4 bits of exponent.

I'm wondering about dedicated values at each hop as that just seems to
be more useful.  Devices along the way don't need to do any floating
point processing for the delay values, and I still need to read more to
understand why the origin correction seems to be worked on so much.

But adding through-the-switch delays of tens of microseconds down to the
nanosecond only needs 6 decimal digits of precision.  I'm sure there are
wizards out there who could come up with even better (or more efficient)
numbers.

Here's another idea of the top of my head.  Use 4 different EF subcodes,
one using fixed-field numbers, the 2nd using 32-bit values, the 3rd
using 40-bit values, and the 4th using 48-bit values.  When sending the
initial "path probe" packet, each intermediate device increases the
counter for the field it wants to add to.  We get this information in
both directions (for a sanity check).  Throw a "hop counter" in there,
or some means to identify which device added which value, if that's
needed.  Anyway, once the packet arrives at its destination, we could
easily "integrate" the various corrections.  If we know the net delay
added by each reporting device, I don't think it matters what order we
use to apply these corrections.

Just an idea.

---

From: Martin Burnicki <[email-masked]>
Date: Wed, 28 Mar 2018 14:32:33 +0200

Harlan,

Harlan Stenn wrote:
[...]
> The reason I like floating point is that with an IEEE 754 32-bit float,

I really dislike using floating point for this purpose.

Many years ago I had to define a data structure to be exchanged between
a microcontroller and a PC, and some of the fields were IEEE 754
floating point numbers.

I had to learn that even though IEEE 754 defines the *format* how to
store the bytes, but it doesn't define in which order to *store* the
bytes in memory. The microcontroller only had software floating point
support and stored floating point numbers as a sequence of 16 bit words,
so it was required to swap each floating point number in a target
specific way to be able to handle it.

Processors in routers or switches may not be very powerful, so unless
the correction field is updated by FPGA hardware or so, the processor
had to swap the floats, do some arithmetic on it, and swap it back
again. Without hardware floating point support this takes *much* longer
than integer arithmetic.

IMO it would be much easier to use e.g. a 64 bit number with a high
resolution, e.g. 1 pico- or even femtosecond for the LSB. This would
still cover a huge range.

> understand why the origin correction seems to be worked on so much.

You don't know how many hops there will be between a server and a
client, so the packet length would vary, which affects the propagation
delay, as Miroslav has already pointed out.

Martin
-- 
Martin Burnicki

Senior Software Engineer

MEINBERG Funkuhren GmbH & Co. KG
Email: [email-masked]
Phone: +49 5281 9309-414
Linkedin: https://www.linkedin.com/in/martinburnicki/

Lange Wand 9, 31812 Bad Pyrmont, Germany
Amtsgericht Hannover 17HRA 100322
Geschäftsführer/Managing Directors: Günter Meinberg, Werner Meinberg,
Andre Hartmann, Heiko Gerstung
Websites: https://www.meinberg.de  https://www.meinbergglobal.com
Training: https://www.meinberg.academy

---

From: Harlan Stenn <[email-masked]>
Date: Wed, 28 Mar 2018 20:05:18 -0700

Martin,

> still cover a huge range.

What about the case where we want 10 or more of these values?  We have a
limited mount of space for EFs.

The intent here is to avoid using 64 bit fields when 16, 24, 32, 40, or
48-bit fields will do.

As long as we define the bit order, what problem do you see?  These
devices along the path just need to add their data value.  For the case
I'm talking about, there is no arithmetic going on.

I'm also not asking that the current fields in the proposal change format.

I'm discussing different packet formats for different purposes.

> delay, as Miroslav has already pointed out.

You seem to have missed that I covered this already.

Assuming the path stays the same the first exchange requests that each
device along the path increments a counter, and the response packet does
the same.  If the path counts match (maybe with some other information),
we're comfortable that the path was the same.

On subsequent packets, the 'request' EF contains the N path count
values, and the 'response' EF contains the information with the corrections.

If we want to allow different data types/sizes per path, then:

The initial request contains 3 (for example) counters, for each of the 3
(in this example) data types.

The subsequent packets contain appropriate numbers of the requested data
types.

This way the packet length doesn't vary, and we're not burning more bits
than we need to in order to convey the data.

---

From: Martin Burnicki <[email-masked]>
Date: Thu, 29 Mar 2018 12:35:55 +0200

Harlan,

> I'm discussing different packet formats for different purposes.

I'm just thinking about what needs to be done e.g. in a switch that
should measure the packet delay, and update the packet on the fly while
it passes through, as this is done with PTP transparent clocks.

In a different email it was mentioned to timestamp an NTP packet that
comes in and goes out, queued or not, eventually at different link speeds.

As far as I know from PTP there has to be a packet matcher which
identifies the patterns of ingoing or outgoing NTP packets, which is
only possible after a number of bytes sufficient to identify a packet
have already come in.

After the pattern has been recognized, a timestamp has to be taken, and
if you want really high accuracy for the compensation you need to know
exactly at which point of the packet this happens, at which link speed, etc.

The same has to happen when the NTP packet goes out on another port. The
packet handler would have to determine the propagation delay, check if
the packet contains a correction field, at which byte offset in the
packet the correction field is located, and update the correction field
*after* the packet started to go out, put *before* also the packet's
correction field has gone out on the wire.

For PTP the correction field is at a well defined byte offset in the PTP
packet, and experiences from my colleagues on PTP plugfests have shown
that even this straight forward approach is often initially not
implemented by manufacturers of hardware that does packet timestamping.

Since the NTP packet format is variable due to the different EFs that
can be appended, it's quite a bit harder to implement this for NTP, even
if there is only a single correction field.

I think it will be *much* harder and *much* more error prone

- if you use both old and new extension fields in a single NTP
  packet, so it's pretty to examine the structure of an extended
  packet properly and unambiguously

- if you add a new correction field for each hop, making sure the
  maximum packet size isn't exceeded, in which case the packet
  handler would not only have to identify a correction field,
  but find the end of a chain of correction fields after which
  a new one can be appended

- if you allow for correction fields in different formats

I think we all can be happy if there are at all manufacturers of
switches or NICs who are willing to implement this, since PTP is already
available, so we should try to keep this as simple as possible.

Martin
-- 
Martin Burnicki

Senior Software Engineer

---

From: Harlan Stenn <[email-masked]>
Date: Sat, 31 Mar 2018 00:00:32 -0700

Hi Martin,

> it passes through, as this is done with PTP transparent clocks.

In this situation it might be better to have an entry that is "cycle
count" and "time per cycle", so the switch doesn't have to do anything
except report the number of cycles the packet was in the switch.

I'm just trying to point out that using "large" time structures seems
wasteful, and that only having a single time structure that gets updated
at each hop seems limited.

> exactly at which point of the packet this happens, at which link speed, etc.

Sure, and one could track (directly or indirectly) how much time had
passed between the BOPacket and the packet was recognized, or one could
grab the BOPacket time indicator and just ignore it if the processing
won't be done.  Switch vendors have options here.

And folks may well want to do this on firewall appliances as well, or on
some computer they are using as a router.

>   packet properly and unambiguously

By "old" and "new" EFs, you mean Autokey v. new stuff?

This proposal CANNOT be used with legacy MACs, so autokey (old EF) +
legacy MAC is out of the picture.

This proposal CAN be used with EFs that use a MAC-EF in some form, as
the correction field(s) would go at the EOPacket.

>   a new one can be appended

Yes, and that should be much easier to do if the packet uses the LAST-EF
in it.  I think it's pretty easy to detect this -
draft-stenn-ntp-extension-fields shows how to do this easily, and for
the fallback case, 7822 is still in-play.

> - if you allow for correction fields in different formats

I don't buy this one.  The EF header contains enough information to let
folks know which one they want to use.

This will work just fine with Miroslav's original proposal.  I submit it
will work just as easily with the alternatives I've described.

> available, so we should try to keep this as simple as possible.

And we're talking about an experimental proposal that can be used to
explore different options to make sure we come up with something that works.

The PTP Switch market (industrial) isn't quite the same as the general
case switch market, which includes a very wide range of devices.

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Sat, 31 Mar 2018 01:25:48 -0700

> On Mar 31, 2018, at 00:00, Harlan Stenn <[email-masked]> wrote:
> 

This seems like a weird thing to try making “easier” for the network gear (versus making the packet format simpler).

> I'm just trying to point out that using "large" time structures seems wasteful, and that only having a single time structure that gets updated at each hop seems limited.

I understand the generic appeal of “more data is better”, but apart from possible debug tool possibilities, what implementation benefits do you get from having a distinct data point from each hop?

To me it seems that either you trust the hops along the way enough to appropriately mess with the packet or you don’t. Keeping it simpler (and more like what’s in PTP) makes it more likely to work right.

Ask

-- 
http://askask.com/

---

From: Harlan Stenn <[email-masked]>
Date: Sat, 31 Mar 2018 02:00:24 -0700

> To me it seems that either you trust the hops along the way enough to appropriately mess with the packet or you don’t. Keeping it simpler (and more like what’s in PTP) makes it more likely to work…

It goes to identifying where the delays are.

If you don't care about where they are, use Miroslav's original packet
format.

If you want to know where they are, use one of the other formats.

If there is no value in the per-hop data, there are several options:

- don't allow it in the protocol
- allow it, and each device can decide for itself what it wants to do.
  Remember, it's the system that sends the original packet that says
  what data formats it's willing to accept.

If multiple formats are allowed, one of my ideas was to send the initial
packet asking each device along the way to increment the counter of what
information type it wanted to use.

If multiple formats are allowed and Miroslav's original format is sent,
that's the only thing that will be used for that measurement.

I'm just looking to provide flexible mechanism here, for what is an
EXPERIMENTAL protocol, allowing local policy choices.  I want to make it
easy and useful for folks to *experiment* with this and decide what is
useful.

That's all.