Subject: Re: [Ntp] WGLC: draft-ietf-ntp-interleaved-modes

From: Denis Reilly <[email-masked]>
Date: Thu, 6 Dec 2018 16:13:22 +0000

Hello!

I have read the NTP Interleaved Modes draft and believe it is ready for submission. I have no comments.

--
Denis Reilly  |  Technical Lead  |  [email-masked]  (585)321-5837

-----Original Message-----
From: ntp <[email-masked]> On Behalf Of Karen O'Donoghue
Sent: Friday, November 30, 2018 3:38 PM
To: [email-masked]
Subject: Re: [Ntp] WGLC: draft-ietf-ntp-interleaved-modes

Folks,

After receiving a few requests, I am going to extend this WGLC for one week. Please submit comments by COB on Friday 7 December. Even if you don’t have any comments, please indicate to the mailing list that you have read the draft and you believe it is ready for submission to the IESG. This is critical to helping the chairs determine working group consensus on this document.

Regards,
Karen

> On Nov 6, 2018, at 3:49 PM, Karen O'Donoghue <[email-masked]> wrote:
>
> Folks,
>
> This message initiates a three plus week working group last call for:
>
> NTP Interleaved Modes
> https://datatracker.ietf.org/doc/draft-ietf-ntp-interleaved-modes/
>
> Please review the referenced document and send any comments to the mailing list including your assessment of whether this document is mature enough to proceed to the IESG. Please note that these messages of support for progression to the mailing list will be used to determine WG consensus to proceed.
>
> Please send all comments in by COB on Friday 30 November. We realize this is a bit longer than normal but we are coming out of an IETF week and heading into the Thanksgiving holiday in the US.
>
> Thanks!
> Karen and Dieter
>

_______________________________________________
ntp mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/ntp
ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

---

From: Harlan Stenn <[email-masked]>
Date: Fri, 7 Dec 2018 13:36:45 -0800

While I am supportive of the goals of this document, I am opposed to its
acceptance as a standards-track document with the document's current
language.

The following are the items I can find immediately.

1: Requiring an interleaved client/server mode places an unacceptable
burden on servers that have a lot of client traffic.  Specifically,
requiring a server to save date is too often Bad/onerous.

2: I would like to see demonstrated testing and test results that show
that for interleaved client/server mode, the specification behaves as
expected in both hostile and "dirty" environments, and does a thorough
job of demonstrating that there are no unintended consequences.  Has the
solid engineering been don?

On the first point I see two ways to advance this document "now".

One is to remove the language around the interleaved client/server mode,
and continue this work in a new document.  Not my favorite idea.  But as
I re-read my points below, it seems much safer at this time.

The other is to change the following and then re-evaluate the results:

The 4th paragraph of 2. begins with:

  A server which supports the interleaved mode needs to save
  pairs of local receive and transmit timestamps.

I recommend the following instead:

  A server which supports interleaved mode for the client needs to save
  pairs of local receive and transmit timestamps.

This is because a server may decide to support interleave mode for a
subset of clients.

The 4th paragraph of 2. ends with:

  The server MAY separate the timestamps by IP addresses, but
  it SHOULD NOT separate them by port numbers, i.e.  clients
  are allowed to change their source port between requests.

Has this been thoroughly tested?  I am specifically concerned about
multiple clients behind a NAT device, where the server will (almost?)
certainly see different port numbers for each client.  How well does
this work when there are multiple interleaved clients using different
ports from the same IP?  Is it sufficient that the server would compare
the incoming origin(?) timestamp to identify which (interleave response)
transmit timestamp should be sent back?  Given that UDP packets may be
lost in transit, how does the protocol behave in this case of a lost
client request?  How long should the server keep an interleaved transmit
timestamp, especially if we are not using the port number to help
identify responses?  This also goes to the case where an interleaved
client "stops" - how long should the server retain the information about
the last transmit timestamp for an "association" that is now gone?
Should the client/server interleaved mode store the client's poll
interval and use that information for this decision?

Paragraph 6 of 2. begins with:

  When the server receives a request from a client, it SHOULD respond
  in the interleaved mode if the following conditions are met:

I think this is too strong, and doesn't clearly allow for the situation
where a server may only offer interleaved client/server mode to a subset
of clients.  I wonder if:

  When the server receives a request from a client and is willing to
  offer an interleaved client/server association, it SHOULD respond
  in the interleaved mode if the following conditions are met:

is sufficiently "better".

To be clear, I'd like this proposal to show exactly how it behaves in
the face of:

- dropped client requests
- multiple clients using different ports from the same IP
- a client association that "just stops"

"Behavior" includes the behavior (including the amount and longevity of
saved state) of the server.

In the case of a NAT proxy in between the clients and the server, it is
possible that two, and possibly more, clients will send packets with the
same transmit timestamp.  As the number of clients behind the NAT proxy
increases, the possibility of identical transmit timestamps also increases.

I am also very concerned about this paragraph in 2.:

  The client SHOULD NOT update its NTP state when an invalid response
  is received to not lose the timestamps which will be needed to
  complete a measurement when the following response in the interleaved
  mode is received.

There are intricate and critical reasons why NTP updates various aspects
of its state depending on how far along packet validation and processing
occurs.  I would want to see a detailed and specific analysis of the
complete effects of the proposed sentence on the protocol machinery.
The primary goal is that the protocol be robust.  The secondary goal is
that interleave mode works.  I am concerned that the identified sentence
above switches these priorities.

As a general question, exactly how much benefit is seen when using
client/server interleaved mode?

As another general "question", how does this proposal operate in the
face of data-minimization?  Should this be stated in the document?

Aside from the above, I'm still recovering from the cold I've had for
over 2 weeks' time and I need to spend more time to study some of the
intricacies of the interleaved client/server mode in particular, and
this proposal in general.
-- 
Harlan Stenn <[email-masked]>
http://networktimefoundation.org - be a member!

---

From: Miroslav Lichvar <[email-masked]>
Date: Mon, 10 Dec 2018 17:05:48 +0100

> acceptance as a standards-track document with the document's current

Thanks for the comments.

> requiring a server to save date is too often Bad/onerous.

The document says "The server SHOULD discard old timestamps to limit
the amount of memory needed to support clients using the interleaved
mode." and "The server MAY always respond in the basic mode."

It can use as much or as little memory for interleaved clients as is
available or configured. It can drop all state at any time. It can
enable the support only for some specific addresses. That's perfectly
fine.

The main use case is synchronization in local networks. There is
nothing that prevents the use of the interleaved mode over Internet,
and there are public servers that support it, but it doesn't make much
of a difference in terms of accuracy, because the jitter and asymmetry
over Internet is typically orders of magnitude larger than the error
in transmit timestamp in the basic mode.

> solid engineering been don?

Yes. An implementation of the protocol has been thoroughly tested with
replays, packet drops, and other MITM attacks. It's also used in
production.

Do you have some specific tests in mind?

There is a minimal client and server implementation in python. Please
feel free to test it and report if anything breaks.

https://github.com/mlichvar/draft-ntp-interleaved-modes

>   pairs of local receive and transmit timestamps.

Ok. If you feel this is not clear from the document, we can say it
explicitly. Is there a reason you dropped the "the"? There seem to be
a lot of rules about using definite/indefinite articles in English.
I'm never sure.

> transmit timestamp should be sent back?

The receive and transmit timestamps in server's responses, which are
copied to the origin timestamp in client requests, are fully under the
control of the server. A server with a sane clock will not respond
with a duplicate receive timestamp, so each client using interleaved
mode sends a unique origin timestamp.

if the clock is not sane, there may be a duplication and a client
could get a transmit timestamp of a response that was sent to a
different client. Both transmit timestamps correspond to the same
receive timestamp, so the error should be small, but it depends on
what exactly happened to the server's clock that it produced the same
timestamp twice. If it was a backward step, it doesn't really matter
if the response is in basic mode or interleaved mode. All clients
polling the server around that time would be affected.

> Given that UDP packets may be
> client request?

If the client request is lost, nothing surprising will happen. The
next request will use the same origin timestamp and it will have an
interleaved response if the server still has the timestamps. If the
response is lost, the next response may be in basic mode if the server
is keeping only one pair of timestamps per client.

> How long should the server keep an interleaved transmit

That's up to the implementation. It doesn't have to be a specific
timeout. It doesn't have to track individual associations. The
simplest implementation would probably be a single queue with constant
length for all clients, which would drop the oldest timestamps as new
timestamps are made. If there is little traffic, the server may keep
timestamps in memory for a long time.

> interval and use that information for this decision?

An advanced implementation could do that, but I'm not sure if it's
worth the trouble.

> is sufficiently "better".

Yes, I think that makes it more clear.

> increases, the possibility of identical transmit timestamps also increases.

That's not a problem. The server doesn't really care what transmit
timestamp is in the request. The only thing it does with it is to
check if it's not equal to the receive timestamp from the same
request, and use it as the origin timestamp when responding in the
basic mode.

> above switches these priorities.

Not at all. The sentence is there to prevent off-path DoS attacks on
the client. A client (unlike a peer) can safely ignore all packets
that don't have a valid origin timestamp.

> client/server interleaved mode?

It depends on how accurate are the timestamps, how jittery/symmetric
is the network, and what clock is actually synchronized (system clock
or HW clock). In a local network with a small number of switches, the
accuracy of a system clock may improve from about ten microseconds to
about a hundred of nanoseconds, so about two orders of magnitude.

> face of data-minimization?  Should this be stated in the document?

Good point. It probably should be stated there. The origin timestamp
is required for the interleaved mode to work, so it must not be set to
zero.

Thanks,

-- 
Miroslav Lichvar

---

From: Harlan Stenn <[email-masked]>
Date: Mon, 10 Dec 2018 17:47:27 -0800

> mode." and "The server MAY always respond in the basic mode."

I'm not talking about old timestamps.

I'm talking about a case where there are LOTS of current clients.

The client/server portion of this proposal changes the daemon from
stateless to stateful.

> enable the support only for some specific addresses. That's perfectly

Sure, and that's why I offered my suggested language change below.

>> solid engineering been done?
> 

Not yet.

> https://github.com/mlichvar/draft-ntp-interleaved-modes

You have this in chronyd then?  Do you have any results on how much of a
difference it makes?

If you're testing this with chrony I would imagine the results would be
slightly different from ntpd, as I don't believe chrony implements the
same internal algorithms and I don't know how significant that would be
to the results.

> a lot of rules about using definite/indefinite articles in English.

To me, my suggested language is a tighter definition and should make it
easier for implementors (how many will there be?) to do the right thing,
and focus on the relevant issues.

I hadn't noticed I dropped the "the" :)

In this case I would ordinarily not add it, but then again I'm a fan of
the Oxford comma, and we always talk about "Network Time Foundation",
"ISC", etc. instead of "the Network Time Foundation" or "the ISC".

My British friends talk about "going to hospital" while in the US (at
least) we talk about "going to the hospital".

English.  What a language.

> polling the server around that time would be affected.

I'm not talking about server responses, I'm talking about the server
properly matching stored timestamps when there are an increasing number
of clients behind a single IP.

> is keeping only one pair of timestamps per client.

I'm not sure how "client" is being defined here, given the discussions
about not paying attention to port numbers.

I also need to look at this again to understand your point about "the
next request will use the same origin timestamp ...".

This brings up the question of what happens if a server response is lost
- in that case the server will have sent its interleaved reply and will
switch the client's saved transmit/origin timestamp.  When the client
doesn't get the response, it will then send the next packet with the
previous origin timestamp, the one the server no longer has because it
already replied using that one?

> timestamps in memory for a long time.

Yes, and one way to manage this is with a capped queue length, and
another way to manage it is with client limits (if there is a reasonable
way to manage this).  My point goes more towards the belief that there
is benefit to removing old/expired timestamps from the queue sooner
rather than later.  So it may make sense to keep a timestamp for, say, 2
seconds past 2x(poll interval), as we don't know if the client might
decide to increase its poll interval based on whatever it learned from
our most recent response.

> An advanced implementation could do that, but I'm not sure if it's

I'm not certain of the costs/risks of keeping old timestamps in the queue.

> request, and use it as the origin timestamp when responding in the

I will have to study this more.

---

From: Miroslav Lichvar <[email-masked]>
Date: Tue, 11 Dec 2018 13:48:02 +0100

> I'm talking about a case where there are LOTS of current clients.

Maybe I'm not using the word "old" correctly? If there is a large
number of clients trying to use the interleaved mode, the server needs
to keep a large number of timestamps to be able to respond in the
interleaved mode to all of them.

> The client/server portion of this proposal changes the daemon from

Yes. A state is needed to save the transmit timestamp. There is no way
around that. If it was sent immediately in a separate message, it
would amplify the traffic, or the length of the response wouldn't be
symmetric to the request.

It's a bit like a TCP-based service. If there is a large number
of clients, or there are DoS attacks, there is no guarantee that a
client will get an interleaved response.

An important difference to TCP is that the protocol gracefully falls
back to the basic mode. Clients can still synchronize to the server.
They must not expect the server to be able to respond in the
interleaved response.

> You have this in chronyd then? 

No, in chrony it's implemented differently. The server support was
basically bolted on the monitoring facility. That was the easiest and
safest way. In ntpd it might be the same.

There is only one timestamp per IP address, which means it doesn't
work over NAT (as documented in the man page). Some of the code is
shared with the symmetric mode and there are optimizations, so it may
be difficult to see what is actually going on in the client/server
interleaved mode.

For implementors interested in the client/server interleaved mode, I'd
rather recommend to first look at the python example.

> Do you have any results on how much of a difference it makes?

It corresponds to the difference in accuracy and symmetry of the
timestamps that the server has. In absolute terms it's usually few
microseconds or tens of microseconds. In relative terms, it can be up
to about two orders of magnitude.

Basically, it makes the accuracy of NTP comparable to PTP. In networks
where switches/routers support PTP, it's worse as the devices
typically don't have a HW support for NTP. In networks without PTP
support it's better than PTP, because PTP is very sensitive to
variability in network delay. Of course, it also depends on the
implementation.

If you just want to see an example of a difference in ideal
conditions, here are two sourcestats comparing the same server to a
PTP clock. In the first one it's using the basic mode, in the second
it's using the interleaved mode.

Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
PTP                         8   5     7     -0.000      0.003     -0ns     3ns
192.168.33.1               21  13    22     +0.004      0.031  +3175ns   240ns

Name/IP Address            NP  NR  Span  Frequency  Freq Skew  Offset  Std Dev
==============================================================================
PTP                        13   7    12     -0.000      0.001     -0ns     4ns
192.168.33.1               22  10    21     +0.001      0.001    +18ns     9ns

So, in this case the stability of NTP measurements improved by a
factor of about 20 and the accuracy relative to the PTP clock by a
factor of about 150.

> same internal algorithms and I don't know how significant that would be

That shouldn't matter much if you just want to see a difference. The
problem is that ntpd doesn't support hardware timestamping or kernel
transmit timestamping. The interleaved mode makes a difference only if
the server has a more accurate transmit timestamp after the
transmission.

> of clients behind a single IP.

Well, unique receive timestamps in server responses is what enables
the server to properly match them with stored transmit timestamps. It
doesn't matter how many clients are there behind a single IP. Each
should get a unique timestamp from the server.

> already replied using that one?

It depends on the implementation. It may drop the timestamp to which
it already responded to, but it doesn't have to. The python example
would keep it, chronyd would not.

> our most recent response.

It might work, but it will need more memory per client. And that
memory could be used for more timestamps and keep it simpler. One pair
of timestamps is 16 bytes, or just 12 bytes in an optimized
implementation. So, even a little bit of extra state can decrease the
number of concurrent clients significantly.

In any case, I think we would need to specify a requirement for
interleaved clients to not zero the "poll" field.

-- 
Miroslav Lichvar

---

From: Harlan Stenn <[email-masked]>
Date: Fri, 14 Dec 2018 13:03:29 -0800

Miroslav,

I'm happy to work with you on this, if you're interested and think it
would be useful.

I note:

- 18ns corresponds to a precision of about -26.  While that's pretty
special, I remain curious about how long the delay is between pulling
the transmit stamp and the packet going out over the wire.  If the basic
precision of the system clock is only -19, we're not going to do better
than 2 microseconds.  For situations like this, I'm really curious if
we'll even see any benefit to client/server interleave mode.

- I'd like to explore a "follow up" message as an alternative.

- I'm wondering if we can do all of this with "plain" packets, or if
there is benefit to using an extension field for some of these exchanges.

H

---

From: Daniel Franke <[email-masked]>
Date: Wed, 19 Dec 2018 12:40:26 -0500

Sorry for not getting this in during the WGLC period. I strongly
object to this document but nonetheless support advancing it because I
have already raised all these objections at several prior meetings and
it's clear that I'm in the rough on WG consensus. I will communicate
them again during the IETF Last Call period and hope that the IESG
finds them persuasive.

The core of my objection is that this proposal radically and
needlessly alters the semantics of existing timestamp fields and then
relies on a very brittle mechanism to disambiguate which semantics are
intended. Servers which implement this proposal will reply with
interleaved semantics if the origin timestamp in the client's request
is non-zero and matches the receive timestamp of a reply the server
previously sent within some scope, where the bounds of that scope (IP,
port, time window) are largely left to implementers' discretion.
Existing RFCs do not adequately ensure that existing implementations
that are not aware of interleaved mode will never send a request of
this form. If a client fully implements RFC 5905 and is there is no
other request in the same scope, then it's okay: it will instead set
its request's origin timestamp to the server's previous reply's
transmit timestamp, and this proposal correctly requires that the
receive and transmit timestamps in the reply not be equal. However,
this breaks down as soon as there are multiple clients in the same
scope (e.g., both behind the same NAT), especially if some of them are
non-conforming. At minimum, servers must then ensure that they never
send a transmit timestamp that matches *any* receive timestamp sent to
the same scope. Even then, non-conforming clients can break conforming
clients within their scope by sending colliding timestamps.

These failures may prove rare in practice, but there is no excuse for
taking the risk given that there is a much cleaner alternative: do not
alter any existing semantics, and instead put interleaved timestamps
into an extension field. Even if all known failure modes can be
satisfactorily resolved without doing this, it's the right thing as a
matter of basic principle and the WG's resistance to taking this
approach baffles me.

Finally, one unrelated nit: [I-D.ietf-ntp-data-minimization] is listed
as an informative reference, but it's referenced in sentences that use
RFC 2119 language, so it needs to be normative.