Subject: Re: [Ntp] Benjamin Kaduk's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS and COMMENT)

From: Denis Reilly <[email-masked]>
Date: Thu, 17 Jan 2019 18:14:07 +0000

Hello, Benjamin! Thanks for the review. I've posted a new version of the draft which incorporate the feedback from the review.

See responses in-line:

--
Denis Reilly  |  Technical Lead  |  [email-masked]  (585)321-5837

-----Original Message-----
From: Benjamin Kaduk <[email-masked]> 
Sent: Wednesday, December 19, 2018 8:54 PM
To: The IESG <[email-masked]>
Cc: [email-masked]; Karen O'Donoghue <[email-masked]>; [email-masked]; [email-masked]; [email-masked]
Subject: Benjamin Kaduk's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS and COMMENT)

Benjamin Kaduk has entered the following ballot position for
draft-ietf-ntp-bcp-10: Discuss

When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.)

Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
for more information about IESG DISCUSS and COMMENT positions.

The document, along with other ballot positions, can be found here:
https://datatracker.ietf.org/doc/draft-ietf-ntp-bcp/

----------------------------------------------------------------------
DISCUSS:
----------------------------------------------------------------------

I see that Ben has already asked about the SHOULDs (vs. MUSTS) for secure key exchange and prevention from disclosure, in Section 4.1, but I will make that a Discuss point.  If these are to remain SHOULD, we should say something about in what case(s) MUST would not be appropriate.

I'm also concerned that there is too much intermingling of general BCP-worthy advice with implementation-specific knowledge for publication as BCP in the current form.  I've tried to note instances of this in the comment section, but for example this includes talking about the "key file"
and the format of the configuration file.  In a similar vein, it's unclear that the guidance in Appendix A will age well, at least without a more explicit disclaimer (including disclaimer of normativity) -- e.g,. are the
-4 and -6 modifiers to restrict still needed or best practice?  IIRC a recent update on my FreeBSD machine updated ntp.conf to just use basic restrict stanzas without an IP version.

I'm also surprised to see no discussion of the (non-)applicability of IPsec for NTP traffic, when authenticity or access control is required.  (E.g., where IP acls are discussed in Section 5.1)

Response:
Regarding the key exchange: 
Our original motivation for keeping these as "SHOULD" was that there was no formal recommendation we could point users to on how to secure these keys,. (Alissa Cooper had already brought this up as a Comment.) But we have received enough feedback from the reviewers here that we will change these to "MUST"s.

Regarding the Key File:
We will refer to the key file as "local key storage" in the body of the document, to make it a bit more generic. We keep the reference to the key file in the ntpd-specific appendix.

Regarding the Appendix:
I admit that we don't quite understand what sort of disclaimer you would like for the appendix. Aren't the appendixes already non-normative?

Regarding IPSec: 
We will add an additional sub-section for "External Security Protocols":

   If applicable, external security Protocols such as IPsec and MACsec
   can be applied to enhance integrity and authenticity protection of
   NTP time synchronization packets.  Usage of such external security
   protocols can decrease time synchronization performance [RFC7384].
   Therefore, operators are advised to carefully evaluate if the
   decreased time synchronization performance meets their specific
   timing requirements.

----------------------------------------------------------------------
COMMENT:
----------------------------------------------------------------------

In general the writing could be tightened up some more, especially to remove duplication and improve transitions.  I've noted several instances in the comments (mostly tagged with "nit"), as well as some more substantive comments.

Section 2.1

                   UDP-based protocols such as NTP are generally more
   susceptible to spoofing attacks then other connection-oriented
   protocols.  [...]

nit: was this intended to be "other, connection-oriented, protocols"?

Response:
"UDP-based protocols such as NTP are generally more
        susceptible to spoofing attacks than connection-oriented
        protocols."

--

               NTP control messages can generate a lot of data in
   response to a small query, which makes it more attractive as a vector
   for distributed denial-of-service attacks.  [...]

nit: more attractive than what?  (I.e., maybe just "makes it attractive")

Response: We agree

--

   BCP 38 [RFC2827] was approved in 2000 to address this.  [...]

nit: maybe, "BCP 38 was published in 2000 to provide some level of remediation against address-spoofing attacks"?

Response: We agree

--

                                               It is RECOMMENDED that
   large corporate networks (and ISP's of any size) implement ingress
   and egress filtering.  More information is available at the BCP38
   Info Web page [BCP38INFO] .

BCP 38 already makes this recommendation, and the current document is supposedly scoped to just NTP, so I would have expected wording more like "It is recommended that [...] that use NTP implement ingress and egress filtering.", if we can even be clear about who this directive is supposed to apply to.

Response:
Personally, I (Denis) like the broader recommendation here, as it seems like a good idea to emphasize the importance of the BCP. But we can move the sentence "Mitigating source address spoofing attacks should be a priority of anyone administering NTP." to appear at the start of the next paragraph. This will better clarify why the recommendation is there.

--

Section 3.1

   Many network security mechanisms rely on time as part of their
   operation.  If attackers can spoof the time, they may be able to
   bypass or neutralize other security elements.  For example, incorrect
   time can disrupt the ability to reconcile logfile entries on the
   affected system with events on other systems.  An application which
   is secure today could be insecure tomorrow once an unknown bug (or a
   known behavior) is exploited in the right way.  Even our definition
   of what is secure has evolved over the years, so code which was
   considered secure when it was written may turn out to be insecure
   after some time.

The first three sentences seem related, but the last two sentences seem to be talking about something qualitatively different (namely, "more vulnerabilities being discovered over time", compared to the original's "accurate time is important for secure and correct operation").  I would suggest a paragraph break and some transitional language.

Response:
We've removed those last two sentences in the latest draft, because we now feel the first three stand on their own.

--

Section 3.2

   But even with 4 or more sources of time, systemic problems can
   happen.  For several hours before and after the June 2015 leap
   second, several operators implemented leap smearing while others did
   not, and many NTP end nodes could not determine an accurate time
   source because 2 of their 4 sources of time gave them consistent UTC/
   POSIX time, while the other 2 gave them consistent leap-smeared time.
   See Section 3.7.1 for more information.

   Operators SHOULD monitor all of the time sources that are in use.  If
   time sources do not generally agree, find out the cause and either
   correct the problems or stop using defective servers.  See
   Section 3.5 for more information.

nit: the transition here is a bit odd.  I would suggest either introducing leap second smearing as a separate concept first (e.g., by forward-reference to Section 3.7), or making the second quoted paragraph mention that leap second smearing is one of many potential causes for disagreement amongst time sources.

Response:
We rewrote that first paragraph to make it flow a bit better:
    But even with 4 or more sources of time, systemic problems can
	happen. One example involves the leap smearing concept detailed in 
	Section 3.7. For several hours before and after the 
	June 2015 leap second, several operators configured their NTP servers 
	with leap smearing while others did not. Many NTP end
	nodes could not determine an accurate time source because 2 of their
	4 sources of time gave them consistent UTC/POSIX time, while the
	other 2 gave them consistent leap-smeared time. This is just one of 
	many potential causes of disagreement among time sources.

--

Section 3.3

nit: the Q&A style in the second paragraph is not something I usually expect to read in a BCP.

Response:
Maybe it's just a reflection of our writing style, but these do reflect
questions that we think operators should ask when determining whether their 
implementations are diverse enough.

--

Section 3.4

                             Used improperly, these facilities can be an
   abuse vector.  [...]

I think (but am not 100% sure) that it's an attack vector on the server itself, as well as an abuse vector.

Response:
We've changed this to "But these facilities can be a vector for amplification attacks when abused."

--

Section 3.4

The BCP 38 recommendation was already made above; do we really need to duplicate it here?

Response:
I (Denis) think it's important to link the two, as this shows additional rationale for implementing BCP38. But we do mention NTP Control messages in the BCP38 section; maybe it's better to just reference that section so we don't repeat the recommendation.

--

Section 3.5

   If a system starts getting unexpected time replies from its time
   servers, that can be an indication that the IP address of the system
   is being forged in requests to its time server.  The goal of this
   attack is to convince the time server to stop serving time to the
   system whose address is being forged.

nit: the writing here could probably be tightened up.  E.g., things like "NTP reply packets that do not correspond requests it sent", "an attacker is forging its IP address in requests to the time server", and "one reason an attacker would do so could be to convince the time server to".

Response:
Changed to
I
f a system starts to recieve NTP Reply packets from a time server
that do not correspond to any requests sent by the system, that can be
an indication that an attacker is forging that system's IP address in
requests to the remote time server. The goal of this attack would be to
convince the time server to stop serving time to the
system whose address is being forged.

--

   If a server's system log shows messages that indicates it is
   receiving timestamps that are earlier than the current system time,
   then either the system clock is unusually fast or somebody is trying
   to launch a replay attack against that server.

Is "receiving timestamps" supposed to be for NTP messages in particular, or all general syslog traffic?

Response:
We are referring to NTP timestamps, and will clarify this in the next draft.

--

Section 4.1

   [RFC5905] specifies a hash which must be supported for calculation of
   the MAC, but other algorithms may be supported as well.  The MD5 hash
   is now considered to be too weak.  [...]

nit: "too weak" for what?  (Maybe "considered to be weak and unsuitable for cryptographic usage" would be better, with a reference to RFC 6151 or similar.

Response:
We are adding:
The MD5 hash is now considered to be too weak and unsuitable for cryptographic
usage.  [RFC6151] has more information on the algorithm's weaknesses.

--

   To use this approach the communication partners have to exchange the
   key, which consists of a keyid with a value between 1 and 65534,
   inclusive, and a label which indicates the chosen digest algorithm.

Surely there is also the actual cryptographic key material itself!

   Each communication partner adds this information to its own key file.

Does the reader know what a "key file" is at this point in the document?
(Alternately, is "key file" an implementation detail and not a protocol
concept?)

Response:
We've replaced "key file" with "local key storage" in the body of the document, and put more specific information about where to insert the key into the ntpd-specific appendix (where it is more appropriate).

--

   Some implementations store the key in clear text.  Therefore it
   SHOULD only be readable by the NTP process.  Different keys are added
   line by line to the key file.

Similarly here; the "key file" is only vaguely and implicitly described (and the line-by-line format is clearly implementation-specific); the main actionable point here is just to ensure that it is only readable to the NTP process and the rest could, I think, be safely omitted.

Response:
We'll omit the last sentence.

--

   An NTP client establishes a protected association by appending the
   key to the server statement in its configuration file.  Note that the
   NTP process has to trust the applied key.

If the configuration file format is not standardized, there's not much useful for us to say here about its contents.  Also, what does "has to trust" mean?

Response:
We can rewrite to be clearer:

An NTP client has to be able to link a key to a particular server
in order to establish a protected association. This linkage is
implementation specific. Once applied, a key will be trusted until
the link is removed.

--

Section 4.2

The reference is provided only for the attack on autokey but not for autokey itself.  Is there a stable reference for the autokey protocol (so that people know what to not use)?

Response:
We've added RFC5906 as an informative reference.

--

Section 5.1

                                                         NTP control
   queries also leak important information (including reference ID,
   expected origin timestamp, etc.) that may be used in attacks
   [CVE-2015-8139].  A remote attacker can learn this information by
   sending control queries to a target system and inspecting the
   response.

Er, so is it the control *query* that leaks information, or the response to that query?

Response:
We can clarify this by changing to "and inspecting the leaked information in the response."

--

           It is recommended that operators SHOULD filter mode 3 queries
   at the edge, or make sure mode 3 queries are allowed only from
   trusted systems or networks.

nit: "at the edge" is not a well-defined concept here.

Response:
Will change to "from outside their networks".

--

   Note well that proper monitoring of an NTP server instance includes
   checking the time of that NTP server instance.

Perhaps more explicitly state that the above recommendations for leaf hosts preclude such monitoring [of leaf hosts]?

Response:
Will change to "An exception to this can be made if a leaf-node host is being actively monitored, in which case incoming packets from the monitoring server can be allowed."

--

Section 5.3

Do we want to say anything about what to do when a (potential) attack is detected (e.g., make an entry in the system log)?

Response:
We don't think so, because the monitoring in this section may not be done
by the NTP implementation itself. If an operator uses an external tool to performance this monitoring, we don't want to impose any requirements on that.

--

Section 5.4

It seems worth reiterating that these KoD packets will be accepted in common usage even when not cryptographically authenticated, which makes the DoS risk more severe.

I am not sure whether the note about KoD packets indicating potential attacks is better here or in the previous subsection.

Response:
We want to keep the note here, but we've added your suggested text:
   Kiss-o'-Death (KoD) packets can be used in denial of service
   attacks.  Thus, the observation of even just one KoD packet with a
   high poll value could be sign that the client is under attack. 
   And KoD packets are commonly accepted even when not cryptographically 
   authenticated, which increases the risk of denial of service attacks.

--

Section 6.1

This is entirely editorial (and thus your preferences outweigh mine), but if I were writing this I would say something like "an up-to-date and secure NTP implementation" rather than "the latest NTP updates applied".

Response: We Agree.

--

Section 7

Would it ever make sense to have multiple (disjoint?) anycast pools so that clients could still benefit from having multiple servers concurrently available to compare?

Response: A good point, we will add it.

--

Appendix A.*

It would be helpful to distinguish which strings are literal syntax that must be used unchanged and which strings are supposed to be user-replaceable.

Response: We've changed the text of the Appendix to make this more clearer, mainly in the Prre-Shared Key section.

--

ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

---

From: Benjamin Kaduk <[email-masked]>
Date: Thu, 17 Jan 2019 19:29:12 -0600

> Hello, Benjamin! Thanks for the review. I've posted a new version of the draft which incorporate the feedback from the review.

Thanks for the updates; they address nearly everything and possibly
actually everything I requested; see below.

> See responses in-line:

Appendices are not inherently non-normative; RFC 8446 is a handy example.
The case here is particularly unclear, since Section 3 explicitly says
"Best Practices that are specific to the Network Time Foundation
implementation are compiled in Appendix A", which is fairly easy to read as
including the appendix by reference.  My understanding (which could be
wrong!) is that a more accurate description would be something like
"Application of these best practices that are specific to the Network Time
Foundation implementation, including example configuration directives valid
at the time of this writing, are compiled in Appendix A."

I've cleared my Discuss in the datatracker, since it seems like we will
easily be able to adjust the text here to match the result of our
continuing discussion, if needed.

>    decreased time synchronization performance meets their specific

That is exactly what I was looking for; thank you!

Better than my suggested text; thanks.

It does help clarify, thanks.

That seems like a fine resolution to my point, though I see in the -11 that
the entire paragraph has been removed and not just the last two sentences.
Was this the intended change (e.g., based on other review comments)?

The forward reference to 3.7.1 helps a lot; thank you!

Looks good.

Ah, thank you.  (I was not entirely sure whether it would make sense or
not, so I appreciate you taking the time to think about it.)

Thanks again for all the updates; they add up to a much tighter document!

-Benjamin

---

From: Joachim Fabini <[email-masked]>
Date: Fri, 18 Jan 2019 08:38:06 +0100

Hello Dennis,

chiming in on the external security paragraph that you propose below for
IPsec. The current wording is correct but I'd expect a bcp to explicitly
warn in its security section that an IPsec tunnel can NOT protect from
timing attacks. Detailed comments inserted inline.

> Hello, Benjamin! Thanks for the review. I've posted a new version of the draft which incorporate the feedback from the review.

The proposed paragraph is correct and accurate but misses to mention an
essential detail. The intended audience of the bcp may misread the
statement as: "your ntp time synchronization is safe if you tunnel your
ntp packets through IPsec". In a bcp I'd be _very_ detailed and accurate
about such statements. I know about commercial deployments of time
synchronization in critical infrastructures that rely on IPsec tunneling
of timesync (over public IP links) without being aware of possible
consequences.

Fact is that timing attacks are possible and trivial, even if timesync
packets are tunneled using IPsec and substantial background traffic
exists to obfuscate the packets. So I propose to explicitly point out
this fact and warn about it, for instance:

"It is important to note that usage of such external security
protocols, besides decreasing time synchronization performance [RFC7384]
can NOT prevent timing attacks on the NTP protocol."

One could go even further to explicitly discourage the use of IPsec for
tunneling timesync packets whenever timing attacks on non-encrypted
packets could do harm. It's up to the authors to decide on such
statements, but I see the bcp ntp draft as the right place to warn on
this kind of security by obscurity.

We've shown the issue for PTP (pre-published version of the paper at
https://arxiv.org/abs/1811.08569 ) but identical reasoning applies for
NTP, too.

regards
Joachim

> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp
>

---

From: Dieter Sibold <[email-masked]>
Date: Mon, 21 Jan 2019 10:12:56 +0000

Dear Joachim,
thanks for your reference to packet delay attacks. Indeed we didn't consider this in the BCP. We can imagine to add short text to the document that indicates the issue related with these kind of attacks. However, since all of the cryptographic security measures compiled in Sec. 4 are vulnerable to this attack, we propose to add the following paragraph at the end of Sec. 4.

"Note that none of the security measures described in this section can prevent from packet delay attacks [cf. Sec. 3.2.6, RFC 7384] on NTP time synchronization messages."

- Dieter

Am 18.01.19, 08:38 schrieb "Joachim Fabini" <[email-masked]>:

    Hello Dennis,

    

    

    regards
    Joachim

---

From: Joachim Fabini <[email-masked]>
Date: Mon, 21 Jan 2019 12:04:48 +0100

Hi Dieter,

> "Note that none of the security measures described in this section can prevent from packet delay attacks [cf. Sec. 3.2.6, RFC 7384] on NTP time synchronization messages."

Yes, that's fine. I'd go one step further and state explicitly that the
problem affects NTP timesync packets in encrypted traffic (tunnels),
too. The referenced RFC section is imho not (sufficiently) explicit in
this regard.

So something equivalent to:
"Note that none of the security measures described in this section can
prevent from packet delay attacks [cf. Sec. 3.2.6, RFC 7384] on NTP.
Such delay attacks can target time synchronization packets sent as
clear-text or even within an encrypted (tunneled) traffic."

thanks
Joachim

---

From: Denis Reilly <[email-masked]>
Date: Fri, 25 Jan 2019 20:11:33 +0000

Hello, Benjamin. Thanks for your follow-up comments.

I have a few quick responses:

First, thanks for your explanation of your thoughts about the normativity of the appendix. We have incorporated your suggestions in the next draft.

You also caught that we removed a bit more in the section on "keeping NTP up to date" than we had indicated in our responses. While discussing the changes, we decided that the rest of that paragraph, while good advice, was already incorporated in other sections of the document and didn't really address the subject of "keeping NTP up to date", so we removed the whole paragraph. 

Finally, Joachim Fabini also responded to your comments: He noted that the new section for "external security protocols" ought to comment on delay attacks. So after some back and forth with him, we have added this to that new section:

   Note that none of the security measures described in Section 4 can
   prevent packet delay manipulation attacks on NTP.  Such delay attacks
   can target time synchronization packets sent as clear-text or even
   within an encrypted tunnel.  These attacks are described further in
   Section 3.2.6 of [RFC7384].

Best Regards,

-----Original Message-----
From: Benjamin Kaduk <[email-masked]> 
Sent: Thursday, January 17, 2019 8:29 PM
To: Denis Reilly <[email-masked]>
Cc: The IESG <[email-masked]>; [email-masked]; Karen O'Donoghue <[email-masked]>; [email-masked]; [email-masked]
Subject: Re: Benjamin Kaduk's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS and COMMENT)

> Hello, Benjamin! Thanks for the review. I've posted a new version of the draft which incorporate the feedback from the review.

Thanks for the updates; they address nearly everything and possibly actually everything I requested; see below.

> Cc: [email-masked]; Karen O'Donoghue 
> <[email-masked]>; [email-masked]; [email-masked]; 
> [email-masked]
> and the format of the configuration file.  In a similar vein, it's 
> unclear that the guidance in Appendix A will age well, at least 
> without a more explicit disclaimer (including disclaimer of 
> normativity) -- e.g,. are the
> I'm also surprised to see no discussion of the (non-)applicability of 
> IPsec for NTP traffic, when authenticity or access control is 
> required.  (E.g., where IP acls are discussed in Section 5.1)
>
> Response:

Appendices are not inherently non-normative; RFC 8446 is a handy example.
The case here is particularly unclear, since Section 3 explicitly says "Best Practices that are specific to the Network Time Foundation implementation are compiled in Appendix A", which is fairly easy to read as including the appendix by reference.  My understanding (which could be
wrong!) is that a more accurate description would be something like "Application of these best practices that are specific to the Network Time Foundation implementation, including example configuration directives valid at the time of this writing, are compiled in Appendix A."

I've cleared my Discuss in the datatracker, since it seems like we will easily be able to adjust the text here to match the result of our continuing discussion, if needed.

That seems like a fine resolution to my point, though I see in the -11 that the entire paragraph has been removed and not just the last two sentences.
Was this the intended change (e.g., based on other review comments)?

> Maybe it's just a reflection of our writing style, but these do 
> reflect questions that we think operators should ask when determining 
> whether their implementations are diverse enough.
>
> --
>
> Section 3.4
>
> f a system starts to recieve NTP Reply packets from a time server that 
> do not correspond to any requests sent by the system, that can be an 
> indication that an attacker is forging that system's IP address in 
> requests to the remote time server. The goal of this attack would be 
> to convince the time server to stop serving time to the system whose 
> address is being forged.
>
> --
>
> The MD5 hash is now considered to be too weak and unsuitable for 
> cryptographic usage.  [RFC6151] has more information on the algorithm's weaknesses.
>
> --
>
>
> An NTP client has to be able to link a key to a particular server in 
> order to establish a protected association. This linkage is 
> implementation specific. Once applied, a key will be trusted until the 
> link is removed.

Looks good.

> We don't think so, because the monitoring in this section may not be 
> done by the NTP implementation itself. If an operator uses an external tool to performance this monitoring, we don't want to impose any requirements on that.
>
> --
>
> Section 5.4
>

Ah, thank you.  (I was not entirely sure whether it would make sense or not, so I appreciate you taking the time to think about it.)

-Benjamin

---

From: Joachim Fabini <[email-masked]>
Date: Wed, 30 Jan 2019 08:29:32 +0100

Hello Dennis,

thank you for including the explicit warning on delay manipulation - the
wording is fine.

best regards
Joachim