Subject: Re: [Ntp] Eric Rescorla's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS and COMMENT)

From: Eric Rescorla <[email-masked]>
Date: Fri, 22 Mar 2019 09:29:09 -0700

On Thu, Jan 17, 2019 at 10:14 AM Denis Reilly <[email-masked]>
wrote:

> Hello, Eric! Thanks for the review. Sorry for the late reply. I've posted
> a new version of the draft which incorporate the feedback from the review.
>
> Please see our responses in-line:
>
> --
> Denis Reilly  |  Technical Lead  |  [email-masked]  (585)321-5837
>
> -----Original Message-----
> From: Eric Rescorla <[email-masked]>
> Sent: Wednesday, December 19, 2018 8:49 PM
> To: The IESG <[email-masked]>
> Cc: [email-masked]; Karen O'Donoghue <[email-masked]>;
> [email-masked]; [email-masked]; [email-masked]
> Subject: Eric Rescorla's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS
> and COMMENT)
>
> Eric Rescorla has entered the following ballot position for
> draft-ietf-ntp-bcp-10: Discuss
>
> When responding, please keep the subject line intact and reply to all
> email addresses included in the To and CC lines. (Feel free to cut this
> introductory paragraph, however.)
>
>
> Please refer to https://www.ietf.org/iesg/statement/discuss-criteria.html
> for more information about IESG DISCUSS and COMMENT positions.
>
>
> The document, along with other ballot positions, can be found here:
> https://datatracker.ietf.org/doc/draft-ietf-ntp-bcp/
>
>
>
> ----------------------------------------------------------------------
> DISCUSS:
> ----------------------------------------------------------------------
>
> Rich version of this review at:
> https://mozphab-ietf.devsvcdev.mozaws.net/D3844
>
> I notice that a number of the recommendations here differ from those in
> NDSS16. In particular the following recommendations from that paper do not
> seem to appear:
>
> - Do not put INIT in the reference ID on restart
> - Do not send KoD
> - Disable fragmentation
> - Randomize source ports
>
> I'm not saying that all of these recommendations need to be in this
> document, but I am curious why they are not and would tend to think that
> one should document why they are not.
>
> Response:
> We worked closely with Sharon Goldberg, one of the authors on the
> referenced paper, when writing this section. While the paper makes the
> recommendations you list, we simply didn't want to include everything in
> the BCP, but rather include the things that would have the most immediate
> impact.
>
> - The advice for not putting INIT in the reference ID on restart might be
> useful to include after all, as we do give other advice for implementors at
> the end of the "Avoiding Daemon Restart Attacks" section, so we will add it.
>
> - The paper lists "eliminate NTP's KoD" as one recommendation, but then
> immediately acknowledges that it eliminates a server's ability to deal with
> heavy volumes of traffic. We elected to not recommend the elimination of
> KoDs, in part because there will be many servers deployed who still use
> them.  Perhaps that is a topic for NTP v5, someday.
>
> - The paper suggests that the attack surface for the NTP fragmentation
> attack is "small but non-negligible". We felt the attack surface was not
> large enough
> to include directly in this BCP. Since the paper is referenced in multiple
> places, though, interested readers (such as yourself) would be able to find
> it.
>
> - The paper mentions Source Port Randomization as a possibility to
> mitigate the fragmentation attack, but then goes on to say that it is not a
> "sufficient defense".
>
>
All right.

> DETAIL
> S 2.1.
> >      response to a small query, which makes it more attractive as a
> vector
> >      for distributed denial-of-service attacks.  (NTP Control messages
> are
> >      discussed further in Section 3.4).  One documented instance of such
> >      an attack can be found here [1], and further discussion in [IMC14]
> >      and [NDSS14].  Mitigating source address spoofing attacks should be
> a
> >      priority of anyone administering NTP.
>
> what does this text mean? What practices can I engage in as an NTP server
> that mitigate source spoofing attacks? The next paragraph seems to largely
> talk about traffic *sources*. Is the assumption that the NTP server is
> supposed to do BCP 38 filtering? That seems not that effective.
>
> As a smaller point, I see that this text says "should", not SHOULD. Is
> that a mistake or is this intended not to have any normative force?
>
> Response:
> As part of another comment, we've moved the "mitigating source address
> spoofing attacks" sentence to the next paragraph, to further highlight
> it's relation to BCP38.
>
> BCP38 would provide some level of protection against spoofing attacks, but
> as you note it's not really effective when applied at the server
> level. This is why we give the guidance for large ISP's and networks.
>
> We do have normative language later in that paragraph now saying that "It
> is RECOMMENDED that ISP's and large corporate networks implement ingress
> and egress filtering", and we feel that's enough.
>

But these aren't things that the *server* can do, really. So, I don't
understand how it's meaningful to say that it should be a priority to
mitigate these attacks for "anyone administering NTP"

> S 3.2.
> >      See Section 3.7.1 for more information.
> >
> >      Operators SHOULD monitor all of the time sources that are in use.
> If
> >      time sources do not generally agree, find out the cause and either
> >      correct the problems or stop using defective servers.  See
> >      Section 3.5 for more information.
>
> It's not really possible to evaluate this advice without any description
> of the threat model, which is pretty sketchily covered below. In
> particular, if an attacker controls my network, then it's basically like
> having one NTP server, no matter how many people I am talking to, and even
> an inaccurate but secure server (e.g., tlsdate) is superior.
>
> Response:
> In Sec. 3.2 we give advice on the number of time sources so that NTP's
> inherent selection mechanisms are enabled to protect against
> unintentionally false time sources. We don't consider the case in which
> adversaries are able to manipulate NTP packets. This is done in Sec. 4.
>

Then you need to state that this isn't intended to deal with this threat
model.

> S 3.2.
> >
> >      o  If there are 2 sources of time and they agree well enough, then
> >         the best time can be calculated easily.  But if one source fails,
> >         then the solution degrades to the single-source solution outlined
> >         above.  And if the two sources don't agree, then it's impossible
> >         to know which one is correct by simply looking at the time.
>
> This isn't strictly true. Consider the case where I have an iPhone and the
> onboard clock reads 2018-12-19 and the NTP server reads 2001. I know the
> NTP server is wrong because iPhones didn't exist in 2001.
>
> Response:
> Only if the iPhone is smart enough to disqualify the rogue source, in
> which case you're still back to 1 source. I think the advice here still
> stands.
>

The problem here is the statement that it is "impossible". That is clearly
false.

> --
>
> S 3.4.
> >      optionally authenticated control of NTP and its configuration.  Used
> >      properly, these facilities provide vital debugging and performance
> >      information and control.  Used improperly, these facilities can be
> an
> >      abuse vector.  For this reason, it is RECOMMENDED that publicly-
> >      facing NTP servers should block mode 6 queries from outside their
> >      organization.
>
> Why are these facilites an abuse vector
>
> Response:
> We've clarified that a bit: "But these facilities can be a vector for
> amplification attacks when abused."
>

OK.

--
>
>
> S 3.5.
> >
> >      If a system starts getting unexpected time replies from its time
> >      servers, that can be an indication that the IP address of the system
> >      is being forged in requests to its time server.  The goal of this
> >      attack is to convince the time server to stop serving time to the
> >      system whose address is being forged.
>
> Why would this work? Some sort of rate limit on the server.
>
> Response:
> We've rewritten this for clarity:
> If a system starts to receive NTP Reply packets from a time server
> that do not correspond to any requests sent by the system, that can be
> an indication that an attacker is forging that system's IP address in
> requests to the remote time server. The goal of this attack would be to
> convince the time server to stop serving time to the
> system whose address is being forged.
>

I think you need to state that the time server will potentially throttle in
response to this.

S 7.
> >      anycast servers may arbitrarily enter and leave the network, the
> >      server a particular client is connected to may change.  This may
> >      cause a small shift in time from the perspective of the client when
> >      the server it is connected to changes.  It is RECOMMENDED that
> >      anycast only be deployed in environments where these small shifts
> can
> >      be tolerated.
>
> Who is this guidance to? It seems like the clients might well not know,
> but they are the ones who tolerate the shift (or not).
>
> Response:
> The guidance is to the network operators, and ultimately to the users of
> the  NTP service. Utilizing Anycast in this manner may affect the quality
> of the
> recovered time. But in some applications, this is preferable to
> potentially  having to change the configuration of a large number of
> clients whenever a server is added or removed.
>

How do the clients know whether anycast is in use?

-Ekr

---

From: Denis Reilly <[email-masked]>
Date: Tue, 26 Mar 2019 22:03:07 +0000

Thanks for your feedback! I see 5 places where you require a response, please find them below with the tag “Mar 26 Response:”

I will be updating the draft with these changes shortly.

Best Regards,

--
Denis Reilly  |  Technical Lead  |  [email-masked]<mailto:[email-masked]>  (585)321-5837<tel:(585)%20321-5837>

From: Eric Rescorla <[email-masked]>
Sent: Friday, March 22, 2019 12:29 PM
To: Denis Reilly <[email-masked]>
Cc: The IESG <[email-masked]>; [email-masked]; [email-masked]; [email-masked]; [email-masked]
Subject: Re: Eric Rescorla's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS and COMMENT)

On Thu, Jan 17, 2019 at 10:14 AM Denis Reilly <[email-masked]<mailto:[email-masked]>> wrote:
Hello, Eric! Thanks for the review. Sorry for the late reply. I've posted a new version of the draft which incorporate the feedback from the review.

--
Denis Reilly  |  Technical Lead  |  [email-masked]<mailto:[email-masked]>  (585)321-5837

-----Original Message-----
From: Eric Rescorla <[email-masked]<mailto:[email-masked]>>
Sent: Wednesday, December 19, 2018 8:49 PM
To: The IESG <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; Karen O'Donoghue <[email-masked]<mailto:[email-masked]>>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>; [email-masked]<mailto:[email-masked]>
Subject: Eric Rescorla's Discuss on draft-ietf-ntp-bcp-10: (with DISCUSS and COMMENT)

When responding, please keep the subject line intact and reply to all email addresses included in the To and CC lines. (Feel free to cut this introductory paragraph, however.)

I notice that a number of the recommendations here differ from those in NDSS16. In particular the following recommendations from that paper do not seem to appear:

I'm not saying that all of these recommendations need to be in this document, but I am curious why they are not and would tend to think that one should document why they are not.

Response:
We worked closely with Sharon Goldberg, one of the authors on the referenced paper, when writing this section. While the paper makes the recommendations you list, we simply didn't want to include everything in the BCP, but rather include the things that would have the most immediate impact.

- The advice for not putting INIT in the reference ID on restart might be useful to include after all, as we do give other advice for implementors at the end of the "Avoiding Daemon Restart Attacks" section, so we will add it.

- The paper lists "eliminate NTP's KoD" as one recommendation, but then immediately acknowledges that it eliminates a server's ability to deal with heavy volumes of traffic. We elected to not recommend the elimination of KoDs, in part because there will be many servers deployed who still use them.  Perhaps that is a topic for NTP v5, someday.

- The paper suggests that the attack surface for the NTP fragmentation attack is "small but non-negligible". We felt the attack surface was not large enough
to include directly in this BCP. Since the paper is referenced in multiple places, though, interested readers (such as yourself) would be able to find it.

- The paper mentions Source Port Randomization as a possibility to mitigate the fragmentation attack, but then goes on to say that it is not a "sufficient defense".

All right.

DETAIL
S 2.1.
>      response to a small query, which makes it more attractive as a vector
>      for distributed denial-of-service attacks.  (NTP Control messages are
>      and [NDSS14].  Mitigating source address spoofing attacks should be a

what does this text mean? What practices can I engage in as an NTP server that mitigate source spoofing attacks? The next paragraph seems to largely talk about traffic *sources*. Is the assumption that the NTP server is supposed to do BCP 38 filtering? That seems not that effective.

As a smaller point, I see that this text says "should", not SHOULD. Is that a mistake or is this intended not to have any normative force?

Response:
As part of another comment, we've moved the "mitigating source address spoofing attacks" sentence to the next paragraph, to further highlight
it's relation to BCP38.

BCP38 would provide some level of protection against spoofing attacks, but as you note it's not really effective when applied at the server
level. This is why we give the guidance for large ISP's and networks.

We do have normative language later in that paragraph now saying that "It is RECOMMENDED that ISP's and large corporate networks implement ingress and egress filtering", and we feel that's enough.

But these aren't things that the *server* can do, really. So, I don't understand how it's meaningful to say that it should be a priority to mitigate these attacks for "anyone administering NTP"

Mar 26 Response:
Now we see the problem. We wanted to emphasize the important of mitigating these attacks, but as you note there’s not much that “anyone administering NTP” can do. And as stated before, we do give normative language later in the paragraph that does not have the same issue. So we have decided to remove that sentence entirely – we think the section still provides the proper guidance without it.

S 3.2.
>      Operators SHOULD monitor all of the time sources that are in use.  If

It's not really possible to evaluate this advice without any description of the threat model, which is pretty sketchily covered below. In particular, if an attacker controls my network, then it's basically like having one NTP server, no matter how many people I am talking to, and even an inaccurate but secure server (e.g., tlsdate) is superior.

Response:
In Sec. 3.2 we give advice on the number of time sources so that NTP's  inherent selection mechanisms are enabled to protect against unintentionally false time sources. We don't consider the case in which adversaries are able to manipulate NTP packets. This is done in Sec. 4.

Then you need to state that this isn't intended to deal with this threat model.

Mar 26 Response:
We will add this paragraph after our list:

This analysis assumes that a majority of the servers used in the
    solution are honest, even if some may be inaccurate.  Operators should
    be aware of the possibility that if an attacker is in control of the
    network, the time coming from all servers could be compromised.

S 3.2.
>         to know which one is correct by simply looking at the time.

This isn't strictly true. Consider the case where I have an iPhone and the onboard clock reads 2018-12-19 and the NTP server reads 2001. I know the NTP server is wrong because iPhones didn't exist in 2001.

Response:
Only if the iPhone is smart enough to disqualify the rogue source, in which case you're still back to 1 source. I think the advice here still stands.

The problem here is the statement that it is "impossible". That is clearly false.

Mar 26 Response:
We will change that sentence to:

And if the two sources don't agree, it will be
difficult to know which one is correct without making use of
information from outside the protocol.

--

S 3.4.
>      information and control.  Used improperly, these facilities can be an

Response:
We've clarified that a bit: "But these facilities can be a vector for amplification attacks when abused."

OK.

--

S 3.5.

Response:

I think you need to state that the time server will potentially throttle in response to this.

Mar 26 Response:
We propose clarifying the last sentence:

If a system starts to receive NTP Reply packets from a remote time server
that do not correspond to any requests sent by the system, that can be
an indication that an attacker is forging that system's IP address in
requests to the remote time server. The goal of this attack is to
adversely impact the availability of time to the targeted system whose
address is being forged. Based on these forged packets, the
remote time server might decide to throttle or rate limit packets, or
even stop sending packets to the targeted system.

S 7.
>      anycast only be deployed in environments where these small shifts can
>      be tolerated.

Who is this guidance to? It seems like the clients might well not know, but they are the ones who tolerate the shift (or not).

Response:
The guidance is to the network operators, and ultimately to the users of the  NTP service. Utilizing Anycast in this manner may affect the quality of the
recovered time. But in some applications, this is preferable to potentially  having to change the configuration of a large number of clients whenever a server is added or removed.

Mar 26 Response:
The clients have no way of knowing whether anycast is in use. so it's up to whoever is setting up the Anycast-based time transfer network to come to the determination that any small shifts can be tolerated. I guess there is an assumption here that the network operators know what the use case of their users are.

So we will change the recommendation:
It is RECOMMENDED that network operators
only deploy anycast NTP in environments where operators know these
small shifts can be tolerated by the applications running on the clients
being synchronized in this manner

-Ekr

ATTENTION: This email came from an external source.
Do not open attachments or click on links from unknown senders or unexpected emails.

---

From: Eric Rescorla <[email-masked]>
Date: Tue, 26 Mar 2019 20:46:24 -0700

These  proposed changes seem satisfactory.

-Ekr

On Tue, Mar 26, 2019 at 3:03 PM Denis Reilly <[email-masked]>
wrote:

> Thanks for your feedback! I see 5 places where you require a response,
> please find them below with the tag “Mar 26 Response:”
>
>
>
> <(585)%20321-5837>
> *From:* Eric Rescorla <[email-masked]>
> *Sent:* Friday, March 22, 2019 12:29 PM
> *To:* Denis Reilly <[email-masked]>
> *Cc:* The IESG <[email-masked]>; [email-masked]; [email-masked];
> [email-masked]; [email-masked]
> *Subject:* Re: Eric Rescorla's Discuss on draft-ietf-ntp-bcp-10: (with
> DISCUSS and COMMENT)
>
>
>
>
>
>
>
> Now we see the problem. We wanted to emphasize the important of mitigating
> these attacks, but as you note there’s not much that “anyone administering
> NTP” can do. And as stated before, we do give normative language later in
> the paragraph that does not have the same issue. So we have decided to
> remove that sentence entirely – we think the section still provides the
> proper guidance without it.
>
>
>
>
>
>
>
> S 3.2.
> The clients have no way of knowing whether anycast is in use. so it's up
> to whoever is setting up the Anycast-based time transfer network to come to
> the determination that any small shifts can be tolerated. I guess there is
> an assumption here that the network operators know what the use case of
> their users are.
>
>
>
> *ATTENTION: This email came from an external source. Do not open
> attachments or click on links from unknown senders or unexpected emails. *
>