Subject: [Ntp] Microbursts of NTP packets

From: Watson Ladd <[email-masked]>
Date: Mon, 17 Aug 2020 21:20:03 -0400

Dear NTP WG,

We're observing short bursts of high numbers of NTP queries at one
point of presence, exceeding the queue length of the listening ntp
socket, and leading to drops. The bursts are very short, so the
overall qps is nothing special. I'm quite mystified as to what the
possible causes could be.

This is unfortunately leading to packet drops. If any operators have
seen this, their input on possible causes and solutions is welcome.

Sincerely,
Watson Ladd

---

From: Steven Sommars <[email-masked]>
Date: Mon, 17 Aug 2020 21:42:24 -0500

These may be the same 10 second NTP request bursts seen by administrators
of NTP pool servers; typically in the 1K to 20K Pkts/sec range, though
we've seen higher.   The bursts come from buggy FortiGate firewalls.
FortiNet tells me that FortiOS version 6.2.5, scheduled for release at the
end of this week, will correct the problem.  There's no telling when the
individual FortiGates will upgrade to that release.

I can provide further details, if there is interest.

Steve Sommars

On Mon, Aug 17, 2020 at 8:20 PM Watson Ladd <[email-masked]> wrote:

> _______________________________________________
> ntp mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/ntp
>

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Mon, 17 Aug 2020 19:44:20 -0700

> On Aug 17, 2020, at 18:20, Watson Ladd <[email-masked]> wrote:
> 

What are the times this happens and do they have a common set of source IPs or ASNs?

The NTP Pool community site might have similar or related data from other NTP operators.

Ask

---

From: Hal Murray <[email-masked]>
Date: Tue, 18 Aug 2020 00:08:32 -0700

[I've been working on this with Steven Sommars.]

> We're observing short bursts of high numbers of NTP queries at one point of
> presence, exceeding the queue length of the listening ntp socket, and leading
> to drops. The bursts are very short, so the overall qps is nothing special.
> I'm quite mystified as to what the possible causes could be. 

You didn't provide any numbers, either for duration or size of burst.  What do 
you consider to be "very short"?

I've been using a gap of 1 second to separate bursts and ignoring bursts with 
less than 1000 packets.

Using tcpdump to gather data, I've seen 2 types of bursts.

Steve mention the bursts from Fortigate.  They are typically a bit under 10 
seconds long with a wide range of rates.  Some are over 150K packets or 15K 
packets/sec.

The other is from systems using NAT.  These bursts are typically 100-180 
seconds long which matches the TTL of the pool DNS info.  The burst rate is 
typically under 25 packets/second with a few over 100.

I confirmed with one admin that his IP address was a NAT box with a lab full 
of systems using the pool.  But that is only one data point so I may be 
jumping to conclusions.

Graphs here:
  http://users.megapathdsl.net/~hmurray/Burst/

> This is unfortunately leading to packet drops. If any operators have seen
> this, their input on possible causes and solutions is welcome. 

You probably want some sort of rate limiting to prevent your systems being 
used as a Reflector during DDoS attacks.

-- 
These are my opinions.  I hate spam.

---

From: "Ulrich Windl" <[email-masked]>
Date: Tue, 18 Aug 2020 09:31:43 +0200

>>> Watson Ladd <[email-masked]> schrieb am 18.08.2020 um 03:20 in Nachricht
> possible causes could be.

I think an important questions is: From the same source or not?
Also: Time queries, or control mode packets? If so it might be an attempt to attack...

---

From: Steven Sommars <[email-masked]>
Date: Tue, 18 Aug 2020 07:56:09 -0500

Each of the 10-second bursts were NTP mode 3 (client) requests coming from
a single source address, almost always IPv4.
Over the course of time we saw 10-second bursts originating from hundreds
of clients.
These are client bugs, not DDoS attacks.  Source addresses are not
spoofed.

Note that the people who provide DDoS mitigation services say that NTP
reflection attacks are still an issue.

NTP server rate limiting, as Hal stated, will reduce the impact of buggy
clients or intentional attacks,
This is not a trivial topic.