Subject: [DNSOP] Root server tar pitting? Is there a better way?

From: Brian Somers <[email-masked]>
Date: Mon, 16 May 2016 14:23:49 -0700

Hi folks,

I work at OpenDNS.  We saw a DoS attack in Miami on Friday night around 10-11:00pm PST, consisting of UDP DNS requests for AAA.BBB.CCC.DDD where each of AAA, BBB, CCC and DDD are three digit numbers not greater than 500.

Each query was answered with an NXDOMAIN by the root servers,   Although our resolvers cached the NXDOMAIN for 1 hour (we cap negative responses at 1 hour despite the larger SOA MINIMUM) it was ineffective in reducing the load on the root servers as every varying query was another root server request.

We eventually blackholed all TLDs from 000 to 500 to stifle the problem (locally delegating them to 127.0.0.1 where we don’t listen).

However, during the attack, we also saw a huge number of TCP sockets in TIME_WAIT talking to root servers (probably all root servers).  I’m curious if

1.  Are root servers doing some sort of tar pitting where they send a TC and then firewall port 53?
2.  Has anyone ever considered a better way than responding with NXDOMAIN?

The second is a loaded question, but it occurs to me that a new type of negative response to (say) 111.222.333.444/IN/A might be an NXDOMAIN with an SOA record (as we do now) but also with an indicator that 444 and below are NXDOMAINs.  I’m not sure what that might look like, maybe "444/IN/NS .” in the AUTHORITY section where “.” is the NS value meaning that 444 is actually delegated to nobody.

Thoughts/comments?

—
Brian

---

From: "Wessels, Duane" <[email-masked]>
Date: Mon, 16 May 2016 21:34:17 +0000

Hi Brian,

I think what you're suggesting has already been proposed.  See https://datatracker.ietf.org/doc/draft-fujiwara-dnsop-nsec-aggressiveuse/ and https://datatracker.ietf.org/doc/draft-wkumari-dnsop-cheese-shop/

DW

> On May 16, 2016, at 2:23 PM, Brian Somers <[email-masked]> wrote:
> 
> Hi folks,
> 
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Marek Vavruša <[email-masked]>
Date: Mon, 16 May 2016 14:40:49 -0700

Why not run a local copy of the root? It should be a good practice for
large recursives, plus you get better latency.

Marek

---

From: bert hubert <[email-masked]>
Date: Mon, 16 May 2016 23:45:07 +0200

> I think what you're suggesting has already been proposed.  See https://datatracker.ietf.org/doc/draft-fujiwara-dnsop-nsec-aggressiveuse/ and https://datatracker.ietf.org/doc/draft-wkumari-dnsop-che…

It is in fact something you can do today. Some of the largest PowerDNS
Recursor sites in the world run with 'root-nx-trust' enabled:

"If set, an NXDOMAIN from the root-servers will serve as a blanket NXDOMAIN
for the entire TLD the query belonged to. The effect of this is far fewer
queries to the root-servers."

This after f-root had enabled RRL slightly too aggressively on some nodes.

We just tested this setting against the "owned Ubiquity" attack and after a
thousand queries or so traffic to the roots dropped off to almost zero.

        Bert

---

From: Shumon Huque <[email-masked]>
Date: Mon, 16 May 2016 18:35:10 -0400

On Mon, May 16, 2016 at 5:45 PM, bert hubert <[email-masked]>
wrote:

> > I think what you're suggesting has already been proposed.  See
> https://datatracker.ietf.org/doc/draft-fujiwara-dnsop-nsec-aggressiveuse/
> and https://datatracker.ietf.org/doc/draft-wkumari-dnsop-cheese-shop/
>

Bert,

PowerDNS's root-nx-trust is I believe an implementation of what is described
in nxdomain-cut:

    https://tools.ietf.org/html/draft-ietf-dnsop-nxdomain-cut-03

rather than the nsec-aggressive-use or cheese-shop drafts - those are about
inferring NXDOMAIN from NSEC/NSEC3 spans.

Shumon.

---

From: John Heidemann <[email-masked]>
Date: Mon, 16 May 2016 16:22:45 -0700

>1.  Are root servers doing some sort of tar pitting where they send a TC and then firewall port 53?

Being asked to retry with TCP sounds like Response Rate Limiting (RRL).

See for example
https://kb.isc.org/article/AA-00994/0/Using-the-Response-Rate-Limiting-Feature-in-BIND-9.10.html

The KB article talks about doing it in response to a large number of
NXDOMAINs per unit time, as it sounds like you encountered.

   -John Heidemann

---

From: Tony Finch <[email-masked]>
Date: Tue, 17 May 2016 01:04:12 +0100

> Hi folks,
 
Hi Brian!
 
> However, during the attack, we also saw a huge number of TCP
> sockets in
> TIME_WAIT talking to root servers (probably all root servers).  I’m
> curious if
>
> 1.Are root servers doing some sort of tar pitting where they send a TC
> and then firewall port 53?
 
This TIME_WAIT problem is a normal consequence of making lots of short-
lived TCP connections for which you initiated the close, i.e. a
tragic mismatch between what a recursive DNS resolver needs and what
TCP provides.
 
The standard TIME_WAIT period is huge compared to typical segment
lifetimes. You can reduce it using sysctl on FreeBSD but not Linux.
Linux instead has a couple of options to recycle TIME_WAIT sockets, one
of which is (IME) ineffective in this kind of situation, and the other
is reportedly unsafe.
 
Tony.
--
f.anthony.n.finch  <[email-masked]>  http://dotat.at/  -  I xn--
  zr8h punycode

---

From: "John Levine" <[email-masked]>
Date: 17 May 2016 00:19:55 -0000

>large recursives, plus you get better latency.

That's what I'd do, too.  It's easy to set up and it avoids a whole
lot of problems.  The root zone is still very small, and it's
surprisingly easy to set up.  I did it on my small system in about ten
minutes.

See RFC 7706 for more details.

R's,
John

---

From: "Peter van Dijk" <[email-masked]>
Date: Tue, 17 May 2016 11:14:01 +0200

Hello Shumon,

> inferring NXDOMAIN from NSEC/NSEC3 spans.

There is a subtle difference. We send the full query to the root, and 
get an
NXDOMAIN for the full name, but with the setting enabled, we believe 
that the
NXDOMAIN was generated from the top label. In other words, we rely on 
the
‘shape’ of the root zone in that every positive entry in it is only 
one label
long.

Kind regards,
-- 
Peter van Dijk
PowerDNS.COM BV - https://www.powerdns.com/

---

From: bert hubert <[email-masked]>
Date: Mon, 16 May 2016 23:43:51 +0200

Bert

---

From: Johan Ihrén <[email-masked]>
Date: Tue, 17 May 2016 12:37:11 +0200

Hi,

On 17 May 2016, at 11:14 , Peter van Dijk <[email-masked]> wrote:

> There is a subtle difference. We send the full query to the root, and get an
> NXDOMAIN for the full name, but with the setting enabled, we believe that the
> NXDOMAIN was generated from the top label. In other words, we rely on the
> ‘shape’ of the root zone in that every positive entry in it is only one label
> long.

It strikes me that this is a case where qname minimization would not only help privacy, but also help with this problem as the resulting NXDOMAIN will cover the entire non-existent TLD.

Johan Ihrén
Netnod

---

From: Shumon Huque <[email-masked]>
Date: Tue, 17 May 2016 09:25:45 -0400

On Tue, May 17, 2016 at 6:37 AM, Johan Ihrén <[email-masked]> wrote:

> >> On Mon, May 16, 2016 at 5:45 PM, bert hubert <[email-masked]
> >
> >> wrote:
> >>

Ah, thanks for the clarification.

>
> It strikes me that this is a case where qname minimization would not only
> help privacy, but also help with this problem as the resulting NXDOMAIN
> will cover the entire non-existent TLD.
>

Yes, indeed.

-- 
Shumon Huque

---

From: bert hubert <[email-masked]>
Date: Tue, 17 May 2016 15:44:29 +0200

> inferring NXDOMAIN from NSEC/NSEC3 spans.

Got it, thanks - and this I-D is indeed useful to beat people over the head
with if they get it wrong.

I expect PowerDNS might extend the root-nx-trust to other domains, and we
might follow: 

"Another exception is that a validating resolver MAY decide to
 implement this behaviour only when the NXDOMAIN response has been
 validated with DNSSEC."

We can report from practice in any case that trusting the root to get it
right cuts down on root traffic a lot.

Thanks!

	Bert

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sat, 21 May 2016 14:52:37 +0200

On Tue, May 17, 2016 at 12:37:11PM +0200,
 Johan Ihrén <[email-masked]> wrote 
 a message of 56 lines which said:

> It strikes me that this is a case where qname minimization would not
> only help privacy, but also help with this problem as the resulting
> NXDOMAIN will cover the entire non-existent TLD.

Yes, QNAME minimisation (RFC 7816) and NXDOMAIN cut
(draft-ietf-dnsop-nxdomain-cut) are friends :-)

Brian, could OpenDNS consider implementing RFC 7816?

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Sat, 21 May 2016 14:55:10 +0200

On Tue, May 17, 2016 at 03:44:29PM +0200,
 bert hubert <[email-masked]> wrote 
 a message of 31 lines which said:

> I expect PowerDNS might extend the root-nx-trust to other domains,

This is what Unbound does (see section 8 of
draft-ietf-dnsop-nxdomain-cut-03).

https://www.unbound.net/documentation/unbound.conf.html

harden-below-nxdomain: <yes or no>
              From  draft-vixie-dnsext-resimprove, returns nxdomain to queries
              for a name below another name that is already known to be  nxdo-
              main.   DNSSEC  mandates  noerror  for empty nonterminals, hence
              this is possible.  Very old software might return  nxdomain  for
              empty  nonterminals  (that usually happen for reverse IP address
              lookups), and thus may be incompatible with  this.   To  try  to
              avoid  this  only  DNSSEC-secure nxdomains are used, because the
              old software does not have DNSSEC.  Default is off.