Subject: Re: [DNSOP] Working Group Last Call: draft-ietf-dnsop-nxdomain-cut

From: Tim Wicinski <[email-masked]>
Date: Tue, 21 Jun 2016 16:34:57 -0400

All

The WGLC last call ended on this awhile ago, and the authors addressed 
all the comments that were raised during the WGLC process.

I want to thank everyone for their work on this.  We'll be moving this 
one along very shortly.

tim

On 5/26/16 8:53 AM, Tim Wicinski wrote:
> There has been some good discussion from the working group on the list,
> and with the recent discussion on root server tar pitting, it makes
> sense to move this document along.
>
> This starts a Working Group Last Call for draft-ietf-dnsop-nxdomain-cut
>
> Current versions of the draft is available here:
>
> https://datatracker.ietf.org/doc/draft-ietf-dnsop-nxdomain-cut/
>
> This is a Standards Track document.
>
> Please review the draft and offer relevant comments. Also, if someone
> feels the document is *not* ready for publication, please speak out with
> your reasons.
>
> This starts a two week Working Group Last Call process, and ends at
> midnight 10 June 2016.
>
> thanks
> tim

---

From: Brian Somers <[email-masked]>
Date: Tue, 21 Jun 2016 16:07:00 -0700

On Jun 21, 2016, at 1:34 PM, Tim Wicinski <[email-masked]> wrote:
> 
> 
> All
> 
> The WGLC last call ended on this awhile ago, and the authors addressed all the comments that were raised during the WGLC process.
> 
> I want to thank everyone for their work on this.  We'll be moving this one along very shortly.
> 
> tim
> 
> 

Sorry I’m late to the party, and hopefully I’m misunderstanding, but…. using tinydns as an authoritative server:

bsomers@dev194:~$ dig dev194.sjc.opendns.com @auth1.opendns.com

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> dev194.sjc.opendns.com @auth1.opendns.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 5740
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 5
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;dev194.sjc.opendns.com.                IN      A

;; ANSWER SECTION:
dev194.sjc.opendns.com. 3600    IN      A       67.215.93.64

;; AUTHORITY SECTION:
opendns.com.            604800  IN      NS      auth1.opendns.com.
opendns.com.            604800  IN      NS      auth2.opendns.com.
opendns.com.            604800  IN      NS      auth3.opendns.com.

;; ADDITIONAL SECTION:
auth1.opendns.com.      172800  IN      AAAA    2620:119:30::53
auth1.opendns.com.      172800  IN      A       208.69.39.2
auth2.opendns.com.      172800  IN      AAAA    2a04:e4c0:53::53
auth2.opendns.com.      172800  IN      A       146.112.60.53
auth3.opendns.com.      172800  IN      A       208.69.39.2

;; Query time: 9 msec
;; SERVER: 2620:119:30::53#53(2620:119:30::53)
;; WHEN: Tue Jun 21 22:56:28 UTC 2016
;; MSG SIZE  rcvd: 220

bsomers@dev194:~$ dig sjc.opendns.com @auth1.opendns.com

; <<>> DiG 9.9.5-3ubuntu0.8-Ubuntu <<>> sjc.opendns.com @auth1.opendns.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 53475
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
;; WARNING: recursion requested but not available

;; QUESTION SECTION:
;sjc.opendns.com.               IN      A

;; AUTHORITY SECTION:
opendns.com.            86400   IN      SOA     auth1.opendns.com. noc.opendns.com. 1466546759 16384 2048 1048576 2560

;; Query time: 5 msec
;; SERVER: 2620:119:30::53#53(2620:119:30::53)
;; WHEN: Tue Jun 21 22:56:41 UTC 2016
;; MSG SIZE  rcvd: 79

If we want to say somethings recursively non-existent, don’t we need an RNXDOMAIN rcode?

—
Brian

---

From: Robert Edmonds <[email-masked]>
Date: Tue, 21 Jun 2016 20:35:48 -0400

> Sorry I’m late to the party, and hopefully I’m misunderstanding, but…. using tinydns as an authoritative server:

Hi, Brian:

You might enjoy these threads:

    http://thread.gmane.org/gmane.ietf.dnsop/13393

    http://thread.gmane.org/gmane.ietf.dnsext/20301

    http://cr-yp-to.996295.n3.nabble.com/Fixing-the-NXDOMAIN-NODATA-bug-in-tinydns-td17150.html

-- 
Robert Edmonds

---

From: Marek Vavruša <[email-masked]>
Date: Tue, 21 Jun 2016 17:46:38 -0700

This affects several major DNS providers currently. I've heard Akamai
is rolling out update, but it's still returning NXDOMAIN for ENTs.
While deploying the draft in the current state of Internet is not
really viable and pointing out various broken implementations is fun,
I think it's good to have this draft as a reference from now on.

Marek

On Tue, Jun 21, 2016 at 5:35 PM, Robert Edmonds <[email-masked]> wrote:
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Mark Andrews <[email-masked]>
Date: Wed, 22 Jun 2016 18:52:23 +1000

> I think it's good to have this draft as a reference from now on.

Which is really not a reason to not deploy.  There are lots of
things broken in deployed nameservers and waiting for them to be
fixed before deploying would result in us never deploying new
features.

We are pushing ahead with deploying DNS COOKIES, on by default,
despite knowing that it will result on resolution failures for a
small percentage of zones, slower (extra round trips for incorrect
rcodes) and very slow (multiple seconds as we discover the poorly
configured firewalls) lookups to others and "wrong" results for
still others (NODATA instead of DATA, NXDOMAIN instead of DATA).

We also shouldn't put on hold deploying changes to depending upon
correct ENT behaviour.  It's not like people haven't had a decade
to fix their servers since the issue was identified.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 22 Jun 2016 12:13:13 +0200

On Tue, Jun 21, 2016 at 05:46:38PM -0700,
 Marek Vavruša <[email-masked]> wrote 
 a message of 20 lines which said:

> pointing out various broken implementations is fun,

It is not "fun", it is the only way to have broken implementations
(Akamai, djbdns) fixed. If we do not name them, they will continue
forever.

---

From: Jim Reid <[email-masked]>
Date: Wed, 22 Jun 2016 11:36:10 +0100

> On 22 Jun 2016, at 11:13, Stephane Bortzmeyer <[email-masked]> wrote:
> 

I doubt that will help Stephane. djbdns has been broken for ~20 years -- no AXFR, no EDNS0, no TCP/53, no DNSSEC, no TSIG, etc, etc -- and likely to be that way forever. DJB no longer supports or maintains the software, apart from fixing security vulnerabilities IIUC. So the DNS is probably always going to be stuck with that abandonware being in the state it was in at its last "stable release" in 2001.

The best we can hope from naming and shaming is to encourage people to keep away from implementations that are broken and/or cause serious interoperability issues. Whether that advice will get heard and acted on is another matter of course. Darwinism will eventually take care of the poor DNS deployment choices anyway.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 22 Jun 2016 13:41:20 +0200

On Wed, Jun 22, 2016 at 11:36:10AM +0100,
 Jim Reid <[email-masked]> wrote 
 a message of 21 lines which said:

> djbdns has been broken for ~20 years -- no AXFR, no EDNS0, no
> TCP/53, no DNSSEC, no TSIG, etc, etc --

I'm may be too picky but these cases are different: djbdns misses some
features, some mandatory (TCP), some facultative (DNSSEC). It is not
the same thing as a bug (violation of the standard).

---

From: Marek Vavruša <[email-masked]>
Date: Wed, 22 Jun 2016 10:55:46 -0700

Naming and shaming is not the only mechanism for change, and it's not
effective when things "work". Sunsetting and deprecating parts of
protocols (like IQUERY or TYPE=ANY) is more important as it helps not
only to keep the protocol concise, but also forces evolution of
protocol implementations (or at least brings implementors back for
discussions).

Marek

---

From: "John Levine" <[email-masked]>
Date: 22 Jun 2016 18:00:13 -0000

>the same thing as a bug (violation of the standard).

Actually, djbdns does TCP and AXFR perfectly well, albeit using a
separate program from the one that handles UDP queries.  I swapped
secondaries with a BIND site for over a decade using its AXFR, perked
up with a little perl script that looked at the SOAs to limit the
useless AXFRs.

It doesn't do EDNS0 or DNSSEC, partly because Dan is stubborn, but
mostly because it's been abandonware for almost 20 years.  I patched
for a while but gave up and switched to unbound and nsd.

R's,
John

---

From: Marek Vavruša <[email-masked]>
Date: Wed, 22 Jun 2016 11:08:21 -0700

> still others (NODATA instead of DATA, NXDOMAIN instead of DATA).

Deploying DNS COOKIES is a completely different risk. It is likely to
result in "small percentage" of resolution failures with some really
broken intermediates. Broken NXDOMAIN for ENT sadly affects large
portion of the Internet (CDNs notably). While I personally would be
okay to flip the switch for testing, I wouldn't do so in a position of
big recursive service.

> to fix their servers since the issue was identified.

How are you imagining to enforce that? The only thing a software
vendor can do is to give operators the option,
but deploying that is decision of someone else. It's not like you and
me are going to rappel down to Google's mainframe tomorrow and deploy
worldwide.

---

From: Dave Lawrence <[email-masked]>
Date: Tue, 28 Jun 2016 00:48:13 -0400

On 22 Jun 2016, at 11:13, Stephane Bortzmeyer <[email-masked]> wrote:
> (Akamai, djbdns) fixed. If we do not name them, they will continue forever.

Even ignoring the loaded "shaming" terminology that Jim added to this
thread, I still have to disagree a bit.  It is not the only way, and
certainly there is brokenness that gets fixed without publicly
pointing fingers.

For example, in the case of the conceptual failure of the Akamai
nameserver to properly identify empty non-terminals in all necessary
answer-seeking paths, this was a problem that I noted internally a
decade ago.  Yet it was triaged well below other work because it had
no practical operational impact at the time and fixing it was
non-trivial.

Only when the qname minimisation draft came around did it really
become an operational issue, and the priority of the change request
was immediately raised.  No one needed to point out to us either that
we had the problem or that operational evolution made it matter.
Working on fixing it was going to happen regardless.

As to why it has taken so long, the public deployment of a fix was
delayed when we discovered that some customers actually relied on the
non-compliant behaviour.  Believe me, I was frustrated by this
probably more than anyone else.

All that said, I don't really object to Akamai's nameserver having
been publicly named as a problem for qname minimisation.  Identifying
potential problems is obviously a crucial part of protocol
development.  Just remember, though, on n'attrape pas les mouches avec
du vinaigre.  We are all colleagues here, working to the same end -- a
better Internet.

---

From: Mark Andrews <[email-masked]>
Date: Tue, 28 Jun 2016 17:30:06 +1000

> du vinaigre.  We are all colleagues here, working to the same end -- a

The problem with that it takes time to get fixes deployed.  For
Akamai the path to fix has distance 1.  For other nameservers the
path has a much higher distance with each link in the deployment
chain introducing delays and quite often you have to argue that the
operator / developer / os maintainer needs to do something.

You have clearly broken nameservers where operators say they have
no problems when you report a issue due to depending upon a protocol
feature.

If we were to deploy EDNS(1) we would have to fight with 15-30% of
the servers dropping the queries because firewall developers are
overly paranoid.  Yes, nameserver developers mishandle EDNS(1) but
not to the extent where you need to block the queries.  The biggest
problem is that they ignore the EDNS version field and after that
they return formerr.  Firewalls are a bigger problem than the broken
nameservers.

When deploying a EDNS option you have nameservers that return
FORMERR, BADVERS and echo the option back.  These are 5 minute fixes
for the developer and really should be made.  Firewalls dropping
the queries are luckily a very small percentage of the errors seen.

Mark