Subject: [DNSOP] Fwd: New Version Notification for draft-livingood-dnsop-negative-trust-anchors-01.txt

From: Warren Kumari <[email-masked]>
Date: Fri, 24 Oct 2014 12:01:04 -0400

Hi all,

This draft has risen from the deep...

It describes a technique that a number of DNS operators use to
surgically / tactically deal with DNSSEC validation failures, for
large-scale outages.

We believe that this is needed -- simply telling customers "This
doesn't work though us, but does work through
$non-validating-competitor because we are better" simply leads to
customers changing to $non-validating-competitor, or operator turning
off DNSSEC for everybody.

I know that there will be some philosophical objections / discussions on this...

W

W

---------- Forwarded message ----------
From:  <[email-masked]>
Date: Thu, Oct 23, 2014 at 1:06 PM
Subject: New Version Notification for
draft-livingood-dnsop-negative-trust-anchors-01.txt
To: Ralf Weber <[email-masked]>, Jason Livingood
<[email-masked]>, Warren Kumari
<[email-masked]>, Chris Griffiths <[email-masked]>, Paul
Ebersman <[email-masked]>

A new version of I-D, draft-livingood-dnsop-negative-trust-anchors-01.txt
has been successfully submitted by Warren Kumari and posted to the
IETF repository.

Name:           draft-livingood-dnsop-negative-trust-anchors
Revision:       01
Title:          Definition and Use of DNSSEC Negative Trust Anchors
Document date:  2014-10-23
Group:          Individual Submission
Pages:          17
URL:
http://www.ietf.org/internet-drafts/draft-livingood-dnsop-negative-trust-anchors-01.txt
Status:
https://datatracker.ietf.org/doc/draft-livingood-dnsop-negative-trust-anchors/
Htmlized:
http://tools.ietf.org/html/draft-livingood-dnsop-negative-trust-anchors-01
Diff:
http://www.ietf.org/rfcdiff?url2=draft-livingood-dnsop-negative-trust-anchors-01

Abstract:
   DNS Security Extensions (DNSSEC) is now entering widespread
   deployment.  However, domain signing tools and processes are not yet
   as mature and reliable as those for non-DNSSEC-related domain
   administration tools and processes.  Negative Trust Anchors
   (described in this document) can be used to mitigate DNSSEC
   validation failures.

   [ Editor note: This document was originally draft-livingood-negative-
   trust-anchors-07 - renamved at the request of the DNSOP chairs ]

Please note that it may take a couple of minutes from the time of submission
until the htmlized version and diff are available at tools.ietf.org.

The IETF Secretariat

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: David Conrad <[email-masked]>
Date: Fri, 24 Oct 2014 09:06:39 -0700

> This draft has risen from the deep...

Like http://www.sfweekly.com/foodie/2008/03/31/dread-cherry-chip-cthulhu-rises-from-the-deep?

> off DNSSEC for everybody.

+1

Regards,
-drc

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 24 Oct 2014 15:39:07 -0400

>> off DNSSEC for everybody.

And since Mr. Conrad has shown fine experience using Internet Search 
Tools, I'm volunteering him for editing and review of this document.

tim

---

From: Doug Barton <[email-masked]>
Date: Fri, 24 Oct 2014 15:06:52 -0700

On 10/24/14 9:01 AM, Warren Kumari wrote:
> doesn't work through us, but does work through

Well that would be an incredibly stupid thing to tell a customer. :)

"Currently the operators of example.com have introduced an error into 
their DNS information which is causing the domain to be unresolvable. 
Once they have corrected that error our customers will be able to reach 
their site again. You can go to the following web page to view detailed 
information about the problem ..."

I'm aware of the problems that providers face when something doesn't 
work and it's not their fault; been there, done that. But ...

> I know that there will be some philosophical objections / discussions on this...

It's not just a philosophical objection, it's an operational one. When 
DNSSEC fails for a domain there are 2 main reasons. Operator error, and 
an actual MITM or similar attack. If the operators of validating 
resolvers simply turn off validation for domains that should be 
validating but are not,* it kicks the door open for the exact problem 
that DNSSEC was designed to solve.

But worse yet, in the operator error case you make such errors painless. 
Instead, if they are painful (as in, screw up DNSSEC and you go off 
line) then it leads to more people taking DNSSEC seriously, and doing it 
right. Of course I realize that the counter-argument is that if DNSSEC 
fails are too painful then people will simply not do it. But to the 
extent that people use that as a line of reasoning it's simply one more 
in a long string of excuses.

The other problem is that this feature is only really useful in the 
DNSSEC ramp-up period. Sure, mistakes are more common now, software is 
immature, etc. etc. But if DNSSEC is successful, the software will get 
better (it already is a lot better than even a few years ago), and 
mistakes will be less common (both on an absolute, and on a percentage 
basis). But once you introduce a feature like this, you cannot remove it.

So can we please let the functionality of DNSSEC stand as designed, and 
not give people a giant trapdoor that they can trigger at the slightest 
sign of trouble? Otherwise, why bother?

... and of course, I should point out that adding this as a knob is 
utterly pointless, since any reasonably competent validating resolver 
operator can engineer their own trapdoor. IMO this further cements the 
argument that adding this as a knob in software will only facilitate its 
use by people who should not be using it.

Doug

* I realize that the proponents of this mechanism believe that everyone 
who uses it will use it "properly," by validating a trusted source for 
information about why it's failing, only leaving it enabled for a short 
time, etc. Can we please be honest about the fact that in the majority 
of cases that simply won't be true?

---

From: Ralf Weber <[email-masked]>
Date: Sat, 25 Oct 2014 07:12:42 +0300

Moin!

> On 25 Oct 2014, at 01:06, Doug Barton <[email-masked]> wrote:
> 
> 
> "Currently the operators of example.com have introduced an error into their DNS information which is causing the domain to be unresolvable. Once they have corrected that error our customers will be able to reach their site again. You can go to the following web page to view detailed information about the problem ..."
How are you going to tell that to your customers? Who will pay for the calls to the ISPs call center?

> The other problem is that this feature is only really useful in the DNSSEC ramp-up period. Sure, mistakes are more common now, software is immature, etc. etc. But if DNSSEC is successful, the software will get better (it already is a lot better than even a few years ago), and mistakes will be less common (both on an absolute, and on a percentage basis). But once you introduce a feature like this, you cannot remove it.
The feature already exists  in the most commonly used validating resolvers (see Appendix A). Beside my tiny contribution to the draft you may notice that the other authors of the draft come from the two biggest currently operational validating resolver farms (Comcast and Google). So I would rather say it is operational reality and so far all the ISPs I talked to who are thinking about doing validation see this as a critical feature also. 

This draft describes that and tries to tell them to be super cautious with that feature. 

So long
Ralf

Sent from my iPhone

---

From: "Livingood, Jason" <[email-masked]>
Date: Sat, 25 Oct 2014 13:57:48 +0000

On 10/24/14, 6:06 PM, "Doug Barton" <[email-masked]> wrote:

>Instead, if they are painful (as in, screw up DNSSEC and you go off line)
>then it leads to more people taking DNSSEC seriously, and doing it right.
>Of course I realize that the counter-argument is that if DNSSEC fails are
>too painful then people will simply not do it. But to the extent that
>people use that as a line of reasoning it's simply one more in a long
>string of excuses.

I think we¹re seeing a collision of how we¹d like the world to work and
how it actually works. ;-) Operators are on the front lines here; trust us
on the realities of getting stuff like this deployed.

While I totally get & 110% agree that when someone screws up their auth
records they should feel the pain - and that this pain is a signal to do
it better/right the next time - the reality is that the real pain is felt
by the DNS operator. Of course I am going to pains to point out who is
really to blame here:
http://tools.ietf.org/html/draft-livingood-dnsop-auth-dnssec-mistakes-00

Anyway, if a popular domain like google.com or facebook.com or netflix.com
was to sign - and then made a mistake - the call centers of an validating
operator would pretty much melt down. It wouldn¹t matter that we were not
to blame, we¹d feel the pain and people would be angry at us. There¹d be
pressure to Œmake the pain stop¹ (and associated costs). Oh - and these
days in the U.S. we¹d probably be blamed for ³blocking access² to the site
- violating net neutrality (that the facts would say otherwise wouldn¹t
matter**).

So - two choices - turn off DNSSEC validation for ALL domains or just the
one affected (assuming you can prove out it is a mistake and not an
attack). Which one is worse?

The market has already answered that: Negative Trust Anchors - turn it off
in as narrow a case as possible. The question for us here is whether we
wish to acknowledge this reality or pretend it does not exist. ;-)

>basis). But once you introduce a feature like this, you cannot remove it.

We address the timeframe issue in the draft. Certainly you are correct
that the feature is still there afterwards, but of course the feature
already exists today so the genie is out of the bottle as they say.

>sign of trouble? Otherwise, why bother?

I wouldn¹t say that it is at the Œslightest¹ sign of trouble. This is
intended for use when there are major problems. Turning this around:
DNSSEC deployment only got as far as it did to date because of NTAs.
Without NTAs, I don¹t believe it¹ll go much further. So do we want to help
the world understand what it takes to push ahead with DNSSEC?

>operator can engineer their own trapdoor.

Yes - we all decided the way to do it was with a Negative Trust Anchor.
That¹s what the draft documents. ;-)

- Jason Livingood

** From the discussion above (scroll back up to refresh your memory). A
good recent example is a claim by some anonymous poster in a Reddit thread
that Comcast was blocking Tor. Within hours it went viral and was posted
to major news sites, never mind that it wasn¹t true. I posted
http://corporate.comcast.com/comcast-voices/setting-the-record-straight-on-
tor and yet still to this day there are tweets and reposts about how we
block Tor, pointing back to the original news articles. I can¹t wait for
what will happen when a major domain breaks if a NTA did not exist.

---

From: David Conrad <[email-masked]>
Date: Sat, 25 Oct 2014 07:55:31 -0700

Doug,

On Oct 24, 2014, at 3:06 PM, Doug Barton <[email-masked]> wrote:
> It's not just a philosophical objection, it's an operational one. When DNSSEC fails for a domain there are 2 main reasons. Operator error, and an actual MITM or similar attack.

In these early days of DNSSEC deployment, we've seen numerous occasions in which the first has applied. Can you provide URLs to the cases where the second has applied?

> If the operators of validating resolvers simply turn off validation for domains that should be validating but are not,* it kicks the door open for the exact problem that DNSSEC was designed to solve.

In the cases I'm aware of when an actual attack that DNSSEC was designed to solve has occurred, there are artifacts in logs. I would presume network operators would, when they get notified that a domain isn't validating, check those logs in their process to determine what's going on. If they notice the artifacts, I suspect they won't install an NTA.

> But worse yet, in the operator error case you make such errors painless.

Only if you believe there will be universal and instantaneous deployment of NTAs any time the zone operator screws up. I suspect this unlikely.

> Of course I realize that the counter-argument is that if DNSSEC fails are too painful then people will simply not do it. But to the extent that people use that as a line of reasoning it's simply one more in a long string of excuses.

No. It is an acceptance of the operational reality that, at least today, the most common failure mode (by far) is for zone operators to screw up and the remedy for that other person screwing up is to either tell the resolver operators' customers "sorry, the guys at <auth zone> made a booboo and you can't talk to them as long as you use our resolver" or turning validation off for ALL zones.

If a zone is sufficiently popular that a resolver operator gets multiple calls when it doesn't validate because the zone's operated boobooed, the resolver operator _will_ turn off validation.  Do that enough times and I suspect it becomes questionable whether they'll bother to turn it on again. 

> The other problem is that this feature is only really useful in the DNSSEC ramp-up period. Sure, mistakes are more common now, software is immature, etc. etc. But if DNSSEC is successful, the softw…

You seem to believe that installing NTAs is without cost. First, it requires operator intervention in response to a notification. Second, I believe it opens the resolver operator up to liability -- by installing an NTA, they are, by direct intervention of their staff, purposefully defeating a protection that they explicitly configured that puts all of their customers at risk. The longer the NTA is left in place, the greater the risk.

> ... and of course, I should point out that adding this as a knob is utterly pointless, since any reasonably competent validating resolver operator can engineer their own trapdoor.

Since not all validating resolvers support NTAs, if you want to supply text, I'd support adding a section describing alternative approaches to implementing NTA functionality.

Regards,
-drc

---

From: Paul Ebersman <[email-masked]>
Date: Sat, 25 Oct 2014 13:30:04 -0600

dougb> It's not just a philosophical objection, it's an operational
dougb> one. When DNSSEC fails for a domain there are 2 main
dougb> reasons. Operator error, and an actual MITM or similar attack. If
dougb> the operators of validating resolvers simply turn off validation
dougb> for domains that should be validating but are not,* it kicks the
dougb> door open for the exact problem that DNSSEC was designed to
dougb> solve.

Have you actually read through the new draft? It specifically prohibits
automatic installation of NTAs and says that you should have folks
familiar with operating DNS servers making any decisions.

That isn't painless. It means that this skips past all 1st tier and gets
to senior engineers. Don't know about you but I hate getting on-call
problems caused by someone else that I have no direct way to fix but
that my customers beat me for.

And that's way more expensive than standard help desk...

But it is a good way to make sure that this isn't a constant and knee
jerk reaction but an operationally sane one.

dougb> The other problem is that this feature is only really useful in
dougb> the DNSSEC ramp-up period. Sure, mistakes are more common now,
dougb> software is immature, etc. etc. But if DNSSEC is successful, the
dougb> software will get better (it already is a lot better than even a
dougb> few years ago), and mistakes will be less common (both on an
dougb> absolute, and on a percentage basis). But once you introduce a
dougb> feature like this, you cannot remove it.

Sure. It will only be temporary/fast. Like rolling out IPv6. Or getting
all the broken CPEs using old dnsmasq and being open resolvers. Or
getting folks to roll out BCP38.

How long have we been trying to get folks to use DNSSEC? How far are we?
And why do we keep insisting that only by painful and expensive
suffering will we do it "correctly"?

---

From: Olafur Gudmundsson <[email-masked]>
Date: Sun, 26 Oct 2014 01:43:28 +0000

> suffering will we do it "correctly"?

Just to amply Paul’s point 
We want humans in the loop, I would love to see a twitter feed when ever Comcast does a Negative Trust Anchor. 
I know of a ISP that is/will deploy DNSSEC is what they call passive mode i.e. they will never give SERVFAIL but will not
set AD bit when validation fails. 
IMHO that is a great way to avoid service calls but ….. 

Right now most of the errors are caused by mistakes by the DNS operator or bugs in new tools. 
In not so distant future we will start seeing lots of DNSSEC Operator Change errors, i.e. a new DNS operator takes over the operation of the
domain (perfectly legitimate), the errors in this case may be long lived (someone forgot a step), or short lived (someone moved to fast or emergency procedure was invoked) 

   Olafur

---

From: Paul Hoffman <[email-masked]>
Date: Sat, 25 Oct 2014 18:50:12 -0700

> We want humans in the loop, I would love to see a twitter feed when ever Comcast does a Negative Trust Anchor. 

Like https://twitter.com/ComcastDNS, for example? Either things haven't been failing much lately, or they're not updating it as often as we had hoped.

--Paul Hoffman

---

From: Warren Kumari <[email-masked]>
Date: Sat, 25 Oct 2014 22:00:30 -0400

> that my customers beat me for.

... or that your customers move away because of...
http://forums.comcast.com/t5/Basic-Internet-Connectivity-And/Re-Weather-Gov-DNS/td-p/1175625
http://www.dslreports.com/forum/remark,25180845?hilite=dns
http://forums.comcast.com/t5/Basic-Internet-Connectivity-And/Comcast-DNS-can-t-won-t-find-state-gov-sites-How-do-I-report/td-p/1824503

> dougb> feature like this, you cannot remove it.

... but you don't have to remove it. If " DNSSEC is successful, the
software will get better (it already is a lot better than even a
few years ago), and mistakes will be less common" then there will be
no need to install NTAs.

> How long have we been trying to get folks to use DNSSEC?

... a long time...

> How far are we?

not far enough...

> suffering will we do it "correctly"?

Not a clue. This is like saying that airbags and selt-belts are a bad
idea in cars -- if we didn't have them, people would learn to drive
more carefully[0]. This is potentially true, but... seriously...

W

[0]: This inserted somewhat so that I can trot out my current favorite quote:
Analogies are like bicycles: It doesn’t matter how many marbles it takes to
fill the outhouse, telco execs will always be making silly metaphors
 — Patrik Gilmore.

>
> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Warren Kumari <[email-masked]>
Date: Sat, 25 Oct 2014 22:12:11 -0400

> Like https://twitter.com/ComcastDNS, for example? Either things haven't been failing much lately, or they're not updating it as often as we had hoped.

Or both...

I suspect it might also be:
Installing a NTA is annoying. It requires poking at running servers,
you may have to talk to lawyers (shudder), you may have to get PR
people in the loop, etc.
This means that they only get put in for actual issues that affect a
large number of users. If maryandjohnsvacation.photo goes bogus
(because Mary typo'ed the entry in her crontab) it is highly unlikely
that you will get DNS operators to go through the rigmarole of
installing an NTA.

W

---

From: Paul Ebersman <[email-masked]>
Date: Sat, 25 Oct 2014 21:27:40 -0600

warren> ... but you don't have to remove it. If " DNSSEC is successful,
warren> the software will get better (it already is a lot better than
warren> even a few years ago), and mistakes will be less common" then
warren> there will be no need to install NTAs.

+1

Key management automation is definitely making progress. I'd love to see
the operational stability of DNSSEC approach "standard" DNS.

Expired keys and blown ZSK key rollovers are certainly common still but
as folks take advantage of better DNSSEC signing software versions, this
will lessen.

As we get vendors and registrars to implement things like RFC 7344
(delegation trust maintenance) and RFC 5011 (automated trust anchors),
KSK rollover failures will also get less common.

But we're not there yet. Until then, having a way to remediate the
occasional DNSSEC failure while keeping DNSSEC validation on helps, not
hinders, DNSSEC adoption.

---

From: "Livingood, Jason" <[email-masked]>
Date: Sun, 26 Oct 2014 13:52:33 +0000

On 10/25/14, 10:12 PM, "Warren Kumari" <[email-masked]<mailto:[email-masked]>> wrote:

On Sat, Oct 25, 2014 at 9:50 PM, Paul Hoffman <[email-masked]<mailto:[email-masked]>> wrote:
On Oct 25, 2014, at 6:43 PM, Olafur Gudmundsson <[email-masked]<mailto:[email-masked]>> wrote:
We want humans in the loop, I would love to see a twitter feed when ever Comcast does a Negative Trust Anchor.

Or both...

Warren - Your suspicions are right on the money. A good reference is http://dns.comcast.net/images/files/dnssec_validation_failure_nasagov_20120118_final.pdf. Take a look at the flak we got on page 9 – truly fascinating. In any case, posting on Twitter and our DNS website is what we have been trying to do based on how people respond. And in nearly every case I have seen so far we have had PR involved since we have usually gotten press calls about it or could expect to (in the NASA.gov example it was ironically enough MSNBC).

- Jason Livingood

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 26 Oct 2014 10:13:19 -0400

On Sun, Oct 26, 2014 at 9:52 AM, Livingood, Jason <
[email-masked]> wrote:

>
> Warren - Your suspicions are right on the money. A good reference is
> http://dns.comcast.net/images/files/dnssec_validation_failure_nasagov_20120118_final.pdf.
> Take a look at the flak we got on page 9 – truly fascinating. In any case,
> posting on Twitter and our DNS website is what we have been trying to do
> based on how people respond. And in nearly every case I have seen so far we
> have had PR involved since we have usually gotten press calls about it or
> could expect to (in the NASA.gov example it was ironically enough MSNBC).
>
>
Unless and until DNSSEC and DNS management is fully automated everywhere,
use of DNSSEC information is going to require curation.

The problem here is a very common one in security: Every security problem
is simple if you only recognize one problem. It is the need to balance
different concerns that makes security hard.

This is also why I try to resist the narrowly focused WG charters designed
to produce a quick specification that only meets half the real world
concerns.

In the real world, the goal is to be able to connect to the correct site.
This means that there are two (main) failure modes, not one:

1) Client connects to the wrong site.
2) Client is unable to connect to the correct one.

Any decision process that is going to be implemented in the end point has
to be able to respond to every set of inputs in a deterministic way. One of
the reasons the DKIM policy discussion was so nonsensical was that people
would suggest solving problem A by adopting decision strategy X and problem
B with strategy Y. But there was no deterministic means of choosing between
X and Y. So the implementations would have to work by magic.

Moving the decision point to the resolver allows a more powerful decision
strategy because a resolver can bring more information to bear. Instead of
discarding expired records, a high volume resolver can make use of them to
make better decisions. So if the DNSSEC records have expired but the A
records have not changed, keep serving them.

Negative trust anchors are one approach to curating the DNSSEC data but
there is a much larger family of possible approaches and curating DNSSEC
for A records is not the same as curating for DANE or security policy
records.

---

From: Paul Ebersman <[email-masked]>
Date: Sun, 26 Oct 2014 10:09:44 -0600

ogud> We want humans in the loop, I would love to see a twitter feed
ogud> when ever Comcast does a Negative Trust Anchor.

phoffman> Like https://twitter.com/ComcastDNS, for example? Either
phoffman> things haven't been failing much lately, or they're not
phoffman> updating it as often as we had hoped.

jlivingood> And in nearly every case I have seen so far we have had PR
jlivingood> involved since we have usually gotten press calls about it
jlivingood> or could expect to (in the NASA.gov example it was
jlivingood> ironically enough MSNBC).

There have been a bunch of staff changes and additions in the comcast
DNS group and use of dns.comcast.net and the @comcastdns twitter feed
has been neglected.

And as Jason points out, we also get to brief our PR as part of the
process.

That said, I'd like to get more consistent about using both methods for
positive and negative news about our DNS.

As for NTAs, I've been at Comcast since Feb and can only recall 2 NTAs
installed on external sites in all that time. And I'm OK with that. I
see NTA as a tool that we should try to never use but which is
invaluable when we do need it.

My agenda in pushing this draft is not to advocate wide spread use but
to guarantee that all of my vendors have an RFC to code against so that
I have consistent behavior and plenty of server choices for server
diversity.

---

From: Paul Wouters <[email-masked]>
Date: Sun, 26 Oct 2014 21:08:00 -0400 (EDT)

> familiar with operating DNS servers making any decisions.

That's my problem with the document. It describes a local policy that a
site might have. And documents three software implementations on how
to make such a negative trust anchor. Is that what an IETF document
should do?

> that my customers beat me for.

I did not get from reading the draft how I suddenly get much better
engineering contacts with big players. In fact, the draft tells me
the NTA's I create should not be distributed outside my administrative
domain.

So I'm confused. What is the goal of this document? How does it help us?

Paul

---

From: Paul Ebersman <[email-masked]>
Date: Sun, 26 Oct 2014 19:23:24 -0600

pebersman> Have you actually read through the new draft? It specifically
pebersman> prohibits automatic installation of NTAs and says that you
pebersman> should have folks familiar with operating DNS servers making
pebersman> any decisions.

pwouters> That's my problem with the document. It describes a local
pwouters> policy that a site might have. And documents three software
pwouters> implementations on how to make such a negative trust
pwouters> anchor. Is that what an IETF document should do?

And documents the reason why it's needed and where and how it does and
does not apply, and when to use them. That sounds informational to
me. It never claims that everyone will use these but does document what
how to do it responsibly if you do find it useful.

If you don't find all this in the draft, point out what's missing. If
what you are saying is that you don't find the idea of NTAs useful to
you, that's your privilege but it doesn't make it useless to others.

pebersman> That isn't painless. It means that this skips past all 1st
pebersman> tier and gets to senior engineers. Don't know about you but I
pebersman> hate getting on-call problems caused by someone else that I
pebersman> have no direct way to fix but that my customers beat me for.

pwouters> I did not get from reading the draft how I suddenly get much
pwouters> better engineering contacts with big players.

I think you're confusing the domain owner and the user of the NTA. My
statement above was that the user of the NTA needs to have engineers who
understand and run DNS servers making decisions of when to use the NTA.

It doesn't attempt to mandate what kind of engineers or contacts the
owner of the borked domain has. Sadly, if the DNSSEC validation is
broken, it's usually because they don't have experienced folks running
their DNS.

Or did I not understand your question/objection here?

pwouters> In fact, the draft tells me the NTA's I create should not be
pwouters> distributed outside my administrative domain.

Right. And you, as the person responsible for running your admin domain
should be sure that whoever is making decisions about using or not using
an NTA for some broken domain elsewhere outside your admin domain is
someone who actually understands DNS, can correctly confirm that the
domain is misconfigured, etc.

pwouters> So I'm confused. What is the goal of this document? How does
pwouters> it help us?

If you are not running a large recursive caching DNS farm, doing DNSSEC
validation and with lots of users who can/will call you when things
break, this may not be compelling to you. Not every RFC is relevent or
useful to everyone on the internet.

If you are running such a DNS farm and you haven't already had this
issue, I'd be very surprised.

---

From: Paul Ebersman <[email-masked]>
Date: Sun, 26 Oct 2014 19:36:23 -0600

The other side of this document is that there 3 of the most popular
vendors of DNS server software have this out. We as the IETF should be
explaining the tradeoffs and risks of using an NTA.

I certainly don't want enterprises, small service providers and govt
agencies that may not have senior DNS folks just putting these into
their configs with no idea of the security implications of doing so.

---

From: Paul Hoffman <[email-masked]>
Date: Sun, 26 Oct 2014 20:54:37 -0700

> to make such a negative trust anchor. Is that what an IETF document

Yes, that's a fine thing to do as compared to being silent on something that has operational impact.

> So I'm confused. What is the goal of this document? How does it help us?

The first part is stated quite well in the document. The second part is that it documents a practice that affects the operation of DNS.

--Paul Hoffman

---

From: Paul Wouters <[email-masked]>
Date: Mon, 27 Oct 2014 09:43:09 -0400 (EDT)

> The first part is stated quite well in the document. The second part is that it documents a practice that affects the operation of DNS.

Okay, if those are the goals than the document seems fine to me. I guess
I was confused about any kind of planned collecting, distributing or
"red button" type signaling methods.

Thanks,

Paul

---

From: Dan York <[email-masked]>
Date: Mon, 27 Oct 2014 14:04:37 +0000

Many others have made the points I would have made about the operational value of NTAs so I won't repeat those... but I want to just say that I think Paul Ebersman nails it here:

On Oct 26, 2014, at 12:09 PM, Paul Ebersman <[email-masked]<mailto:[email-masked]>>
 wrote:

 I see NTA as a tool that we should try to never use but which is
invaluable when we do need it.

Exactly!  Hopefully everything "just works" 99% of the time... but in the event something doesn't work right the operators have a narrow "scalpel" tool in their toolbox that they can pull out rather than resorting to more drastic measures such as, for example, disabling all DNSSEC validation.

Ideally NTAs never get used and as DNSSEC deployment moves along and DNS operators get increasingly familiar with the operational practices required of DNSSEC then the need for NTAs will eventually fade away.

Yes!   If we have an operational need to have a way to generate DNSSEC validation exceptions, let's please have *one* way that we can collectively agree upon rather than having every different operator come up with some custom mechanism that works only for them.  This will make the overall deployment that much easier if the one method is spread across multiple software vendors and systems.

My 2 cents,
Dan

P.S. Nice quote, Warren!

--
Dan York
Senior Content Strategist, Internet Society
[email-masked]<mailto:[email-masked]>   +1-802-735-1624
Jabber: [email-masked]<mailto:[email-masked]>
Skype: danyork   http://twitter.com/danyork

http://www.internetsociety.org/deploy360/

---

From: Warren Kumari <[email-masked]>
Date: Wed, 29 Oct 2014 17:11:09 -0400

Over on the BIND-Users list there is currently a discussion of
fema.net (one the "Federal Emergency Management Agency" domains)
being DNSSEC borked
(https://lists.isc.org/pipermail/bind-users/2014-October/094142.html)

This is an example of the sort of issues that an NTA could address --
I'd like to note that currently neither Google Public DNS (8.8.8.8)
nor Comcast (75.75.75.75) have put in an NTA for it, but if it were
fema.gov, and this were during some sort of national disaster in the
US, things might be different...
W

On Mon, Oct 27, 2014 at 10:04 AM, Dan York <[email-masked]> wrote:
> Many others have made the points I would have made about the operational
> value of NTAs so I won't repeat those... but I want to just say that I think
> Paul Ebersman nails it here:
>
> On Oct 26, 2014, at 12:09 PM, Paul Ebersman <[email-masked]>
>  wrote:
>
> Exactly!  Hopefully everything "just works" 99% of the time... but in the
> event something doesn't work right the operators have a narrow "scalpel"
> tool in their toolbox that they can pull out rather than resorting to more
> drastic measures such as, for example, disabling all DNSSEC validation.
>
> Ideally NTAs never get used and as DNSSEC deployment moves along and DNS
> operators get increasingly familiar with the operational practices required
> of DNSSEC then the need for NTAs will eventually fade away.
>
> Yes!   If we have an operational need to have a way to generate DNSSEC
> validation exceptions, let's please have *one* way that we can collectively
> agree upon rather than having every different operator come up with some
> custom mechanism that works only for them.  This will make the overall
> deployment that much easier if the one method is spread across multiple
> software vendors and systems.
>
> My 2 cents,
> Dan
>
> P.S. Nice quote, Warren!
>
> --
> Dan York
> [email-masked]   +1-802-735-1624
> Jabber: [email-masked]

---

From: Morizot Timothy S <[email-masked]>
Date: Wed, 29 Oct 2014 21:55:24 +0000

> US, things might be different...

If an authoritative domain (e.g. irs.gov) screwed up its delegation NS records so it effectively went dark or made some similar sort of authoritative DNS or nameserver error, we wouldn't expect the recursive, caching side to resolve those sorts of errors. The domain's DNS would simply be unavailable until they resolved their problem.

I'm not sure I understand why DNSSEC is somehow different. If a domain owner chooses to sign its authoritative zones and at some point screws up either their signing or their chain of trust, they should reasonably expect their DNS to go dark to a certain percentage of the world. (I believe in the United States currently, that's around a quarter of the population, at least according to APNIC Labs numbers. That tends to be the part of the world I watch most closely.)

I do understand the need ISPs had to manage customer perceptions, especially for the earliest adopters like Comcast. Support calls cost money and in some instances, an irate customer may choose to switch providers. That likely persists to some extent today, but with Google on board the pressure is, at least, less than it was before. And as those implementing DNSSEC validation continue to increase, that pressure will continue to drop.

Outside of the ISP early adopter use case, though, I'm not sure I understand the need for NTAs. We've had DNSSEC validation of Internet queries enabled for our enterprise since 2011. On the enterprise side, we simply explain the problem and that it's on the domain provider's end and that it's their responsibility to fix it. Until they do we won't be able to resolve their domain. We've never viewed it as our responsibility to try to fix problems on the authoritative side of DNS for domains we don't own or manage. Truthfully, we don't really encounter as many issues as we once did.

Given the limited nature of the use case, I'm not convinced it matters if there's a single specification for implementing it or not. I'm not really opposed to the idea either, nor do I have any issues with the draft. But after several years of experience without NTAs from a non-ISP perspective, I do know it hasn't been a burden or major issue.

Scott

---

From: David Conrad <[email-masked]>
Date: Wed, 29 Oct 2014 15:47:09 -0700

Tim,

On Oct 29, 2014, at 2:55 PM, Morizot Timothy S <[email-masked]> wrote:
> I'm not sure I understand why DNSSEC is somehow different.

Because folks who aren't validating see no problems, thus discouraging people from leaving validation on.  

To wit, on NANOG:

> From: Ray Van Dolson <[email-masked]>

"I saw the same errors in dnsviz, but was unsure if they were sufficient
to cause lookup failures (they were "warnings" only).

# dig @8.8.8.8 disa.mil MX +dnssec
...
I do note that once we disabled DNSSEC on our resolvers we were able to
push mail out to these domains.  May have been coincidental -- needs
further testing."

I figure it would be nice to give people the option of disabling validation for a single domain instead of turning validation off for everything.

Regards,
-drc

---

From: Mark Andrews <[email-masked]>
Date: Thu, 30 Oct 2014 09:56:49 +1100

> US, things might be different...

It is also a example of why valid whois data is required.

whois fema.net -> contact details
whois fema.gov -> a farce 

Mark

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Thu, 30 Oct 2014 10:14:27 +1100

In message <[email-masked]>, David Conrad writes:
> > If an authoritative domain (e.g. irs.gov) screwed up its delegation NS
> records so it effectively went dark or made some similar sort of
> authoritative DNS or nameserver error, we wouldn't expect the recursive,
> caching side to resolve those sorts of errors. The domain's DNS would
> simply be unavailable until they resolved their problem.
> >
> I figure it would be nice to give people the option of disabling
> validation for a single domain instead of turning validation off for
> everything.

I suspect you will find there are ways to do this in all the
validators.

BIND has had the following for ages which I know David knows.

	disable-algorithms <string> { <string>; ... };

BIND 9.11 will allow for disabling via rndc with automatic periodic
testing and re-enabling when validation of the SOA succeeds.
Validation will also be automatically re-enabled after a timer goes
off.

Mark

---

From: "Rose, Scott" <[email-masked]>
Date: Thu, 30 Oct 2014 19:43:44 +0000

On the subject of NTA's that should be there -  

Should there be text describing auto-adding of NTA's based on important domains (for the ISP/resolver's definition of important)?  So that domains that are used by low level services don't fail that also aren't normally visible to end users?  One example is nist.gov.  When nist.gov messed up and went DNSSEC BOGUS, time.nist.gov was unreachable by validating resolvers.  Unless the log files were detailed, a user may not know what is going on or that NTP is having issues.  

This could be a monitor, or a pre-loaded NTA for certain domains.  Not crazy about the pre-loaded idea, but it would avoid a period of scrambling.

Scott

> US, things might be different...

===================================
Scott Rose
NIST
[email-masked]
+1 301-975-8439
Google Voice: +1 571-249-3671
http://www.dnsops.gov/
===================================

---

From: Warren Kumari <[email-masked]>
Date: Thu, 30 Oct 2014 17:37:33 -0400

On Thu, Oct 30, 2014 at 3:43 PM, Rose, Scott <[email-masked]> wrote:
> Should there be text describing auto-adding of NTA's based on important domains (for the ISP/resolver's definition of important)?
>  So that domains that are used by low level services don't fail that also aren't normally visible to end users?  One example is nist.gov.  When nist.gov messed up and went DNSSEC BOGUS, time.nist.gov was unreachable by validating resolvers.  Unless the log files were detailed, a user may not know what is going on or that NTP is having issues.
>

Sorry, but to me this sounds like a bad idea -- you should find out
that you "not normally visible to end users" failures are happening
because your network monitoring system goes "Beep Beep Beep" when low
level important services die.
The NOC then goes off and investigates and discovers that e.g the NTP
monitor it sad because time.nist.gov is unresolvable.

At this point there really needs to be a *human* in the loop to decide
what to do, if the failure really *is* a DNSSEC failure  and, more
importantly, if installing an NTA is the right answer.
This requires a human judgment call if the outage requires serious
action. For example, for the *huge* majority of folk time.nist.gov
going unresolvable was simply not an issue -- but many years a go I
worked for a digital timestamp / nonrepudiation company, and things
failed spectacularly if *they* couldn't talk to time.nist.gov.
This is, and should be, a "Break Glass" type event, not an automated best guess.

Neither Google Public DNS, not Comcast have yet installed an NTA for fema.net...

> This could be a monitor, or a pre-loaded NTA for certain domains.  Not crazy about the pre-loaded idea, but it would avoid a period of scrambling.

I think that some scrambling is preferable to incorrectly overriding
DNSSEC validation in the case of an "actual" issue.

If the thingie you are talking to is sufficiently critical to your
operations you *really* should be monitoring it.

When the monitor fires your well trained NOC leaps into action, opens
the binder and flips to the checklist - if they discover that the
failure is truly a DNSSEC issue (e.g they call up the operators of
criticalservice.org who tells them that the HSM and hidden master are
both under 3 ft of piranha infested water) they flip to the section
marked "Installing an NTA".

This has another checklist that confirms that it really is A: an
DNSSEC failure and B: that criticalservice.org is actually critical.

Once confirming this, the NOC folder references Appendix A in
[draft-livingood-dnsop-negative-trust-anchors] and installs an NTA for
criticalservice.org. They then note this in the ticketing system, add
it to the daily NOC log and go back to playing Minecraft. The
following NOC shift has an open ticket and periodically tests if the
issue is resolved. Once it is, the NTA gets removed.

This should all be a very rare occurrence - but if, and when you need
it, you *really* need it.  Having the NTA option available in an
emergency is really useful, but it should not be used in an automated
manner. If a name is critical enough that you are considering
automatically installing an NTA there are probably 3 questions you
need to ask:
1: DNSSEC is designed to prevent MITM attacks -- if it is this
critical to me, is failing "open" the right answer?
2: If the service is *this* critical to me, perhaps I have too many
eggs in one basket and need to find a second provider?
and
3: If I'm seeing these failures often enough that I'm considering
this, have I selected the wrong supplier here?

W

---

From: Paul Ebersman <[email-masked]>
Date: Fri, 31 Oct 2014 08:26:01 -0600

srose> Should there be text describing auto-adding of NTA's based on
srose> important domains (for the ISP/resolver's definition of
srose> important)?  So that domains that are used by low level services
srose> don't fail that also aren't normally visible to end users?  One
srose> example is nist.gov. When nist.gov messed up and went DNSSEC
srose> BOGUS, time.nist.gov was unreachable by validating resolvers.

warren> Sorry, but to me this sounds like a bad idea -- you should find
warren> out that you "not normally visible to end users" failures are
warren> happening because your network monitoring system goes "Beep Beep
warren> Beep" when low level important services die.  The NOC then goes
warren> off and investigates and discovers that e.g the NTP monitor it
warren> sad because time.nist.gov is unresolvable.

warren> At this point there really needs to be a *human* in the loop to
warren> decide what to do, if the failure really *is* a DNSSEC failure
warren> and, more importantly, if installing an NTA is the right answer.

I'd hope it would be good operational sense for folks to have automated
checks of critical things and checks of DNS logs for DNSSEC validation
failures and that we shouldn't have to spell that out.

But do we want to at least have a mention of doing such kinds of checks
as a better way of noticing DNSSEC failures than pissed off customers or
puzzled NOC folks?

I do agree that we should not be inserting NTAs automatically for
anything.

---

From: Warren Kumari <[email-masked]>
Date: Fri, 31 Oct 2014 11:53:28 -0400

> as a better way of noticing DNSSEC failures than pissed off customers or

Nope -- because now you have the problem of where to draw the line. Do
we also suggest the folk monitor error rates on WAN circuits? Failing
RAID arrays? Excessive BIND memory usage?

I think that would be document creep, creep!

> I do agree that we should not be inserting NTAs automatically for

Yah.
W