Subject: Re: [DNSOP] draft-wkumari-dnsop-dist-root-01.txt

From: "yzw_iplab" <[email-masked]>
Date: Wed, 9 Jul 2014 11:30:16 +0800

Hi, all,
There are currently two solutions proposed to distrbute the DNS root service more widely.In my opinion, we should work on this issue following the two steps:
1) we should discuss their feasibility from technological aspects. the technological requirements of them should be gathered and listed ,and then analyzed one by one.  
2) these two solutions consider the similar issue from different levels: one is the recursive-level and another one is the authorative-level. (if both of them are feasible) we should figure out respective scenarios and requirements suitable for each of them.
BR,
Zhiwei Yan

---

From: Declan Ma <[email-masked]>
Date: Wed, 9 Jul 2014 13:35:36 +0800

Zhiwei,

	Your proposal seems reasonable. But we can not separate the recursive-level and authorative-level, as you call it by the way, since the DNS is an integrated one. If both of the two solutions are feasible, we need to figure out how and to what extent, both solutions could collaborate on root zone file distribution, not in the “respective scenarios” .

Declan Ma
ZDNS

在 2014年7月9日，上午11:30，yzw_iplab <[email-masked]> 写道：

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop

---

From: Paul Vixie <[email-masked]>
Date: Wed, 09 Jul 2014 00:43:16 -0700

> 	Your proposal seems reasonable. But we can not separate the recursive-level and authorative-level, as you call it by the way, since the DNS is an integrated one. If both of the two solutions are f…

in my opinion, the applicability statement of a recursive solution would
be: "if you want these benefits and can manage these risks, then you can
configure your rdns as follows". whereas the applicability statement for
an authoritative solution would be: "if you want to serve root dns
content to a loopback, lan, campus, or global network, then configure
your adns and your routing as follows."

separate from applicability, there is vision. the vision statement for
an rdns solution would be: "to allow self selected recursive dns
operators to become less dependent on the root name server system, the
following proposal is offered." whereas the vision statement for the
adns solution would be: "to better server root dns content to the
internet, the following proposal is offered."

vixie

---

From: Dan York <[email-masked]>
Date: Wed, 9 Jul 2014 12:20:05 +0000

I would suggest there is also a third angle here:

On Jul 8, 2014, at 11:30 PM, "yzw_iplab" <[email-masked]<mailto:[email-masked]>>
 wrote:

Hi, all,
There are currently two solutions proposed to distrbute the DNS root service more widely.In my opinion, we should work on this issue following the two steps:
1) we should discuss their feasibility from technological aspects. the technological requirements of them should be gathered and listed ,and then analyzed one by one.
2) these two solutions consider the similar issue from different levels: one is the recursive-level and another one is the authorative-level. (if both of them are feasible) we should figure out respective scenarios and requirements suitable for each of them.
3) we should discuss how easily the solutions can actually be *deployed* and used.

I realize this is perhaps a subset of #1, but I want to call it out specifically because this step seems to be sometimes overlooked.  If, for instance, a solution requires changes to the way stub resolvers work and requires updates to the zillions of devices out there that now provide embedded DNS resolvers, the chances of that solution being *widely* deployed are significantly less than a solution that requires changes at only, say, authoritative name servers.  Not to say the first solution *couldn't* be deployed, but we just need to be realistic up front about what it might take to get the solution out there.

My 2 cents,
Dan  (who spends his days looking at how to get DNSSEC more widely deployed)

--
Dan York
Senior Content Strategist, Internet Society
[email-masked]<mailto:[email-masked]>   +1-802-735-1624
Jabber: [email-masked]<mailto:[email-masked]>
Skype: danyork   http://twitter.com/danyork

http://www.internetsociety.org/deploy360/

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 9 Jul 2014 07:27:53 -0700

> internet, the following proposal is offered."

These statements assume that there is a need for an "authoritative solution", which has yet to be discussed. A lot of the criticism of draft-wkumari-dnsop-dist-root has come from people who seem to question the need for expanding the operational base of authoritative servers beyond what it is now (a set of 12/13 administrators, many of whom use anycast).

Given that you are one of the co-authors of draft-lee-dnsop-scalingroot, can you say why your authoritative proposal is significantly better than the current operational base?

--Paul Hoffman

---

From: Paul Vixie <[email-masked]>
Date: Wed, 09 Jul 2014 10:45:54 -0700

Paul Hoffman wrote:
> These statements assume that there is a need for an "authoritative solution", ...

no. these statements offer specific motives to specific self-identified
parties.

> ...

yes. in
<https://www.icann.org/en/system/files/files/report-21feb14-en.pdf>
(section 9.4) i wrote as follows:

<< Criticisms of the current and historical Root Name Server System
include lack of resistance to DDoS
attack, noting that even with the current wide scale anycasting by every
Root Name Server Operator,
there are still only a few hundred name servers in the world who can
answer authoritatively for the DNS
root zone. We are also concerned that reachability of the Root Name
Server System is required even for
purely local communication, since otherwise local clients have no way to
discover local services. In a
world sized distributed system like the Internet, critical services
ought to be extremely well distributed. >>

vixie

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 9 Jul 2014 11:08:46 -0700

On Jul 9, 2014, at 10:45 AM, Paul Vixie <[email-masked]> wrote:

> yes. in <https://www.icann.org/en/system/files/files/report-21feb14-en.pdf> (section 9.4) i wrote as follows:
> 
> << Criticisms of the current and historical Root Name Server System include lack of resistance to DDoS 
> attack, noting that even with the current wide scale anycasting by every Root Name Server Operator, 
> there are still only a few hundred name servers in the world who can answer authoritatively for the DNS 
> root zone. We are also concerned that reachability of the Root Name Server System is required even for 
> purely local communication, since otherwise local clients have no way to discover local services. In a 
> world sized distributed system like the Internet, critical services ought to be extremely well distributed. >>

Apologies, but that doesn't answer the question. In the face of lack of resistance to DDoS attacks, why is it better to have more *authoritative* root servers, as compared to validating recursive resolvers that have an up-to-date signed copy of the root? Similarly, for purely local communication, why is it better to have more *authoritative* root servers? The last sentence above makes good sense, but it too is not related to the number authoritative servers.

--Paul Hoffman

---

From: Paul Vixie <[email-masked]>
Date: Wed, 09 Jul 2014 11:15:51 -0700

Paul Hoffman wrote:
>> << Criticisms of the current and historical Root Name Server System include lack of resistance to DDoS attack, noting that even with the current wide scale anycasting by every Root Name Server Operator, there are still only a few hundred name servers in the world who can answer authoritatively for the DNS root zone. We are also concerned that reachability of the Root Name Server System is required even for purely local communication, since otherwise local clients have no way to discover local services. In a world sized distributed system like the Internet, critical services ought to be extremely well distributed. >>
>

my comparison of the recursive vs authoritative approach to scaling root
name service was given in the attached e-mail. --vix

Subject: Re: [DNSOP] draft-wkumari-dnsop-dist-root-01.txt
From: Paul Vixie <[email-masked]>
To: Joe Abley <[email-masked]>
Cc: dnsop WG <[email-masked]>, Paul Hoffman <[email-masked]>
Date: Sat, 05 Jul 2014 09:27:21 -0700
List-Id: IETF DNSOP WG mailing list <dnsop.ietf.org>

Joe Abley wrote:
> Hi Paul, Warren,
>
> On 4 July 2014 at 16:50:08, Paul Hoffman ([email-masked]) wrote:
>
>> Greetings. Warren and I have done a major revision on this draft, narrowing the design  
>> goals, and presenting more concrete proposals for how the mechanism would work. We welcome  
>> more feedback, and hope to discuss it in the WG in Toronto.
>
> I think there is much in the language of this draft that could be tightened up, but this is an idea for discussion so I'll avoid a pedantic line-by-line dissection. But I can give you the full pedantry if you like :-)
>
> On the pros and cons, however (crudely pasted below), see below.
>
> TL;DR: there are way more cons than pros to this proposal. The pros listed are weak; the cons listed are serious. I don't see a net advantage to the DNS (or to perceived performance of the DNS for any client) here. This proposal, if implemented, would represent non-trivial additional complexity with minimal or no benefit. I am not in favour of it, if that's not obvious.
>
> ...
>

i am +1 to joe's comments above and analysis elided. i consider joe's
response to be at least as good as the "because" i owe DRC after my
first short answer to the -00 version of this draft. however, i want to
add two other countervailing points.

first, this approach was tried in freebsd, when doug barton (then
freebsd's BIND9 maintainer) made it the default. all hell broke loose
due to changes in firewall config. debugging cost was maximized, and the
debugging skills were minimized. this config was removed with prejudice.
this experience, where freebsd users are actually among the smartest of
all dns users, should serve as a warning sign to others: "abandon hope
all ye who enter here." so, joe's recommendation that this draft be
retitled and rekerned to be
"draft-ietf-dnsop-slaving-root-on-resolvers-harmful" resonates very
strongly with me.

second, this approach asks millions or tens of millions of rdns servers
to change their config in order to actually scale the capacity of the
root. (so, high cost and low benefit, as joe said.) the reason i
proposed a radically different solution in my ICANN ITI contribution
(now described at
http://tools.ietf.org/html/draft-lee-dnsop-scalingroot-00) is to allow a
few dozen to a few hundred self selected highly skilled and motivated
network-level admins to scale the capacity of the root for everyone.
it's a plain hierarchical solution allowing the operators of any
loopback network on any host, or any LAN segment, or any campus,
corporate, or ISP network to add root name service as a local
capability; it also allows unlimited scale at the global BGP layer. it
is patterned after the success of AS112. it is apolitical since existing
rootops are also expected to participate.

vixie

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 9 Jul 2014 11:37:19 -0700

On Jul 9, 2014, at 11:15 AM, Paul Vixie <[email-masked]> wrote:

> my comparison of the recursive vs authoritative approach to scaling root name service was given in the attached e-mail. --vix

I'll take that as a "no" to you having answers to the questions about why it is better to have the additional servers be authoritative.

--Paul Hoffman

---

From: Paul Vixie <[email-masked]>
Date: Wed, 09 Jul 2014 13:47:18 -0700

> I'll take that as a "no" to you having answers to the questions about why it is better to have the additional servers be authoritative.

i don't know how to state the case more clearly. my answer is not "no"
as you surmise. the cost of the recursive solution is high and the
benefit low. the cost of the authoritative solution is low and the
benefit high. if you consider the number of operators involved, their
skill and motivation levels, and the number of affected end users -- in
each case -- you should be able to reason your way to the same conclusion.

vixie

---

From: Paul Hoffman <[email-masked]>
Date: Wed, 9 Jul 2014 13:57:39 -0700

On Jul 9, 2014, at 1:47 PM, Paul Vixie <[email-masked]> wrote:

> i don't know how to state the case more clearly. my answer is not "no" as you surmise. the cost of the recursive solution is high and the benefit low. the cost of the authoritative solution is low and the benefit high. 

a) You didn't actually calculate the cost of the recursive solution, you simply pulled a large number of servers needed out of thin air.

b) You didn't say why would need any fewer authoritative servers to get the same benefit.

The amount of benefit of either solution is being debated. However, I am not sure you can show any greater benefit to an end user of getting a validated answer for a root query with the AA bit set on vs. off.

> if you consider the number of operators involved, their skill and motivation levels, and the number of affected end users -- in each case -- you should be able to reason your way to the same conclusion.

I am not. (But I won't claim that you "should be able to reason" your way to the opposite conclusion.) You postulated a larger number of unskilled operators without any justification for the recursive servers.

If the goal is to get the correct answers for root queries closer to more users, both proposals do that.

--Paul Hoffman

---

From: Paul Vixie <[email-masked]>
Date: Wed, 09 Jul 2014 14:17:29 -0700

> a) You didn't actually calculate the cost of the recursive solution, you simply pulled a large number of servers needed out of thin air.

about 20M IP addresses answer as "open recursive". most of these are low
quality CPE and other servers without skilled operators. i don't believe
that the rdns-root proposal is aimed at this set of users, but if it is,
we've got a NOTIFY problem as well as a debugging problem.

to the best of my knowledge opendns and google dns already operate as
stealth secondaries for the root, not that they can answer AA for root
data, but that they don't need to ask a real root name server for any
TLD information, and they can generate NXDOMAIN without having to swim
upstream to get a purpose built NXDOMAIN for the root. if not, both
should. if the rdns-root proposal is aimed at this set of users, then
that's great, it just needs an applicability statement similar in
concept to the one i suggested up-thread.

the middle of the market is where we lose cognitive traction. from my
point of view this set of operators is not skilled enough, and is not
ever going to be skilled enough, to both configure and operate and audit
and debug a stealth root slave. the freebsd experiment proved this to my
satisfaction, though i was previously informed on this topic by selling
BIND support for a decade or two. but if we imagine that the skills gap
could be closed with good documentation and good software, the portion
of the internet that can be isolated from root name server reachability
errors using this approach is still very small -- and the roll out time
for vendors like nominum, microsoft, infoblox, ISC and others to inform
their customers of this option is measured in years. and the total mass
of held state in terms of software revisions, version numbers, stored
content has a multiplier in the tens of thousands if not hundreds of
thousands.

there's no thin air here.

based on the freebsd experiment with stealth root zone service, it can't
scale. (the freebsd community is Very Smart in the grand scheme of
things.) whereas based on the AS112 experience with unowned anycast,
that approach can scale -- all we needed was DNSSEC so as to erase the
temptation of local root operators to amend the zone content in any way.

> b) You didn't say why would need any fewer authoritative servers to get the same benefit.

i said why fewer authoritative servers would have a greater benefit.

> If the goal is to get the correct answers for root queries closer to more users, both proposals do that.

i'd like a small number of operators to be able to effect internet-wide
improvement, and specifically, i'd like a moderate number of edge
network operators to be able to offer root name service to their
rdns-running populations without them having to pirate the address space
of an existing root name server operator.

i want to avoid having this burden shift to every single rdns operator
who wants the benefit of better access to root zone content, because (a)
the world's supply of rdns operators motivated and skilled enough to do
that is insignificant; (b) the complexity of configuration and
monitoring and debugging will scale with the affected population; and
(c) the affected population will be extremely small compared to the size
of (for example) the affected population of the AS112 system.

so my goal is different -- both more broad and more narrow -- than the
goal you're describing.

i'll stop here. feel free to have the last word.

vixie

---

From: David Conrad <[email-masked]>
Date: Wed, 9 Jul 2014 14:38:02 -0700

Paul,

On Jul 9, 2014, at 2:17 PM, Paul Vixie <[email-masked]> wrote:
> the freebsd experiment proved this to my satisfaction,

My understanding of "the freebsd experiment" you keep referencing was that one individual on the FreeBSD core team decided to change the default on an OS distribution in a very unexpected and infrastructure impacting way without significant (any?) public notice, resulting in an 'interesting' set of surprises for folks who merely upgraded their OS.  Can you please point to your evidence that the failure of "the freebsd experiment" was driven by a lack of skill "to both configure and operate and audit and debug a stealth root slave"? 

Thanks,
-drc

---

From: Paul Vixie <[email-masked]>
Date: Wed, 09 Jul 2014 14:45:19 -0700

David Conrad wrote:
> My understanding of "the freebsd experiment" you keep referencing was that one individual on the FreeBSD core team decided to change the default on an OS distribution in a very unexpected and infrastructure impacting way without significant (any?) public notice, resulting in an 'interesting' set of surprises for folks who merely upgraded their OS.

that happened, except that the backout of this config feature was driven
not just by the experiences of those surprised by it, but also by those
who knew it was coming and knew it was happening. two wizard level
freebsd power users went so far as to describe a failure caused by the
root zone timing out due to an upstream firewall change. rather than
teach the system how to monitor and report this sort of thing, and back
out the local root zone if it went stale, there was a brief and overdue
cost:benefit analysis after which this config was backed out of freebsd
itself.

>   Can you please point to your evidence that the failure of "the freebsd experiment" was driven by a lack of skill "to both configure and operate and audit and debug a stealth root slave"? 

i did not keep notes as to who did or said what and on which dates.
perhaps doug barton could be persuaded to say more.

vixie

---

From: David Conrad <[email-masked]>
Date: Wed, 9 Jul 2014 16:11:42 -0700

On Jul 9, 2014, at 2:45 PM, Paul Vixie <[email-masked]> wrote:
> rather than teach the system how to monitor and report this sort of thing, and back out the local root zone if it went stale, there was a brief and overdue cost:benefit analysis after which this config was backed out of freebsd itself.

My takeaway from this was that (a) if you're going to slave the root, it is you who should configure it, not have it magically appear one day without your knowledge and (b) if the zone transfer cannot succeed (for whatever reason), you need to fall back to the legacy roots (screaming as you do so).

In the draft, (b) is already explicitly addressed (section 2) while (a) is sort of implied, but not explicitly stated (that should probably be fixed).

> i did not keep notes as to who did or said what and on which dates. perhaps doug barton could be persuaded to say more.

I just waded through the discussion on the dns-operations and freebsd-current mailing lists during "the freebsd experiment" timeframe (see 
https://lists.dns-oarc.net/pipermail/dns-operations/2007-July/001803.html and http://lists.freebsd.org/pipermail/freebsd-current/2007-August/075813.html).  My reading was that the vast majority of complaints were inline with (a) above (and the variation that it was rude for FreeBSD to insert a dependency for zone transfer from root servers without asking them first).

I did not see any evidence that the rejection of "the FreeBSD experiment" was driven by a lack of skill "to both configure and operate and audit and debug a stealth root slave." In fact, a number of folks appeared to indicate it was an interesting idea worth exploring.

Regards,
-drc

---

From: Warren Kumari <[email-masked]>
Date: Wed, 9 Jul 2014 19:33:32 -0400

On Wed, Jul 9, 2014 at 5:17 PM, Paul Vixie <[email-masked]> wrote:
> i don't know how to state the case more clearly. my answer is not "no" as
> you surmise. the cost of the recursive solution is high and the benefit low.
> the cost of the authoritative solution is low and the benefit high.
>
> a) You didn't actually calculate the cost of the recursive solution, you
> simply pulled a large number of servers needed out of thin air.
>
>
> stealth secondaries for the root, not that they can answer AA for root data,
> but that they don't need to ask a real root name server for any TLD
> information, and they can generate NXDOMAIN without having to swim upstream
> to get a purpose built NXDOMAIN for the root. if not, both should. if the
> rdns-root proposal is aimed at this set of users, then that's great, it just
> needs an applicability statement similar in concept to the one i suggested
> up-thread.
>
> the middle of the market is where we lose cognitive traction. from my point
> of view this set of operators is not skilled enough, and is not ever going
> to be skilled enough, to both configure and operate and audit and debug a
> stealth root slave.

True -- which is why this isn't a stealth root slave. "Configuring"
this involves the operator turning on or off a single option -- "an
example name for this option could be "transfer-and-validate-root
[yes|no]"".
If there is a failure (for example, the firewall admin decides that
all DNS is UDP53 and so it is unable to transfer / retransfer), the
solution fails back to legacy operation instead of returning SERVFAIL.

> though i was previously informed on this topic by selling BIND support for a
> decade or two.

Yup. This is different to the FreeBSD experiment - failures result in
messages in your log file, not inability to resolve.

> but if we imagine that the skills gap could be closed with
> good documentation and good software, the portion of the internet that can
> be isolated from root name server reachability errors using this approach is
> still very small -- and the roll out time for vendors like nominum,
> microsoft, infoblox, ISC and others to inform their customers of this option
> is measured in years.

Yup. 'm fine with years. No one said we needed this tomorrow....

> and the total mass of held state in terms of software
> revisions, version numbers, stored content has a multiplier in the tens of
> thousands if not hundreds of thousands.
>
> scale. (the freebsd community is Very Smart in the grand scheme of things.)

This is different to the FreeBSD experiment -- both in terms of the
complexity of enabling it, and the failure modes when things end
poorly.

> whereas based on the AS112 experience with unowned anycast, that approach
> can scale --

Errrr... how many AS112 nodes are there? The 2012 survey discovered 72
(https://www.as112.net/2012-survey.html) -- there could be more, there
could be less. Screwing up the dist-root solution (not quite sure how,
it's designed to fail open, but I'm sure someone will manage) will
affect those folk using this resolver. Screwing up an AS112 type
solution screws up the folk using that resolver, *and* everyone who
listens to the BGP route that will inevitably leak...

> all we needed was DNSSEC so as to erase the temptation of local
> root operators to amend the zone content in any way.

Yup. In either solution we need DNSSEC to prevent folk from futzing
with the zone contents.

> improvement, and specifically, i'd like a moderate number of edge network
> operators to be able to offer root name service to their rdns-running
> populations without them having to pirate the address space of an existing
> root name server operator.
>
> i want to avoid having this burden shift to every single rdns operator who
> wants the benefit of better access to root zone content, because (a) the
> world's supply of rdns operators motivated and skilled enough to do that is
> insignificant; (b) the complexity of configuration and monitoring and
> debugging will scale with the affected population; and (c) the affected
> population will be extremely small compared to the size of (for example) the
> affected population of the AS112 system.
>
> so my goal is different -- both more broad and more narrow -- than the goal
> you're describing.
>