Subject: [DNSOP] call to work on edns-client-subnet

From: Joe Abley <[email-masked]>
Date: Tue, 6 May 2014 13:18:50 -0400

Hi all,

I'm seeing increasing discussion about edns-client-subnet (most recently documented, I think, in the expired document draft-vandergaast-edns-client-subnet-02), both in commercial and open source venues (there's an active thread on the unbound-users mailing list right now, for example).

Google DNS supports edns-client-subnet, which by recent GIH+GGM count means 10%+ of all client queries now trigger queries to authority servers with that option included.

On the authority side, support for this option has a potential impact on query load. On the recursive side, support for this option has a potential impact on cache size.

With multiple implementations, there are interop issues.

If I recall the history of draft-vandergaast-edns-client-subnet-02, it stalled because various persuasive people in IETF working groups reacted to the vomity taste it left in their mouths (by which I refer to the concept, not the quality of the documentation). I may well have been one of them.

However, I now feel that regardless of any vomity taste, what we are looking at is a proposal that has been implemented in multiple code bases (and hence must interoperate), has seen significant deployment and the use of which has operational consequences. Both the protocol changes and the impact on operations should be documented.

I think dnsop should pick up some or all of this work. I think not picking up this work will result in implementation and operational problems. (I am reminded of the consequences of not standardising NAT, for example.)

I would be happy to contribute reviews and/or text.

Thoughts?

Joe

---

From: Tim Wicinski <[email-masked]>
Date: Tue, 06 May 2014 13:25:04 -0400

> I would be happy to contribute reviews and/or text.

Joe,

The Chairs have been discussing this as recently as yesterday, with the 
advent of what appears to be the acceptance of our new charter.  We were 
going to reach out and restart that discussion.

Our initial thought was that with all the EDNS extensions that once they 
are given their IANA codepoint, any documentation that can be provided 
is an added bonus.   We were thinking that these would be relatively 
easy to move these along as 'Informational' level and not 'Standards 
Track'.  We're wondering what the working group thinks.

The Chairs were waiting to bring this once some of the older work has 
moved through our system, to not overwhelm the members.

(This also includes Paul's drafts which also need a more formal 
conversation).

tim

---

From: Doug Barton <[email-masked]>
Date: Tue, 06 May 2014 10:27:42 -0700

On 05/06/2014 10:18 AM, Joe Abley wrote:
> I think not picking up this work will result in implementation and operational problems.

Those of us who still have the vomity taste would like the problems with 
interop and implementation to continue.

Just because we _can_ make something work better doesn't mean we _should_.

Doug

---

From: Joe Abley <[email-masked]>
Date: Tue, 6 May 2014 13:30:59 -0400

On 6 May 2014, at 13:27, Doug Barton <[email-masked]> wrote:

> Those of us who still have the vomity taste would like the problems with interop and implementation to continue.
> 

As a matter of principle, I don't think people who never plan to use an extension should stand in the way of those who do. The IETF doesn't exist to stop people doing things.

(And again, see NAT.)

Joe

---

From: David Conrad <[email-masked]>
Date: Tue, 6 May 2014 11:00:03 -0700

> As a matter of principle, I don't think people who never plan to use an extension should stand in the way of those who do. The IETF doesn't exist to stop people doing things.

+1 (actually, +∞, but not sure that's interoperable)

Regards,
-drc

---

From: Paul Wouters <[email-masked]>
Date: Tue, 6 May 2014 14:08:14 -0400 (EDT)

> I would be happy to contribute reviews and/or text.

I reluctantly agree, and I'm willing to review and contribute as well.
This has to be handled carefully to avoid privacy/tracking issues,
but clearly we also want to stimulate a more de-centralised resolver
infrastructure that can still deal with geolocation and CDN's.

Paul

---

From: Ralf Weber <[email-masked]>
Date: Tue, 6 May 2014 20:09:32 +0200

Moin!

> With multiple implementations, there are interop issues.
Probably, but we don't know as so far everybody implementing has implemented both sides of it and used that. The only "interop" experiences I know of where researchers using the option to find out about CDNs utilising it.

[...]

> I would be happy to contribute reviews and/or text.
Fully agree.

So long
-Ralf

---

From: Colm MacCárthaigh <[email-masked]>
Date: Tue, 6 May 2014 14:51:56 -0700

On Tue, May 6, 2014 at 10:18 AM, Joe Abley <[email-masked]> wrote:

> On the authority side, support for this option has a potential impact on
> query load. On the recursive side, support for this option has a potential
> impact on cache size.
>

Just to add some limited data; CloudFront (a large CDN) has been using
EDNS0 client subnet for a few months now, and publically announced a month
ago. In general, the uptick on the authority side has been surprisingly
modest.

> With multiple implementations, there are interop issues.

We've noticed some inconsistency around the subnet lengths being required
in responses, but nothing unmanageable. In general, it's been pretty smooth!

The biggest operational problem is probably a lack of support in diagnostic
tools, with an RFC, I'm hopeful we could get a patch into the standard
version of dig - which would be useful for debugging and so on.

There might also be a place for an informational document/rfc on
source-dependent answers in general. For example, even for those who
believe that source dependent answers has a place, having source-dependent
NS, DS records or delegation paths can be just plain unworkable.

-- 
Colm

---

From: "Jiankang Yao" <[email-masked]>
Date: Wed, 7 May 2014 09:04:32 +0800

Section 3.1.1.  Responses Tailored to the Originator in the draft-iab-dns-applications-07 
has some related discussion to this topic.

From the IAB draft, it seems that IAB does not prefer to tailor dns response based on the originator.

Jiankang Yao

From: Joe Abley
Date: 2014-05-07 01:18
To: DNSOP WG
Subject: [DNSOP] call to work on edns-client-subnet
Hi all,

Thoughts?

Joe
_______________________________________________
DNSOP mailing list
[email-masked]
https://www.ietf.org/mailman/listinfo/dnsop

---

From: Colm MacCárthaigh <[email-masked]>
Date: Tue, 6 May 2014 18:14:32 -0700

On Tue, May 6, 2014 at 6:04 PM, Jiankang Yao <[email-masked]> wrote:

>   Section 3.1.1<http://tools.ietf.org/html/draft-iab-dns-applications-07#section-3.1.1>.
> Responses Tailored to the Originator in the draft-iab-dns-applications-07

3.1.1 reads pretty neutral to me, even saying that it "introduces little
harm" (for web portals) and that it has broad adoption in the field.  It
just notes that it doesn't have much support in the community.

But it clearly has broad support on the internet. At this point a majority
of DNS responses are likely based on the originator (that's my guess based
on local data, but it'd be interesting to see real data).

-- 
Colm

---

From: "Jiankang Yao" <[email-masked]>
Date: Wed, 7 May 2014 09:44:30 +0800

One view about this issue based on the previous discussion years ago is that the dns implementors may choose to tailor the dns response in their own way, but ietf is unlikely to standardize it.

Jiankang Yao

From: Colm MacCárthaigh
Date: 2014-05-07 09:14
To: yaojk
CC: Joe Abley; dnsop
Subject: Re: [DNSOP] call to work on edns-client-subnet

3.1.1 reads pretty neutral to me, even saying that it "introduces little harm" (for web portals) and that it has broad adoption in the field.  It just notes that it doesn't have much support in the community. 

But it clearly has broad support on the internet. At this point a majority of DNS responses are likely based on the originator (that's my guess based on local data, but it'd be interesting to see real data).

-- 
Colm

---

From: Doug Barton <[email-masked]>
Date: Tue, 06 May 2014 19:34:20 -0700

> As a matter of principle, I don't think people who never plan to use an extension should stand in the way of those who do. The IETF doesn't exist to stop people doing things.

Would you feel the same way about a protocol that made it easier for the 
NSA to trace users? Or make it easier for China to firewall the West? Or 
for some other theoretical government to hunt down and kill dissidents 
that post criticisms of that government?

You could say that I'm arguing 'ad absurdum' here, but I'm not. There 
really are such things as bad ideas, and it's perfectly reasonable for 
the IETF to decide that something is a bad idea, and shouldn't be done. 
Or at least, shouldn't be made easier to do.

> (And again, see NAT.)

So NAT is an interesting case, since there's no doubt that the IETF 
dropped the ball on that. But the problem there was not that the IETF 
chose not to act in order to not support NAT, the problem there was that 
the collective decision process failed by determining that NAT was a bad 
idea.

The remedy to that error is not to swing the pendulum all the way in the 
other direction, and support every idea no matter how bad. The answer is 
to make better decisions.

Doug

---

From: Jewforice <[email-masked]>
Date: Wed, 7 May 2014 10:45:16 +0800

Tencent has been supporting edns client subnet for nearly a year . On the authority side , we benifit so much from ECS for the extra accuracy  of user geo-location identification ,which helps a lot for our users' network access optimization and our access traffic scheduling at a totally tolerable increased query load. The shift process is smooth and nothing goes wrong in the production environment .

According to my operation experience with ECS, I think ECS is good for Internet Content Providers, CDN vendors and cloud service providers who schedule their access traffic based on DNS resolution and making ECS a standard track may help them a lot to guide the implementation for both the authority side and the recursive side.

Weijian Liao

> 在 2014年5月7日，5:51，Colm MacCárthaigh <[email-masked]> 写道：
> 
> 
> Just to add some limited data; CloudFront (a large CDN) has been using EDNS0 client subnet for a few months now, and publically announced a month ago. In general, the uptick on the authority side has been surprisingly modest. 
>  
> We've noticed some inconsistency around the subnet lengths being required in responses, but nothing unmanageable. In general, it's been pretty smooth!
> 
> The biggest operational problem is probably a lack of support in diagnostic tools, with an RFC, I'm hopeful we could get a patch into the standard version of dig - which would be useful for debugging and so on.
> 
> There might also be a place for an informational document/rfc on source-dependent answers in general. For example, even for those who believe that source dependent answers has a place, having source-dependent NS, DS records or delegation paths can be just plain unworkable. 
> 
> 
> -- 
> Colm

---

From: Edward Lewis <[email-masked]>
Date: Wed, 7 May 2014 10:04:13 -0400

> One view about this issue based on the previous discussion years ago is that the dns implementors may choose to tailor the dns response in their own way, but ietf is unlikely to standardize it.

At the risk of repeating an unpopular opinion, what’s the purpose of the standards?  Is it to record the way in which the Internet is working (code before words) or to mandate a particular way of operating (words before code)?

In recent years I’ve seen much more talk on DNS (industry) operations in DNS-OARC than inside the IETF.

One can ask - what makes something worthy of standards track (as opposed to informational)?  Depending on that criteria, the likelihood of the IETF standardizing client-subnet can be debated, but I have never seen such criteria.  And "not seeing it" might be because I haven’t looked hard enough.

---

From: Stephane Bortzmeyer <[email-masked]>
Date: Wed, 7 May 2014 16:11:26 +0200

On Wed, May 07, 2014 at 10:04:13AM -0400,
 Edward Lewis <[email-masked]> wrote 
 a message of 105 lines which said:

> to record the way in which the Internet is working 

This is not "standards", it is journalism :-)

If I were to document the way the Internet is really working, the RFC
would be full of "do not forget to insert bugs here", "please do not
document" and "throw a dice before making a choice".

---

From: Edward Lewis <[email-masked]>
Date: Wed, 7 May 2014 10:17:29 -0400

> document" and "throw a dice before making a choice".

I have ask for clarity: What do you consider “best current practices”?

…and...What do you call a situation in which major players in the industry want to interoperate but do not want to illegally collude?  Bugs and unintended consequences aside, what would you call documenting a business arrangement that anyone can be party to if they desire?

---

From: Jim Reid <[email-masked]>
Date: Wed, 7 May 2014 15:42:32 +0100

> document" and "throw a dice before making a choice".

Surely the experience of getting DNSSEC out the door showed that quite a few earlier DNS RFCs had those attributes even if they weren't explicitly labelled as such? :-)

---

From: Paul Wouters <[email-masked]>
Date: Wed, 7 May 2014 11:38:11 -0400 (EDT)

On Tue, 6 May 2014, Doug Barton wrote:

> So NAT is an interesting case, since there's no doubt that the IETF dropped 
> the ball on that. But the problem there was not that the IETF chose not to 
> act in order to not support NAT, the problem there was that the collective 
> decision process failed by determining that NAT was a bad idea.

The collective decision had the right outcome. NAT is bad - don't do it.
It is however just like climate chance - those doing it don't care about
the fall out and aren't forced the pay the price of the problems they
cause.

The sheer amount of protocol workaround for not having a peer-to-peer
internet anymore is a huge cost that everyone collectively bears just
because a few players wanted a cheaper internet method that has caused
great pollution.

> other direction, and support every idea no matter how bad. The answer is to 
> make better decisions.

The problem is not the IETF. The problem is capitalism making decisions.
Look at the IPv4 to IPv6 transition. I don't think the IETF made a bad
choice. They gave everyone over a decade to work things out. Capitalism
doesn't care. IPv6 was too expensive until it was a requirement. That's
also why NATs came into existence.

As for DNS, I do like that people can use random DNS resolvers out on
the internet (and hopefully securely and privately soon as well). The
edns-subnet option is a decent compromise in revealing rough locations
for a large geographic region. I am still a little fearful of abuse, but
that same abuse would happen if I queried using my own validating DNS
resolver on my mobile device, except they would use the exposed IP
address directly.

Paul

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 May 2014 12:36:18 -0400

On 6 May 2014, at 22:34, Doug Barton <[email-masked]> wrote:

> You could say that I'm arguing 'ad absurdum' here, but I'm not. There really are such things as bad ideas, and it's perfectly reasonable for the IETF to decide that something is a bad idea, and shouldn't be done. Or at least, shouldn't be made easier to do.

Consider the two possible outcomes:

(a) use of edns-client-subnet effectively involves a large depth of undocumented experience and knowledge about specific implementations and where those specific implementations are used. Use of the option is constrained to applications supported by big money or big government, since no individual (e.g. unfunded, open source) implementer can realistically hope to understand such a moving target with accuracy. The extent to which end-user privacy is affected in the big picture is difficult to characterise since the landscape is so fluid.

(b) use of edns-client-subnet is documented, oddities that come up following implementation are rolled into the documentation, and we have a stable resource that exactly describes how the option works against which interop testing has half a chance of bearing fruit, and using which privacy implications can be easily understood.

I think (b) is preferable to (a).

>> (And again, see NAT.)
> 
> So NAT is an interesting case, since there's no doubt that the IETF dropped the ball on that. But the problem there was not that the IETF chose not to act in order to not support NAT, the problem there was that the collective decision process failed by determining that NAT was a bad idea.

NAT *is* a bad idea. And the amount of global effort required to work around the differences in every implementation is absurd, now that it has become a de-facto implementation standard in IPv4 networking.

> The remedy to that error is not to swing the pendulum all the way in the other direction, and support every idea no matter how bad. The answer is to make better decisions.

The IETF has documented lots of protocols that nobody uses. Those are, by reasonable measure of uptake, bad protocols. The IETF is not the packet police. De-centralisation of innovation is what led to the phone network becoming an Internet application, rather than the other way round.

"The mission of the IETF is to make the Internet work better by producing high quality, relevant technical documents that influence the way people design, use, and manage the Internet."

Joe

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 7 May 2014 12:47:26 -0400

On Wed, May 07, 2014 at 12:36:18PM -0400, Joe Abley wrote:
> 
> (a) use of edns-client-subnet effectively involves a large depth of undocumented experience and knowledge about specific implementations and where those specific implementations are used.

> NAT *is* a bad idea. And the amount of global effort required to work around the differences in every implementation is absurd, now that it has become a de-facto implementation standard in IPv4 net…

Indeed, just to emphasise what Joe is saying, there were so many
different ways to do NAT things that once the IETF finally decided
that it needed to cope with the actual deployed Internet, we had to
have a whole WG (BEHAVE) to figure out how everything worked, and
write that down.  Understanding the DNS is already hard enough without
making even more of it a mysterious arcane topic that you can only
learn about by hanging out on secret-handshake mail lists.

Moreover, we edns0-client-subnet has a code point in the EDNS0 OPT
registry.  Doug's argument seems to be, "Let's have that code point
and let it be mysterious."  I think that would be a perverse outcome.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: P Vixie <[email-masked]>
Date: Wed, 07 May 2014 19:06:34 +0200

I think if we want good engineering then we should recommend on host or on net validating resolvers.

I think if we want interoperability then we have to standardize anything anybody is doing.

If ietf documents client-subnet then it should be an FYI. That's hardly a death sentence... Look what happened with SRV.
-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 7 May 2014 13:09:26 -0400

On Wed, May 07, 2014 at 07:06:34PM +0200, P Vixie wrote:

> If ietf documents client-subnet then it should be an FYI. 

Can't do that.  https://tools.ietf.org/html/rfc6360, "Conclusion of FYI RFC Sub-Series".

A

-- 
Andrew Sullivan
[email-masked]

---

From: P Vixie <[email-masked]>
Date: Wed, 07 May 2014 19:12:21 +0200

Ouch. Well so if a large body of ietf participators think wide area rdns is a bad idea and that this option should never be recommended then we would presumably have to say so in the document which standardized the option. Strange.

---

From: Suzanne Woolf <[email-masked]>
Date: Wed, 7 May 2014 13:13:08 -0400

This sounds to me like a) support for working on edns-client-subnet (and possibly things like it in the future), with b) a resulting RFC as "Informational".

I've found this discussion very helpful in solidifying the thoughts Tim already wrote about, particularly with regards to carrying out our new charter. Thank you all.

best,
Suzanne

On May 7, 2014, at 1:06 PM, P Vixie <[email-masked]> wrote:

> Sent from my Android device with K-9 Mail. Please excuse my brevity._______________________________________________
> DNSOP mailing list
> [email-masked]

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 May 2014 13:15:04 -0400

> Ouch. Well so if a large body of ietf participators think wide area rdns is a bad idea and that this option should never be recommended then we would presumably have to say so in the document which…

I think documenting a bad idea and including text that describes why it is bad is better than ignoring it.

I'm not saying that I think edns-client-subnet is necessarily bad; I'm observing that others here believe it is. Even if we all thought it was definitively bad, given that it has been implemented I am still strongly in favour of documenting it.

Joe

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 7 May 2014 13:18:14 -0400

> Ouch. Well so if a large body of ietf participators think wide area rdns is a bad idea and that this option should never be recommended then we would presumably have to say so in the document which…

No, Informational status is still available.  There's nothing wrong with that.

Also, however, it seems to me that even if this went up on the
Standards track, one wouldn't have to say whether it was a good idea.
But you _could_ write a separate doc (and try to get it published or
else publish it on the Independent stream) that said, "Wide Area
Recursive DNS Considered Harmful."  I think that's a separate question
from, "How to deliver topological information from a recursive server
to an authoritative?"

A

-- 
Andrew Sullivan
[email-masked]

---

From: P Vixie <[email-masked]>
Date: Wed, 07 May 2014 19:23:30 +0200

Joe... To clarify... Client subnet is not what I an complaining about. It's wide area rdns itself that I think is a bad idea. One reason wide area rdns is a bad idea is that it needs client subnet options.

Centralized rdns is not necessary and it makes the internet brittle. Better alternatives exist. The architecture of DNS assumes localized rdns. If we're going to document client subnet then all that advice will have to go into it.

On May 7, 2014 7:15:04 PM CEST, Joe Abley <[email-masked]> wrote:
>> Ouch. Well so if a large body of ietf participators think wide area
>rdns is a bad idea and that this option should never be recommended
>then we would presumably have to say so in the document which
>standardized the option. Strange.
>
>I'm not saying that I think edns-client-subnet is necessarily bad; I'm
>observing that others here believe it is. Even if we all thought it was
>definitively bad, given that it has been implemented I am still
>strongly in favour of documenting it.
>
>
>Joe

---

From: P Vixie <[email-masked]>
Date: Wed, 07 May 2014 19:26:51 +0200

The architectural context of a feature should not be divorced from its specification. RFC is an imprimatur. With great power comes great responsibility.

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 7 May 2014 10:48:20 -0700

> Centralized rdns is not necessary and it makes the internet brittle. Better alternatives exist. The architecture of DNS assumes localized rdns. If we're going to document client subnet then all tha…

Not necessarily.  "centralized" is often really "anycast".  

E.g. if you look at Comcast there are multiple anycast responders in their own internal network for 75.75.75.75. Likewise, '8.8.8.8' is insanely anycasted.  This is not brittle, but remarkably robust.

In this case, still, edns client subnet is very useful.  It is, frankly, a mess to map "client subnet to recursive resolver", but it is an insanely powerful optimization when you can.  

edns_client_subnet makes this mapping trivial, and therefore acts to significantly improve end user performance.

--
Nicholas Weaver                  it is a tale, told by an idiot,
[email-masked]                full of sound and fury,
510-666-2903                                 .signifying nothing
PGP: http://www1.icsi.berkeley.edu/~nweaver/data/nweaver_pub.asc

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 7 May 2014 14:40:31 -0400

Dear Uncle Ben,

> The architectural context of a feature should not be divorced from its specification. RFC is an imprimatur. With great power comes great responsibility.

I disagree with this point of view.  I see nothing at all wrong with
one thing that states clearly and precisely how a feature works, and
another one that says, "Here is the wider environment in which
such-and-thus thing works."  Even the DNS protocol itself has this
separation of duties, between 1034 and 1035 (though heaven knows the
actual protocol specification could use some attention).

If this were not the case, then in fact we'd have had to discuss the
entire architectural context of any DNS feature.  It only seems like
the DNS RFCs are infinitely long.

Best,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Wed, 7 May 2014 13:50:04 -0500

> Centralized rdns is not necessary and it makes the internet brittle. Better alternatives exist. The architecture of DNS assumes localized rdns. If we're going to document client subnet then all tha…

While I am sure many readers will sympathize with your point about centralized rdns, it's worth noting that it's equally true that relying on topology to determine the answer that the resolver gives to certain queries is also brittle.  The "architectural assumption" you are claiming is not actually an architectural assumption in the DNS protocol, but rather an assumption commonly made by a particular (admittedly fairly important) set of publishers of DNS data, because it is convenient to do so.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 07 May 2014 15:37:45 -0700

> Dear Uncle Ben,

keep it civil, please.

> such-and-thus thing works."

let me show you something wrong with that, then.

http://www.cisco.com/c/en/us/products/collateral/wireless/5700-series-wireless-lan-controllers/data_sheet_c78-722607.html

an rfc is seen by sellers and buyers as an unalloyed good. we can (and i
do) criticize uneducated people making appeals to authority, but that's
a far cry from pretending it won't happen or that any undesirable result
from its happening is not the responsibility of those who write and edit
and approve RFC documents.

the current thrust is to move from "ietf should do good engineering" to
"ietf should document anything that has to be interoperable", and in
that move we have to plan on being quite detailed in our "applicability
statement" sections. my draft for that section of a client subnet option
goes more or less like this:

<< APPLICABILITY STATEMENT

The method described in this document is controversial, as is the use of
wide area anycasting for Full Service Resolvers (recursive DNS servers).
The Domain Name System as originally contemplated and implemented made
the assumption that a Full Service Resolver would either share a host
with, or a LAN with, or a campus with its end-user stub resolvers. In
that scenario, the need for something like the client subnet option does
not arise. Since at the time of this writing, personal electronics are
far more complex and resource intense than a Full Service Resolver, even
when including DNSSEC validation and DNS "firewall" policy. The purpose
of this document is to describe the client subnet option so that
cooperating systems can interoperate. No recommendation is expressed or
should be implied as to whether this method should be used, or that its
antecedent, wide area anycasted Full Service Resolvers (recursive DNS
servers) should be used. >>

if i thought i could get consensus on an addition, i'd add:

<< The client subnet option is only useful when other earlier
assumptions about Internet architecture are violated. Specifically, it's
when a market driven CDN (content delivery system) desires to "tune" DNS
authority responses to control end-user web performance, is reached by
an end user who uses a market driven anycast recursive DNS service.
Neither the described authority DNS tuning nor the described anycasted
recursive DNS service are Internet architecture requirements, or even
recommendations. A web site without a CDN front end will still work, as
will a CDN who has no insight into each requestor's actual address at
the time it receives an authority DNS request. End users who use local
recursive name servers will receive faster service per query due to the
reduced round trip time (one millisecond instead of a dozen or more
milliseconds) and will be less affected by "cloud outages". Design by
"market force" can lead to unnecessary features and not always to "good
engineering". >>

obviously i'm miffed that Stupid DNS Tricks ultimately demands more
complexity for the DNS, and so on.

>   Even the DNS protocol itself has this

that's a terrible example since neither one of them talks about the
limited applicability of the other.

> the DNS RFCs are infinitely long.

that's a terrible example since any internet draft describing a
controversial technique either failed or got an applicability statement.
if you think there are cases where RFC 2136 should not be used, over and
above the IAB-demanded statement about authentication, speak on.

vixie

---

From: Paul Vixie <[email-masked]>
Date: Wed, 07 May 2014 15:54:48 -0700

> E.g. if you look at Comcast there are multiple anycast responders in their own internal network for 75.75.75.75. Likewise, '8.8.8.8' is insanely anycasted.  This is not brittle, but remarkably robu…

what i mean by brittle isn't well exemplified by the comcast approach,
since 8.8.8.8 really will be answered close to the user, and because
it's IGP anycast it really will tend to be the closest server to the
user. BGP anycast as practiced by googledns and opendns is less predictable.

however, even in the comcast case, "rdns fade" due to misconfiguration
is likely to affect large populations per "fade" event. when rdns is
on-host or on-lan or on-campus, "rdns fade" events are more localized,
and strike smaller populations. (and, the RTT tends to be more like the
1ms that's been a design assumption for a long while, vs. 10ms or longer
as in the BGP anycast case.)

of course, more rdns servers means more opportunities (per universe) for
misconfiguration. my assumption is that end users or LAN or campus
network operators will provision and monitor according to the importance
they place on this service. the number of uncontrollable risks and other
people's moving parts you depend on for wide area RDNS is where i get my
"brittle" depiction.

> edns_client_subnet makes this mapping trivial, and therefore acts to significantly improve end user performance.

services that don't use CDN, or whose CDN does not use what i call
"Stupid DNS Tricks", won't see any improvement. so, this is a problem
created by ADNS and RDNS operators going off the reservation in the
first place, and it only affects those operators and their customers.

i'm not saying we shouldn't document it in an RFC. (informational, i
guess.) non-interoperable client subnet signalling is even worse in my
view than interoperable client subnet signalling.

i'm saying it deserves a strong applicability statement, as all
controversial technologies do.

vixie

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 7 May 2014 19:17:52 -0400

Hi Paul,

> keep it civil, please.

I think I was being civil.  If you're going to quote simplistic
bromides from the pages of a comic book as the basis of your argument,
you should expect to be teased.  
 
> an rfc is seen by sellers and buyers as an unalloyed good. 

I'm not sure what you think is wrong with that page.  It is a product
page.  It tells you what it implements.  You are presumably supposed
to be an educated consumer who can make decisions about whether a
given feature is one you want.  Certainly, I do not believe everything
Cisco says, so I'm unlikely to believe them when they say, "Bees'
knees, man!  Just have some of this RFC!"

Is it the case that people often use, "It's an RFC," to be evidence
for, "It is good"?  Sure.  But first,
http://tools.ietf.org/html/rfc1796.  Second, people who are looking
for an excuse not to think do not have to work very hard at it.
Moreover, defending against the impulse you're worried about is one of
the reasons that nobody can ever get anything published around the
IETF, a problem that you have frequently complained about in public.

> "ietf should document anything that has to be interoperable"

That is at once a straw person and a false dichotomy.  First, some of
us are arguing that part of good Internet engineering is in fact to
document stuff that we know people are deploying and that needs
interoperability.  That doesn't seem like such a controversial
proposition to me, so I'd be interested to hear why you think it to be
false.  Second, I don't think anyone is arguing that the IETF should
document _anything_ that has to be interoperable -- unless by "has to"
you mean something critical enough to the Internet (such as the DNS)
that, again, I'd have thought the proposition uncontroversial.

> with, or a LAN with, or a campus with its end-user stub resolvers.

I just had a quick look through 1034 and 1035, and I do not think that
the text offers very strong evidence for your claim there.  There is
admittedly the text in section 5.3.1 of 1034, "or can centralize the
cache for a whole local network or organization."  On the other hand,
there is the text in 5.1, "A very important goal of the resolver is to
eliminate network delay and name server load from most requests by
answering them from its cache of prior results.  It follows that
caches which are shared by multiple processes, users, machines, etc.,
are more efficient than non-shared caches."  This latter text
militates toward broadening the scope of a resolver.  Since many of
the "wide area full service resolvers" are in fact anycast and
therefore topologically close, and therefore able to eliminate
"network delay and name server load", it seems to me that the
architecture is entirely ambivalent on this topic.  So I'd appreciate
a citation.  Or, you know, a draft, "Wide Area Recursive Service
Considered Icky", or something.

> controversial technique either failed or got an applicability statement.

Really?  We had a _whole working group_ whose entire output was about
NAT and its tricks.  We have another one (MIF) that has suggested all
manner of hideous hacks in order to figure out which network you're
in.  And so on.  I dearly wish you were right that controversial
techniques always fail or get an AS.  

Anyway, I think I should be embarrassed to have a meta-argument about
whether we even ought to work on a draft.  Those of us who think
edns0-client-subnet ought to be documented can work on it, so that's
what I'm off to do.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Nicholas Weaver <[email-masked]>
Date: Wed, 7 May 2014 16:19:36 -0700

On May 7, 2014, at 3:37 PM, Paul Vixie <[email-masked]> wrote:
> an rfc is seen by sellers and buyers as an unalloyed good. we can (and i do) criticize uneducated people making appeals to authority, but that's a far cry from pretending it won't happen or that any undesirable result from its happening is not the responsibility of those who write and edit and approve RFC documents.
> 
> the current thrust is to move from "ietf should do good engineering" to "ietf should document anything that has to be interoperable", and in that move we have to plan on being quite detailed in our "applicability statement" sections. my draft for that section of a client subnet option goes more or less like this:
> 
> The method described in this document is controversial, as is the use of wide area anycasting for Full Service Resolvers (recursive DNS servers)...

So what if its controversial.  Global warming is "controversial".  Doesn't mean it isn't happening.  And it doesn't mean that "controversial" doesn't mean "shouldn't be done".

Giving out different DNS results to different clients is a behavior that is not only well established, but is indeed a good idea.  If I have a server in California, and a server in China, I'm going to want my US visitors to go to California, while my asian visitors to China.  

And anycasted recursive resolvers are a good idea:  It allows you to select a DNS provider independent the one forced upon you by the DHCP server (which, with far too many ISPs, is a server which outright lies), and have good performance from that DNS provider.

And to Paul's comment to me:

> services that don't use CDN, or whose CDN does not use what i call "Stupid DNS Tricks", won't see any improvement. so, this is a problem created by ADNS and RDNS operators going off the reservation in the first place, and it only affects those operators and their customers.

Yes, such systems don't see a benefit.  

But its an option that they can conveniently and safely ignore, so they don't see a cost either.  And, no doubt, you will ensure that Bind will never, ever implement it.  Who cares?  As long as the servers properly handle unknown options (which is something that any server MUST do), this has no effect.

As for your "rdns fade" argument: Its actually the opposite.  

By being centralized, things are more likely to get fixed, which means if you assume error rate per server is the same (and its not, Google runs a much better DNS resolver than 90% of those who instal Bind as a recursive resolver), being big means it actually gets fixed.

If random local campus resolvers have issues with a domain name, it won't get fixed, because the domain owner doesn't know about it, and the client sees just a failure.  

But, if you monitor the Google Public DNS list, you see that resolution failures on the 'big, central point of failure' actually get addressed.

Likewise, you see that the "Broken DNSSEC" self-DOS generally doesn't affect users using Comcast DNS or Google Public DNS, even though they are validating servers, because for those servers, the DNSSEC error gets noticed and a temporary override gets put in.  Lets see this happen on a world full of small networks running Bind with DNSSEC validation turned on...

But, in the end, it doesn't matter.  

The big DNS operators who would benefit from edns_client_subnet have just gone ahead and done it.  It works, its running today, its interoperable (we actually have a python server that supports it even), and it doesn't matter what the IETF thinks.  

If you want something implemented, do it.  If you want to argue about it, with people who object to it on philosophical grounds of enabling "stupid DNS tricks", write an RFC.

This discussion is just one more datapoint in the set of "This is why the IETF is where protocols go to die".

---

From: Danny McPherson <[email-masked]>
Date: Wed, 7 May 2014 23:53:39 -0400

> I've found this discussion very helpful in solidifying the thoughts Tim already wrote about, particularly with regards to carrying out our new charter. Thank you all.

I support publication of this “stupid DNS trick” as an RFC .. of some sort, particularly given the breath of deployment.  That said, I’ve seen some of the warts and am sympathetic to Paul’s aging architectural concerns and I think some review from the abundance of DNS experts here will likely serve it well :-)

As for the publication track here, or any other [E]DNS extensions similar application, whether deemed unorthodox or unfashionable, I don’t think there should be blanket discretion and edict by the chairs, we have processes that say what can be published on Experimental, Informational, or Standards Track, IIRC…

Then again, perhaps DNSOP is not the place to document deployed and operational DNS stuff and we should revive DNSEXT to discussed operationally deployed DNS stuff, particularly given the charter discussions as of late.  Or, um..

-danny

---

From: Tony Finch <[email-masked]>
Date: Thu, 8 May 2014 16:09:46 +0100

Barry Margolin pointed out an amusing interaction between two stupid DNS
tricks on the bind-users list:
https://lists.isc.org/pipermail/bind-users/2014-May/093171.html

If you have an authoritative server with "ANAME" or "CNAME flattening"
support, and the target of the ANAME is a CDN that does source-based
answer selection, then the synthetic A / AAAA records will be based on the
auth server address rather than the client address, unless you have
some special arrangement between the auth server and the CDN like
edns-client-subnet support.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Tyne, Dogger: South or southwest 4 or 5, occasionally 6 later. Slight or
moderate. Rain then showers. Good, occasionally poor.

---

From: Paul Vixie <[email-masked]>
Date: Thu, 08 May 2014 08:47:34 -0700

> edns-client-subnet support.

madness. this way lies madness. the dns design had moving parts and
nonmoving parts. the dns implementation is becoming something else entirely.

see also "What DNS Is Not" <https://queue.acm.org/detail.cfm?id=1647302>.

vixie

---

From: Ralf Weber <[email-masked]>
Date: Thu, 8 May 2014 18:15:28 +0200

Moin!

> nonmoving parts. the dns implementation is becoming something else entirely.
There is madness, but the madness is in mixing authoritative and recursive functions in one server and not in using DNS to direct traffic. After all that's what all lookups do, give you an IP address you connect to.

All of this also is secondary to edns-client-subnet, which is something we should work on IMHO.

So long
-Ralf

---

From: Paul Vixie <[email-masked]>
Date: Thu, 08 May 2014 09:17:43 -0700

Ralf Weber wrote:
> ...
> There is madness, but the madness is in mixing authoritative and recursive functions in one server and not in using DNS to direct traffic.

while i'm on record has holding that view, it turns out that RFC 1035
does describe recursion and authority as co-residing in a server. so
while this is in my view a dangerous practice and a bad idea, it's well
supported by the scriptures.

> After all that's what all lookups do, give you an IP address you connect to.

i don't think so. dns lookups have many purposes unrelated to returning
IP addresses. i'd like to see 100 more things like SSHFP this decade.

vixie

---

From: Christopher Morrow <[email-masked]>
Date: Thu, 8 May 2014 12:32:36 -0400

#1 - support doing the work to finalize the edns-client-subnet standard.

now... (I hope my inline response is accepted by the readers of this
wg's list, I would note that someone's quoting is all jacked up... oh
well)

<pull up waders>

On Thu, May 8, 2014 at 12:17 PM, Paul Vixie <[email-masked]> wrote:
>
>
> Ralf Weber wrote:
>
> ...
>
> There is madness, but the madness is in mixing authoritative and recursive
> functions in one server and not in using DNS to direct traffic.
>

It seems, to me at least, that a bunch of the problems with dns
'tricks' which are more than: "Oh good, my zone loads! Lookey, I have
an A record! what is an A record again?"

are that folk should not play with sharp knives if they aren't
prepared to get cut occasionally. Ideally you understand the
implications of tricks like:
  bind views
  dns anycast
  edns-client-subnet
  etc

before you deploy them... That's all beside the point of good
documentation being available to support inter-operability between
vendor code for these features though. Should the IETF mint a standard
for 'feature-X' in the software system that makes up the DNS? Where's
the bar used to measure whether or not a feature has critical enough
mass/interest to be written up? Should all feature ideas get adopted
and then those which prove to wither be permitted to die out before
WGLC?

>
> while i'm on record has holding that view, it turns out that RFC 1035 does
> describe recursion and authority as co-residing in a server. so while this
> is in my view a dangerous practice and a bad idea, it's well supported by
> the scriptures.
>
>
> i don't think so. dns lookups have many purposes unrelated to returning IP
> addresses. i'd like to see 100 more things like SSHFP this decade.

ah, so you seem to be in the camp of 'let a thousand flowers bloom' or
whatever... that seems like fun as well. Back on topic though, should
the edns-client-subnet work get attention and potentially move forward
in this WG?

-chris

---

From: Suzanne Woolf <[email-masked]>
Date: Thu, 8 May 2014 12:43:06 -0400

> I support publication of this “stupid DNS trick” as an RFC .. of some sort, particularly given the breath of deployment.  That said, I’ve seen some of the warts and am sympathetic to Paul’s aging a…

Fully agreed.

> As for the publication track here, or any other [E]DNS extensions similar application, whether deemed unorthodox or unfashionable, I don’t think there should be blanket discretion and edict by the…

Ah, sorry. Was trying to reflect what the discussion was saying, not impose an “edict”. It seemed like a reasonable starting position.

Do you disagree? If so I’ll hope you’ll say what you think on the subject….

> Then again, perhaps DNSOP is not the place to document deployed and operational DNS stuff and we should revive DNSEXT to discussed operationally deployed DNS stuff, particularly given the charter d…

:)

best,
Suzanne

---

From: Colm MacCárthaigh <[email-masked]>
Date: Thu, 8 May 2014 09:44:20 -0700

> functions in one server and not in using DNS to direct traffic.

That's a pretty big assumption to jump to. It's pretty unlikely that all
ANAME implementors do that, as they'd have significant availability
problems. I'd say it's likely that some query the ANAME target periodically
and put what they find into the authoritative data store.

Of course that's even less compatible with edns-client-subnet, but there
you go.

-- 
Colm

---

From: Danny McPherson <[email-masked]>
Date: Fri, 9 May 2014 21:19:34 -0400

> Do you disagree? If so I’ll hope you’ll say what you think on the subject….

Yes, I think I do disagree with your previous conclusion, actually.  Specifically, the “sounds to me like” and:

“b) a resulting RFC as "Informational”.

I’m not sure I’d have so quickly reached the same conclusion if I were chair or based on what I’ve seen in the discussion here (!even after DW, who invented a strikingly similar feature for http ~16 years ago, pointed out a slewwww of concerns [and opportunities] with this particular feature oob), but I wasn’t actually counting opinions here like I presume you were and I do think this particular thing has demonstrated utility already, warts et al.  

Of course, I’m not sure I really care so much about this particular feature as to take a hard position, it’s more the meta topic.

Quite frankly, given implementation and deployment scope already this is the sorta thing that arguably coulda-shoulda been published years ago as Informational OR Experimental RFC even absent ANY working group support (and still could be, of course) -- and then [e.g., now] lending deployment experience to a potential Standards Track RFC IF it doesn’t perturb to many things and gains traction.  

OTOH, if the goal of putting it through the wringer here is to simply continue publication as an Informational RFC (albeit with a bunch of extra text talking about applicability, deployment experience, security and privacy considerations, architectural fashion sense, and other important stuff) then I think we might be missing an opportunity, or even worse...  IF DNSOP is going to muck with bits on the wire in Standards Track protocols, in addition to documenting operational DNS fun, then we should expressly aim to not push implementers away.  That’s the only reason I’m really responding to this thread at all, FWIW.

On a related noted, and in lieu of the earlier references from Jiankang Yao to draft-iab-dns-applications (i.e., RFC6950), folks considering this with context might want to also apply RFC5218 considerations here - which may well leave you more firmly on the fence or perhaps even more entrenched on either side of this particular discussion :-)

-danny

---

From: "Marco Davids (SIDN)" <[email-masked]>
Date: Thu, 15 May 2014 08:53:10 +0200

> Just because we _can_ make something work better doesn't mean we _should_.

Off course we should, it's our mission: :-)

"The goal of the IETF is to make the Internet work better.

(http://www.ietf.org/)

-- 
Marco

---

From: Ted Lemon <[email-masked]>
Date: Thu, 15 May 2014 09:02:43 -0400

> "The goal of the IETF is to make the Internet work better.

Indeed so.   And making things _other_ than the Internet work better can be damaging to the Internet.   It isn't always, but it's something we need to consider when evaluating proposals.

I think it's worth documenting this option because there's a code reserved for it, but I think it's highly questionable whether it makes the internet better, because it encourages practices with DNS that wind up violating the expectations resolvers might have for consistency of zones and so on.   See for instance my DISCUSS here: http://datatracker.ietf.org/doc/draft-ietf-cdni-framework/ballot/

This wound up opening up a huge can of worms about the various assumptions that CDNs make about how resolvers will process DNS records, how this mechanism interacts with DNSSEC, etc.   These things are definitely worth documenting, because people are doing them.  But whether they improve the internet is very much open for debate.   The CDNI document specifies other ways of accomplishing the same thing that I think are much less fraught.

---

From: S Moonesamy <[email-masked]>
Date: Fri, 16 May 2014 02:35:45 -0700

Hi Ted,
At 06:02 15-05-2014, Ted Lemon wrote:
>I think it's worth documenting this option because there's a code 
>reserved for it, but I think it's highly questionable whether it 
>makes the internet better, because it encourages practices with DNS 
>that wind up violating the expectations resolvers might have for 
>consistency of zones and so on.   See for instance my DISCUSS here: 
>http://datatracker.ietf.org/doc/draft-ietf-cdni-framework/ballot/
>
>This wound up opening up a huge can of worms about the various 
>assumptions that CDNs make about how resolvers will process DNS 
>records, how this mechanism interacts with DNSSEC, etc.   These 
>things are definitely worth documenting, because people are doing 
>them.  But whether they improve the internet is very much open for 
>debate.   The CDNI document specifies other ways of accomplishing 
>the same thing that I think are much less fraught.

What makes the internet better is usually subject to discussion.  The 
words "internet" and "better" in the previous sentence are not defined. :-)

I sent a few comments about that CDNI draft.  The DNS discussion in 
the draft was problematic.  It is worth documenting what people are 
doing.  It is worthwhile to consider whether the mechanism should be 
standardized by the IETF.

Regards,
S. Moonesamy

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 07:56:47 -0400

On May 16, 2014, at 5:35 AM, S Moonesamy <[email-masked]> wrote:
> I sent a few comments about that CDNI draft.  The DNS discussion in the draft was problematic.  It is worth documenting what people are doing.  It is worthwhile to consider whether the mechanism should be standardized by the IETF.

Did you feel that your comments were adequately addressed by the working group?

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 16 May 2014 08:18:19 -0400

On Thu, May 15, 2014 at 09:02:43AM -0400, Ted Lemon wrote:
> consistency of zones and so on. 

Despite my personal feelings about CDN tricks and the DNS, I would
observe two things.

First, if resolvers have expectations about consistency of zones and
so on, then they're broken.  The DNS has only ever been loosely
coherent.  You're simply _not allowed_ to make that assumption from
any point in the network except inside the authoritative server and,
maybe for certain kinds of consistency, in the slaves of the same
zone.  

Second, the Internet is actually working today using those kinds of
CDN tricks.  Indeed, some of the most important and most successful
nodes on the Internet rely very heavily on various DNS tricks, and
without them wouldn't be operating.  If we are serious about "making
the Internet better", surely we ought to have an opinion on how to
offer (as well as can be achieved given other strictures) the
functionality that is in fact ubiquitous.  I realise that you already
conceded the point that this particular extension ought to be
documented since it has a code point.  But it seems to me we ought to
be more enthusiastic than resigned in this case, even if we have to
hold our collective nose as well.  Either those who understand how the
DNS works will document what to do, or else people who have no clue
will make more "improvements".

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Tim Wicinski <[email-masked]>
Date: Fri, 16 May 2014 09:31:12 -0400

> will make more "improvements".

In this bucket you can put in all the 'CNAMEs at Zone Origin" work done 
by both the CDN vendors and the commercial DNS vendors in this space.

I was thinking once the Charter had been finished being blessed, I would 
put my toe in that water and see what lies beneath the surface.

tim

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 09:41:07 -0400

On May 16, 2014, at 8:18 AM, Andrew Sullivan <[email-masked]> wrote:
> But it seems to me we ought to

The big can of worms to which I was referring in the previous message was DNSSEC.   Deploying CDN functionality with DNSSEC is hard.   Not impossible, but definitely hard.   I'm not convinced it's the right way to solve the problem.   But then, I'm not convinced that DNS is the right way to solve these problems generally, although as you say, those with operational skin in the game seem to have good reason to have chosen this solution out of those available.

---

From: Olafur Gudmundsson <[email-masked]>
Date: Fri, 16 May 2014 10:00:46 -0400

> Did you feel that your comments were adequately addressed by the working group?

As a chair of the working group where this draft started (dnsext), I can state the discussion was “dysfunctional” there where some people 
strongly supporting this and few people saying this would cause the world to end and calling the proponents names. 
Most people avoided getting into the food fight, thus there is was no way to judge if the proposal had significant support. 

I personally want to see this published, 
	a)  as the type code has been allocated (by the ENDS0 Options Expert: yours truly) 
	b) this is in use and this option has a role in certain environments. 
	c) we need to be more rational in judging proposals on their merits not just by the initial smell test. 

	Olafur

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 16 May 2014 07:11:06 -0700

On Fri, May 16, 2014 at 6:41 AM, Ted Lemon <[email-masked]> wrote:

> The big can of worms to which I was referring in the previous message was
> DNSSEC.   Deploying CDN functionality with DNSSEC is hard.   Not
> impossible, but definitely hard.   I'm not convinced it's the right way to
> solve the problem.   But then, I'm not convinced that DNS is the right way
> to solve these problems generally, although as you say, those with
> operational skin in the game seem to have good reason to have chosen this
> solution out of those available.
>

Just to back that up; DNS tricks do play an important role in keeping the
internet robust and healthy. They're a key part of many DDOS mitigation
techniques, and network failure mitigation too. In my experience DNS tricks
are also much better than the alternative (pure anycast, redirects, etc
..).

It is harder to deploy DNSSEC around these tricks, and one must consider
that signed answers are replayable across "views" - but it's not that
significant in comparison to the overall challenges of deploying DNSSEC at
scale.

-- 
Colm

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 16 May 2014 07:24:02 -0700

On May 16, 2014, at 6:41 AM, Ted Lemon <[email-masked]> wrote:

> The big can of worms to which I was referring in the previous message was DNSSEC.   Deploying CDN functionality with DNSSEC is hard.   Not impossible, but definitely hard.  

No its not.  All you have to be willing to do is release the constraint on "all signatures offline".  Doing online signatures allows all the CDN functionality you want to be DNSSEC validated (not like DNSSEC really does much good for A records anyway...).

And even 4096b RSA signatures only take a handful of milliseconds to construct on the fly, you can cache signature validity for minutes even in the very dynamic case, and this is one of those operations that parallelize obscenely well.

If you want a funky-python-server, that really bends DNS, and supports dynamic signatures (including dynamic NSEC3 lies), tell me, I can give you my (ugly-ass) source.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 10:24:32 -0400

On May 16, 2014, at 10:00 AM, Olafur Gudmundsson <[email-masked]> wrote:
> few people saying this would cause the world to end and calling the proponents names. 

FWIW, when mailing list subscribers behave this way, what they say probably can't easily be considered part of the consensus evaluation, since they aren't participating in the discussion.

That said, I was actually talking about the CDNI draft, not the edns-client-subnet draft, although of course the two drafts are somewhat related.

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 16 May 2014 07:29:15 -0700

On Fri, May 16, 2014 at 7:24 AM, Nicholas Weaver
<[email-masked]>wrote:

> No its not.  All you have to be willing to do is release the constraint on
> "all signatures offline".  Doing online signatures allows all the CDN
> functionality you want to be DNSSEC validated (not like DNSSEC really does
> much good for A records anyway...).
>

There's no incompatibility between offline signing and returning different
answers to different source IPs; just sign every variant.

And even 4096b RSA signatures only take a handful of milliseconds to
> construct on the fly, you can cache signature validity for minutes even in
> the very dynamic case, and this is one of those operations that parallelize
> obscenely well.
>

You won't survive a trivial DOS from a wristwatch computer with that
approach :) Having static answers around greatly increases capacity, by
many orders of magnitude.

-- 
Colm

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 10:30:59 -0400

> No its not.  All you have to be willing to do is release the constraint on "all signatures offline".  Doing online signatures allows all the CDN functionality you want to be DNSSEC validated (not l…

It's quite a bit more complicated than that.   The main problem is that certain existing practices won't work and have to be changed somewhat.   It's entirely do-able; the problem is that someone who is already doing those practices may see the work required to add DNSSEC support prohibitively risky.

If you want to get into the details, I can see if there's a way to get you a copy of the discussion, but at this point it's moot since the document has been approved.   

I think it would be beneficial if someone with a real interest in this topic could write a draft about it explaining existing practice in more detail and talking about how to morph that so that it works with DNSSEC.   As you say, on-line signing is the first step.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 10:32:25 -0400

On May 16, 2014, at 10:29 AM, Colm MacCárthaigh <[email-masked]> wrote:
> You won't survive a trivial DOS from a wristwatch computer with that approach :) Having static answers around greatly increases capacity, by many orders of magnitude. 

Argh.   Having just agreed with Nick, I have to admit that this is a good and true point... :)

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 16 May 2014 07:34:31 -0700

> You won't survive a trivial DOS from a wristwatch computer with that approach :) Having static answers around greatly increases capacity, by many orders of magnitude. 

Actually, you can.  You prioritize non-NSEC3 records, since thats a finite, identifiable, priority set, and cache the responses.  Thus if you have 10k valid names, each with 100 different possible responses, and have a max 1 minute TTL on signatures, thats only 16k signatures/s in the absolute worst case, which you can do on a single, 16 core computer.

---

From: Jelte Jansen <[email-masked]>
Date: Fri, 16 May 2014 16:41:17 +0200

> maybe for certain kinds of consistency, in the slaves of the same

There are always assumptions; For every little bit that doesn't happen
to be specified (of which there were an awful lot back in 103X), you
assume an interpretation (or one of a select set). We try to minimize
them by exhaustive specification, but that doesn't always work.
Furthermore, sometimes there are views and models derived from the
actual rules, which can be seen as rules because they are not
contradicted in the current set of specifications. Changing something so
that those are no longer valid can be done, but the implications need to
be considered. This to me is also why there are Considerations chapters
in RFCs.

Some assumptions are considered broken ("DNS packets always < 512 bytes,
and firewalls should drop larger ones"), some are not ("any order of A
records in an rrset is equally valid so always just pick the first
one"). On yet others the verdict isn't always clear ("SERVFAIL means a
server isn't authoritative for the queried zone").

In this case, an assumption that changes is what 'loosely coherent'
actually means, and whether or not you can, in fact, cache answers, and
hand the same out to whomever sends them the same question. They may not
make any assumptions about the content they pass on, but they certainly
do make a few about its validity and timeframe (namely, that it doesn't
change until the TTL expires, and that it is the same for everyone).

To implement client-subnet means to implement a form of views within
your resolver in the form of split caches. If you don't implement it at
all there is no problem, but it certainly does change the model of the
world that resolvers have for those that do.

I don't think this is necessarily a problem, and as far as CDN's go, I
think the proposed draft is quite nice, and actually addresses this
problem. But things like that should certainly be considered. To name
something, what is the effect on forwarders? (what are forwarders in the
first place? what are their assumptions, do we consider those wrong? are
there any in deployment? is it a problem?)

Jelte

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 16 May 2014 07:44:04 -0700

On Fri, May 16, 2014 at 7:34 AM, Nicholas Weaver
<[email-masked]>wrote:

> Actually, you can.  You prioritize non-NSEC3 records, since thats a
> finite, identifiable, priority set, and cache the responses.  Thus if you
> have 10k valid names, each with 100 different possible responses, and have
> a max 1 minute TTL on signatures, thats only 16k signatures/s in the
> absolute worst case, which you can do on a single, 16 core computer.
>

16k/second is nothing, and I can generate that from a wristwatch computer.
Caching doesn't help, as the attackers can (and do) bust caches with
nonce-names and so on :/  A 16 core machine can do a million QPS relatively
easily - so it's a big degradation.

-- 
Colm

---

From: Paul Vixie <[email-masked]>
Date: Fri, 16 May 2014 07:50:10 -0700

>> maybe for certain kinds of consistency, in the slaves of the same

"not allowed" is interesting language, since you're "not allowed" to
have a cname and other data at the same node, but a lot of CDN's do it.
there is no on-the-wire enforcement of many of the "not allowed"'s in
the internet system; rather, there are a lot of hopes and prayers many
of which are encoded into RFC documents and writ large they say "if you
do this it might not work" or "if you don't this it might constrain
future evolution" or "if you do this it might irritate others."

anycast recursive DNS was in some sense "not allowed" because authority
DNS servers now judge the topology of the end system's browser by the
topology of the RDNS, which authority DNS servers are "not allowed" to
do. centralized RDNS is "not allowed" because end system apps are
written with the assumption that these resources will be within the
on-campus RTT range (about a millisecond), and those apps are, you
guessed it, "not allowed" to make that design assumption.

what the marketplace of ideas called the internet system actually
enforces is first-mover action. to my shame, microsoft once had to
cripple their AXFR logic to send only one record per header because of a
widespread bug in BIND4. i said "BIND4 is not allowed to insist on this"
and i fixed it, but until the fix got out to the long tail, microsoft
was stuck with a bug they never made.

the operative rules of the internet system aren't writ in RFC's but
rather in the systems deployed earlier than yours, or in some cases, in
the systems deployed more widely than yours. we don't have some kind of
super-panel who gets to veto ideas that don't fit into the vision of the
system as it is evolving and as it will evolve.

what we do have is advice: "if you're going to do this, here is a way
that works." in many cases, and DNSSEC is an example, the advice has an
additional property: "if you want a system like this, here is how
everybody else is doing it." in the past, the DNS advice offered by the
IETF has all had both properties -- these things work and we're trying
to get everybody to do it the same way because we have a vision of the
whole internet having this new feature."

today we're saying that second part doesn't matter any more. we're
saying "because of the market power of centralized RDNS and
trickster-ADNS, many old assumptions are no longer valid, and if you
want your part of the internet system to perform in this ad-hoc world as
remade by the CDN industry, you will need to pass the end system
browser's topology information through the RDNS so that the ADNS can see
it."

this is at best an engineering compromise, not engineering excellence
according to some visionary architecture. i'm all for documenting it,
but not because prior end systems are in any way "not allowed" to make
the assumptions they're making about DNS coherency, or RDNS latency.
we're deliberately saying only "if you want to do this, here is what
some people are doing" and deliberately withholding any statement to the
effect that (as with DNSSEC for example) "the consensus of the DNS part
of the IETF is that everybody ought to do this".

if we're going to withhold that second part, we have to do so
explicitly, because it has been implicit up until now.

>> without them wouldn't be operating.

that last part is a stretch. the CDN world had alternatives and still
has alternatives. things like sending deviant A/AAAA responses to
hopefully get the end system to connect to a proxy that's nearby does
not always work and there exist alternatives that are just as
statistically successful. we're not in a world that demanded what i call
"stupid DNS tricks" but rather in a world where "stupid DNS tricks" was
a way for the CDN industry to shift its costs onto the rest of the
internet system -- and they did -- and now we're paying another
installment of those costs, with the subnet option.

>>   If we are serious about "making
>> functionality that is in fact ubiquitous.

that's not my job now, but when it was, i had other ideas. happy to
discuss at the bar, but they aren't germane to this thread. simply put,
the costs have been shifted, it doesn't matter what the cost shifter's
motives or alternatives were. this turd is in our front yard now and
we're far behind on our cleanup effort. my vision of a better internet
has less state, and less combinations and permutations of state, than
what's implied by subnet option draft. your vision may differ. neither
of our visions is worth a damn at this point, we're not going to "make
the internet better", we're going to document the weird things people
are doing (subnet option) to work aroudn the weird things other people
did (move RDNS to a cloud; make ADNS incoherent). nowhere should we say,
and in no way can we expect consensus, that any of this is making the
internet better, or that it's good engineering.

>> I realise that you already

if you're capable of being enthusiastic while holding your nose, more
power to you. i am not. furthermore, it doesn't matter. all of us are
ready to see this option documented, whether holding our noses or not.
let's document it. but let's also be clear that this is something the
IETF did not develop and that the IETF makes no recommendation about.
unlike DNS itself or DNSSEC, the subnet option does not represent an
IETF consensus of a vision for the best engineered system having these
features. it's just part of a patchwork quilt designed by market forces.

vixie

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 16 May 2014 07:54:48 -0700

On May 16, 2014, at 7:44 AM, Colm MacCárthaigh <[email-masked]> wrote:

> 16k/second is nothing, and I can generate that from a wristwatch computer. Caching doesn't help, as the attackers can (and do) bust caches with nonce-names and so on :/  A 16 core machine can do a million QPS relatively easily - so it's a big degradation.

You miss my point.  That server is doing a million QPS, but its only providing ~16k/s distinct answers.

Your wristwatch computer can only cause a dynamic server a problem if its competing with the legitimate query stream's priority category.  The "priority" category, assuming 10k names and 100 options/name and 1m max TTL requires only a single system to support.

Thus your wristwatch loaders can only act to load the non-priority category, which would be NSEC3.  If you actually care about zone enumeration, you MUST generate NSEC3 records on the fly, because lets face it, NSEC3 in the static case doesn't stop trivial enumeration of the zone.

Basically, its observing that what you really want is "semi-online": The names you care about have at least some history/cacheability, and some level of finite space, but only on the order of a minute.   Once that property is there, you can do dynamic signing to your heart's content.

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 10:57:47 -0400

I will reiterate that it would be really wonderful if all this creative energy could go into writing a document explaining how to do this, rather than being squandered in a mailing list debate (although of course some debate can be expected to ensue anyway if the principals go down this path).

Ideally it would be nice to see both the static-signing and dynamic-signing-plus-cache alternatives documented and compared.

Just sayin'.

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 16 May 2014 08:03:38 -0700

> whole internet having this new feature."

There may still be an opportunity to give sensible proscriptive advice. For
example; it might be sensible to strongly caution against variable NS or DS
rrsets, and that's a behavior resolvers could be advised to reject when
they see it. It's an example of something that's not common, but that some
authoritative operator might experiment with (it's interesting to think
about) but would likely cause some real complexity chaos.

-- 
Colm

---

From: Colm MacCárthaigh <[email-masked]>
Date: Fri, 16 May 2014 08:13:56 -0700

On Fri, May 16, 2014 at 7:54 AM, Nicholas Weaver
<[email-masked]>wrote:

> > 16k/second is nothing, and I can generate that from a wristwatch
> computer. Caching doesn't help, as the attackers can (and do) bust caches
> with nonce-names and so on :/  A 16 core machine can do a million QPS
> relatively easily - so it's a big degradation.
>

That's not a typical CDN environment though. CDNs typically have far more
names than that. But you're right; online signing + caching probably is
workable in some environments.

> Your wristwatch computer can only cause a dynamic server a problem if its
> competing with the legitimate query stream's priority category.  The
> "priority" category, assuming 10k names and 100 options/name and 1m max TTL
> requires only a single system to support.
>

I've never been able to make prioritisation really work at microsecond
scale. I can imagine a dedicated process for signing and having prioritized
queues to it, but that would need so much packet copying that it would
likely degrade throughput seriously. Alternatively the DNS handling process
may defer the signing and keep its own queue locally, but that introduces
scheduling overhead. Every time I've tried it, I've found that taking out
prioritisation and smart scheduling made the overall average faster.

> Thus your wristwatch loaders can only act to load the non-priority
> category, which would be NSEC3.  If you actually care about zone
> enumeration, you MUST generate NSEC3 records on the fly, because lets face
> it, NSEC3 in the static case doesn't stop trivial enumeration of the zone.
>

Another approach to this is to pre-sign a fixed number of NSEC3 records per
zone, regardless of the zone's real size or contents :)

-- 
Colm

---

From: Nicholas Weaver <[email-masked]>
Date: Fri, 16 May 2014 08:30:04 -0700

On May 16, 2014, at 8:13 AM, Colm MacCárthaigh <[email-masked]> wrote:
> I've never been able to make prioritisation really work at microsecond scale. I can imagine a dedicated process for signing and having prioritized queues to it, but that would need so much packet copying that it would likely degrade throughput seriously. Alternatively the DNS handling process may defer the signing and keep its own queue locally, but that introduces scheduling overhead. Every time I've tried it, I've found that taking out prioritisation and smart scheduling made the overall average faster. 

Good point, although it really only takes 2 queues:

If in cache->return it.

If "high", queue 1
if "low", queue 2.

You need at least one queue anyway, because you want to do the cache check first, and you need to keep doing cache checks while your crypto is ongoing to avoid the trivial DOS.

> Another approach to this is to pre-sign a fixed number of NSEC3 records per zone, regardless of the zone's real size or contents :)

You need the order of records to be HUGE, because you walk the NSEC3 zone, and you then use a dictionary attack:  NSEC3 in a static case, even with a huge number of bogus NSEC3 records you can only hide names if names are not a-priori predictable from a reasonable attacker.

---

From: S Moonesamy <[email-masked]>
Date: Fri, 16 May 2014 08:02:29 -0700

Hi Ted,
>Did you feel that your comments were adequately addressed by the 

I gave up on reading the first response to my comments as I did not 
want to push back strongly; it's an effort and it can be viewed as 
antagonistic.

Regards,
S. Moonesamy

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 16 May 2014 11:54:04 -0400

> world that resolvers have for those that do.

It changes the model of the world that resolvers _who implement this_
have.  The draft calls all of that out explicitly, I think (early
versions of the draft didn't adequately, I thought, but this one
certainly does in my reading).  A resolver that doesn't implement this
stuff is unaffected.  And what of a resolver that doesn't do this and
which is querying another resolver that does?  Well, of course, those
cases hurt in other ways too; but also, that's part of the risk of
deploying these tailored answers on the authoritative side anyway.  I
think that's out of scope for this document, though I'm certainly
prepared to work on another doc that talks about these issues.

A

-- 
Andrew Sullivan
[email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Fri, 16 May 2014 13:04:48 -0400

On May 16, 2014, at 11:02 AM, S Moonesamy <[email-masked]> wrote:
> I gave up on reading the first response to my comments as I did not want to push back strongly; it's an effort and it can be viewed as antagonistic.

I think there's a fine line between ratholing and not getting the point across.   I would really encourage you, if you feel that you are in danger if ratholing with someone, to simply note that your comment has not been addressed in the draft and then stop talking.   Mention it again during last call.   If the chairs do not account for this in their consensus call, complain.

This bypasses the ratholing process.   If the chairs are part of the ratholing, then complain to the AD, or the IESG.   This is not antagonism: it is opposition.   Opposition is a key part of the IETF process.   It doesn't mean that you get to have your way, but valid points you raise should be addressed.

When the IESG reviews a document that's made it all the way up through the process, we feel somewhat bound to accept what the working group says unless there's a really serious problem.   So it's vitally important that working group participants' feedback be considered and addressed in the consensus call.   By the time we get it, it's too late for major course changes.

(I realize that I am not telling you anything you don't already know, but I'm playing to the crowd here... :)

That said, I would rather document this and say "here abide monsters" than not document it; what I would have liked to see in the CDNI document would have been a great deal more DNS fu during the working group process.

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 16 May 2014 13:17:36 -0400

I don't want to spend a lot of time in this rathole (and this will be
my last remark on it), but I want to clarify something about what I
was trying to say.  I suppose I should have picked better language.

> have a cname and other data at the same node, but a lot of CDN's do it.

The CNAME example is actually the "not allowed" meaning that I had in
mind: it's contradictory to the meaning of the term.  You're not
allowed to depend on consistency among answers in the DNS because, by
definition, the DNS is a distributed, loosely coherent system, and
therefore depending on consistency is contrary to the definition of
the system.  You're not allowed to use a CNAME and other data at the
same node, because "CNAME" means "the owner name is really this other
name": the prohibition of other data there is a consequence of that
and having other data at that owner name is absurd.  Both of these
cases are examples of why violating such assumptions frequently
results in surprising behaviour.  (I cannot count the number of people
to whom I have explained why sometimes answering with a CNAME at their
apex makes mail bounce, for instance).

> future evolution" or "if you do this it might irritate others."

That's all what I meant by "not allowed" is on the Internet.  There
are no protocol cops, as you know.  Therefore, stuff that is contrary
to the definition of the system might work sometimes, depending on
what various implementations do.  But if you rely on it, you could be
hosed.

> guessed it, "not allowed" to make that design assumption.

These are both "not allowed" in the sense of "I'm in charge here, and
I scold you."  That's not what I meant.  These cases are not
entailments of the very definition of the system, but instead are at
most entailed by other inferences about the way the system is
designed.

Best regards,

A

-- 
Andrew Sullivan
[email-masked]

---

From: Paul Vixie <[email-masked]>
Date: Fri, 16 May 2014 10:33:06 -0700

one of the icann ssac (then "secsac") consensus processes i am most
proud to have been part of was this, in 2004:

http://www.icann.org/en/groups/ssac/report-redirection-com-net-09jul04-en.pdf

indeed, this document and the process followed in creating it may still
stand today, as ssac's finest hour.

see especially section 2.3, "design principles and good practice in the
internet technical community", starting on page 8.

unless i can think of anything new to say on this topic, this will be my
final comment on this thread.

consensus tally people: i'm "yes we must document it, but we must also
explicitly withhold recommendation of it, both in the introduction
section and in some kind of applicability section down near the iana
considerations."

vixie

---

From: Dan York <[email-masked]>
Date: Fri, 16 May 2014 19:12:31 +0000

Tim,

On May 16, 2014, at 9:31 AM, Tim Wicinski <[email-masked]<mailto:[email-masked]>>
 wrote:

functionality that is in fact ubiquitous. <snip> Either those who understand how the

In this bucket you can put in all the 'CNAMEs at Zone Origin" work done by both the CDN vendors and the commercial DNS vendors in this space.

I was thinking once the Charter had been finished being blessed, I would put my toe in that water and see what lies beneath the surface.

On a personal level I'd love to see some work done on "CNAMEs at Zone Origin" as the issue just bit me with one of my own sites. For a good number of years I've been giving out just the domain name as the web address and that worked fine with me providing an A and AAAA record pointing over to the hosting provider (all DNSSEC-signed, of course).    Then recently the hosting provider got slammed with a massive DDoS attack and in fighting off that attack they put up a CDN in front of all the hosted web sites.

We customers were then told to ensure we were using CNAMEs instead of direct addresses... for me this has meant unfortunately having to switch to using "www.<domainname>" for that site and setting the CNAME there.  I'm still working out which of the various "DNS tricks" from different providers that I'll have to use to get the raw domain name to point correctly over to the web hosting provider.... and given that I have many years of posts out there with links to the raw domain name this is kind of important to me.

For some time now I've been seeing in the marketing/communication space an increasing usage of dropping the "www" and just promoting the raw domain name.  I think with the many hundreds of newgTLDs coming out we'll probably see people there seeking to use only the domain name for their website.

And many of them will want to use hosting providers or CDNs where the desired configuration is to point a CNAME over to the hosting/CDN provider.... which leaves them stuck - or resorting to "DNS tricks" - if they want to use the zone name itself as their web site address.

So yes, I think this is very definitely a current operational problem that needs to be worked on.

My 2 cents,
Dan

--
Dan York
Senior Content Strategist, Internet Society
[email-masked]<mailto:[email-masked]>   +1-802-735-1624
Jabber: [email-masked]<mailto:[email-masked]>
Skype: danyork   http://twitter.com/danyork

http://www.internetsociety.org/deploy360/

---

From: Mark Andrews <[email-masked]>
Date: Sat, 17 May 2014 09:00:19 +1000

You would be insane to publish varient DS records with edns-client-subnet
to the public as there is no requirement for clients to use the
same address to lookup DS records as they use to lookup DNSKEY
records.  Similarly for DNSKEY or RRSIGs based on DNSKEYs which are
varient based on edns-client-subnet.

Using different DNSKEY's is problematic with just the internal /
external version of zones.  Similarly signed vs unsigned versions
of the same zone.

When we (ISC) get problem reports about signed external / unsigned
internal the standard response is "sign all versions of the zone".

When we (ISC) get asked about what keys to use we recommend using
the same keys for both internal and external zones.  Now there are
highly controlled environments where you can have differing keys but
you also have to publish alternate trust anchors, you can't have
machines jumping from inside to outside without flushing caches and
adjusting trust anchor sets.  etc.

-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]