Subject: [dnsext] Personal position on adopting the client-subnet option

From: Colm MacCárthaigh <[email-masked]>
Date: Mon, 31 May 2010 15:37:48 -0700

Because it might be useful, and hopefully lead to less mail - on my
part at least - I thought I'd write up what I think the client subnet
option is actually about. In part because Paul suggested it (well he
suggest a blog post, but this will have to do :) ).

I am broadly in favour of the option, and its adoption as a work item,
for the reasons I'll set out below.

Full-disclosure reminder; I've contributed towards the draft, know
many of the authors, and have implemented DNS tricks with multiple
name-servers on behalf of various employers. I am however, only
representing myself.

But first, my impression of the background.

Controversy one;
CDNs and other providers have chosen to use "DNS tricks" to direct
users to their "best" node. Some maintain that these tricks are
ineffective and place an unfair burden of support on intermediate
operators. For this reason, and also a perceived danger to the
stability of the internet - some also maintain that these tricks
should be actively discouraged.

Others maintain that these tricks are effective, and that any
operational burden is not a significant source of complaints or
problems. Worse still is that the two sides of the argument seem to be
unable to come to consensus on what would even be a fair test of
"effective", making what should be a simple data-driven engineering
consideration intractable.

Controversy two;
Some operators are now running large open resolvers. Those operators
maintain that there are benefits to using their services including
better caching ratios, more security against cache poisoning and more
reliable service than many in-ISP resolvers. Some maintain that this
is a flawed architecture, and an anti-pattern to be discouraged.
Further more, some resolver operators offer to filter queries and
intercept NXDOMAIN responses - a practice some find to be an abuse of
the DNS, to be actively discouraged.

The engineering problem;
Open Resolvers and CDN DNS tricks do not work well together - because
of the much broader client base of open resolvers, the source IP of
the query towards the CDN has very poor network locality.

The proposed fix;
Open resolver and CDN operators would both like to see the problem
fixed. Together, they have put effort into a draft that would allow
some client ip information to be shared anonymously. They do not
intend for this option to be used by anyone else (though they are free
to) and are not attempting to add to operational burden for
intermediaries.

This kind of option could be implemented in private, on an ad-hoc
basis, however instead there is a published mechanism being proposed.
The mechanism takes into account client anonymity and caching
efficiency, and while there is tweaking to be done - appears to have
been given serious engineering consideration.

I believe that this is 100% appropriate as a work item, because;

   1. It is a proposed DNS extension.

   2. It attempts to solve a real engineering problem on the real internet. [1]

   3. Although some feel that CDNs shouldn't use DNS tricks, and that
open resolvers should not exist in the first place - nevertheless;
they do. It would be worse to see a working group take on the role of
moral guardian of a protocol. That risks marginalisation, as
implementers will simply forge ahead regardless and there will be
poorly standardised non-inter-operable implementations complicating
the internet.

Note the word "influence" in the IETF mission statement, standards are
not mandatory - there is no police force.  Where implementers have
strayed from the advice of the community - must they be punished with
non-cooperation for all time, or encouraged back into the fold?

  4. IETF working groups have successfully worked on proposals far
more controversial in terms of abuse of intermediaries (e.g. p2p
protocols) without having to adopt a position of endorsement or
support.

  5. Pragmatically, I do not think it very likely that CDNs will cease
using DNS tricks or that Open resolvers will cease to exist in even
the medium term future. While some feel that they should be
discouraged from their practices nevertheless, we should not be slaves
to idealism. Engineering for the internet as it exists - rather than
as we would wish it to exist - is, in an important thing.

  6. I think it's great to encourage implementers to publish
references for techniques they could otherwise do in-private. It gets
more eyes on a problem, and benefits the entire engineering community.

  7. Finally, and least importantly, I am unconvinced by the technical
objections stated so far on this list.

   a) There has been some discussion of alternatives in other
protocols (HTTP redirection, changing BGP, inventing a new DNS
specifically for tricks) - but none have stood out as sufficient or
realistic.

  b) There has been some discussion about inefficacy, but no such
concerns from the putative implementers who would be most affected by
this. If it is ultimately ineffective, it is they who will bear that
burden.

 c) There has been some discussion to the effect that there is an
end-to-end principle for DNS query payloads, as I've described
elsewhere I don't agree that there is, for simple technical reasons.

  d) DNS consistency has been held up as a core architectural property
of the DNS, however DNS has never had atomic consistency (temporal or
spatial) and is explicitly inconsistent by design. (partial updates,
effects of TTL and caching, and so on). Inconsistent replies are a
fact of life in DNS, tricks or not.

The concept of using DNS as a routing mechanism may represent a
departure from its original intent. Over time, DNS has trended towards
being used more and more as a lookup mechanism than as a database, and
"dynamic DNS" servers can trace a heritage that is almost as old as
DNS itself. While the effects of a low TTL on intermediary caches are
a third-party overhead - it does not seem very large, in the context
of overall internet domain name growth and other operational practices
such as enforcing max-ttls for end-user convenience. In that context I
do not think that the nature of DNS extensions need be constrained by
a  "DNS is a global, consistent database" model. The NSID option is
also some precedent for treating DNS as a mere lookup mechanism.

Instead we should be discussing the meat of the proposal, and figuring
out how it interacts with DNSSEC, and what the effects of "views"
within a resolver cache might be things like glue.

[1] "The mission of the IETF is make the Internet work better by
producing high quality, relevant technical documents that influence
the way people design, use, and manage the Internet"  - IETF mission
statement.

-- 
Colm

---

From: Mark Andrews <[email-masked]>
Date: Tue, 01 Jun 2010 09:34:03 +1000

Part of the problem is that caches now go from 1 RRset per name to
1 RRset per client.  That is a fundamental change in design
constraints.

Open resolvers get their speed benefit from having a large client
base map to a single cache entry.  When you switch to per client
RRsets you defeat that cache benefit.  This is very much a balancing
act.

It's also not just the large open resolvers.  It's every large ISP
on the planet as well that will be presured to turn this on.
Chicago vs LA for NA ISP's.  Sydney vs Perth for Australian ISP's.

Also once the code is out there it becomes impossible to retrieve
and impossible to prevent abuse of.  It's better not to write the
code in the first place.

Mark
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: [email-masked]

---

From: Mark Andrews <[email-masked]>
Date: Tue, 01 Jun 2010 09:48:20 +1000

In message <[email-masked]>, =?IS
> the way people design, use, and manage the Internet"  - IETF mission

This is not a zero cost solution.  There are down sides to deploying
this technology.  It will require bigger more complicated caches to
support the same number of clients.

---

From: Colm MacCárthaigh <[email-masked]>
Date: Mon, 31 May 2010 16:49:19 -0700

> RRsets you defeat that cache benefit.  This is very much a balancing

This proposal has been authored by the operators of 2 different Open
Resolver services. (Google and Neustar). It has been publicly endorsed
by a third (OpenDNS). That's some indication to me at least that the
problem space is tractable :)

Maximising cacheability of the answers - which the proposal does - and
the ability to prefetch, makes it seem like there are credible, robust
solutions for the balancing act.

> Chicago vs LA for NA ISP's.  Sydney vs Perth for Australian ISP's.

Large ISPs are already facing this pressure. Indeed for the same
reasons, many are operating large fleets of resolvers that are
geographically dispersed. E.g.;

http://dns.comcast.net/status.php

ISPs will always face pressure for "better" internet service from
users. Surely adding to their options can only be a good thing? It's
hard to see how the caching hit ratios possible with the proposal
would be any worse than having a large distributed fleet.

> and impossible to prevent abuse of.  It's better not to write the

Abuse in what sense? If people create - say a bind9 or dnscache or
whatever patch - for this feature, and post it online, so that it
might be easier for ISPs to use. Where's the abuse? The option is more
burdensome for implementers and adopters - yes, but I think it's a
long from way from "responding to user complaints" to "forced into
doing something they don't like".

-- 
Colm

---

From: Colm MacCárthaigh <[email-masked]>
Date: Mon, 31 May 2010 16:52:48 -0700

> support the same number of clients.

But nowhere in the proposal does it somehow require cache operators to
implement it. And 3 of the largest, most affected, cache operators are
telling us they're ok with it.

Now one could be cynical and frame it as incumbents seeking to raise
the cost of entry for newcomers - but a) the IETF is not an
anti-competition authority  b) it's hard to take seriously, the option
is not *that* burdensome, memory and storage are cheap.

-- 
Colm

---

From: Mark Andrews <[email-masked]>
Date: Tue, 01 Jun 2010 10:12:16 +1000

In message <[email-masked]>, =?IS
O-8859-1?Q?Colm_MacC=E1rthaigh?= writes:
> > This is not a zero cost solution. =A0There are down sides to deploying
> > this technology. =A0It will require bigger more complicated caches to

It has other impacts as will.  You can no longer combine queries
from multiple clients.  Instead of have 100 clients waiting on one
resolution you will have 100 clients waiting on 100 resolutions.

You have the problem of rogue authoritative servers delibrately
sending large prefixes to make caching ineffective.  Combine that
with a attack that generates lots of DNS traffic directed at
authoritative servers that are not responding.

Mark

---

From: Colm MacCárthaigh <[email-masked]>
Date: Mon, 31 May 2010 18:47:53 -0700

> resolution you will have 100 clients waiting on 100 resolutions.

I don't think that's a fair statement. The proposal will certainly
diminish and complicated the ability to combine some queries - but it
will not eliminate it. It's a relatively simple heuristic, for
example, to remember the subnet size for a particular client subnet,
and to treat that as a single caching bucket until informed otherwise
(e.g. by a change of subnet size upstream). That's certainly
implementable in software - with safeguards.

Additionally, any potential impact of this inability to combine
queries will be born by the CDNs and resolver operators. Again, they
are free to turn it off if it really is a problem for them.

> sending large prefixes to make caching ineffective.

I don't think that's fair or realistic either. As operators here have
stated - nobody forsees enabling this as an "always on" option.
Instead resolvers, in particular, and possibly also CDNs are more
likely to maintain small lists of servers with which they exchange the
option. It is plainly in the interest of adopters to not do as above,
as better cacheability helps save latency too. Interests and
incentives seem to be well-aligned in this regard.

> Combine that

I'm not sure I follow this part. If a CDN is made more vulnerable to
attack by adopting these tricks - it will be interest to ensure that
their implementation is attack-resistant.

Something else which is important to note though is that your
statements could easily be made about DNSSEC;

   * Increases complexity for resolver operators - check
   * Creates pressure on ISPs to adopt a complex protocol they may not
like - check
   * Increases the vulnerability of the DNS to amplification and
cache-busting attacks - check

But the working group found a way to accommodate those changes :)

-- 
Colm

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 01 Jun 2010 10:50:11 +0900

> 1 RRset per client.  That is a fundamental change in design

True.

And, the first thing to do for this creeping featurism is to
define an extension of zone transfer format for such zones.

The next thing to do is extension of NOTIFY for servers to be
able to invalidate cached data of clients upon routing metric
changes.

						Masataka Ohta

---

From: Doug Barton <[email-masked]>
Date: Mon, 31 May 2010 20:20:22 -0700

> http://dns.comcast.net/status.php

Good, problem solved then. Next?

> users.

I don't see the pressure on this idea coming from users.

> It's hard to see how the caching hit ratios possible with the
> proposal would be any worse than having a large distributed fleet.

I've done "large distributed fleets" of resolvers, it's really not that 
hard. You also have a lot of advantages in terms of less SPOF's, etc.

Doug

-- 

	... and that's just a little bit of history repeating.
			-- Propellerheads

	Improve the effectiveness of your Internet presence with
	a domain name makeover!    http://SupersetSolutions.com/

---

From: John Payne <[email-masked]>
Date: Tue, 1 Jun 2010 00:23:08 -0400

> support the same number of clients.

When does the IETF limit itself to zero cost solutions?

---

From: John Payne <[email-masked]>
Date: Tue, 1 Jun 2010 00:30:36 -0400

> 1 RRset per client.  That is a fundamental change in design

Only a problem for a resolver implementing this... And that might not be a difference in design constraint for some of them. 

> RRsets you defeat that cache benefit.  This is very much a balancing

I'm going to assume that the authors of this draft are aware of this, and are willing to trade DNS latency for application latency in some cases.

> Chicago vs LA for NA ISP's.  Sydney vs Perth for Australian ISP's

CDNs already ask those ISPs to setup distributed resolvers.  Although they might represent a lot of end users, distributing resolvers improves general DNS latency for their end users, and provides sufficient granularity for the "stupid DNS tricks" so they're really not the audience. 

> and impossible to prevent abuse of.  It's better not to write the

The first sentence is hard to argue with.  The second causes me to say "I don't believe you wanted to say that"

---

From: Mark Andrews <[email-masked]>
Date: Tue, 01 Jun 2010 14:34:49 +1000

In message <[email-masked]>, John Payne wri
tes:
> > In message =
> <[email-masked]>, =3D?IS
> > O-8859-1?Q?Colm_MacC=3DE1rthaigh?=3D writes:

It was being presented as all benefit when it has downsides.  Those
downsides need to be taken into consideration.