Subject: Re: [dnsext] Chairs statement on "client-option" debate

From: Wilmer van der Gaast <[email-masked]>
Date: Mon, 26 Jul 2010 01:24:46 +0200

As authors of edns-client-subnet, we never sent a reply addressing The
Chairs' comments.

The main issues seem to be related to 1 - clarifying the problem
statement, 2 - explaining why DNS is the best solution space, and 3 -
have a discussion about the costs, making sure that the right people
are paying for them.

We'll provide a clarified problem statement in edns-client-subnet-02
(really called -00 thanks to the counter reset). We believe 2 and 3
are discussions out of the scope of the draft itself, so we'll explain
our thoughts on those in this e-mail.

In short, we understand that edns-client-subnet is not a perfect
solution. We believe, however, that a perfect solution would require
making changes to how the user agent works. We can't do that, we only
get to change the nameservers and recursive resolvers. Changing all
user agents will take years (note how there are still many people
using MSIE6 or even Netscape). Having all interested parties (and only
them) roll out edns-client-subnet can be done in less than a year.

Currently users of open resolvers (which are used for various reasons
including performance, filtering of undesired (family-unfriendly,
etc.) content, and avoiding their ISP's ad-serving nameservers) have
problems with accessing some CDN content (going from severe latency
penalties to content simply being unreachable), we need to fix that.

This said... there were several alternative solutions proposed in
various threads that also don't need work on the client. Most of these
are known with the CDN industry, some are actually used. However, DNS
tricks are still the most popular. Here's some of the reasons why we
believe this is the case:

* If something breaks, one of the easiest and more reliable ways to
send users somewhere else is to... change DNS records, and have short
TTLs. No matter how much redundancy you build in your infrastructure,
eventually you'll have to do maintenances on it, or it'll break. DNS
is a simple, painless, and easy way to control where your users end
up. If you have a highly distributed infrastructure, this can happen
often, and from here.. to returning results that are geolocated, the
step is small.

Two ways were proposed to do this on the HTTP level instead:

* Framesets. These were deprecated several years ago for good reasons
outside the scope of this proposal, so no viable solution to this
problem.

* The other proposed solution was to use HTTP 30x redirects. There are
a few problems with this:
  - The main weakness is the latency penalty. A user has to resolve
www.foo.com first, connect to that server, send a query, get a 30x
response. Then disconnect, resolve the new hostname, etc etc. A
well-written app can limit this to just the first query, but this
requires putting this logic into the web application, which can be
considered a layer violation.
    The latency cost of establishing a TCP connection with servers
just for a redirect is sensibly more costly than a UDP query (which is
also only done on the first request, and often answered from cache),
and can definitively affect the overall latency and user experience.

  - This will either mean forwarding user agents to URLs with IP
addresses in them, or to URLs like http://lhr00.mail.foo.com/. This by
itself gives several problems:
    * These URLs look ugly, are not easy to remember, not practical to
bookmark/pass from person to person (many CDN nodes are accessible
only to customers of ISPs hosting the node), needlessly expose
internals to the user (yesterday my request went to London, and today
it goes to Amsterdam??).
    * With bookmarks in mind, content providers will be required to
keep old datacenters/servers in DNS even though they're
decommissioned. Also, taking down a datacenter temporarily gets more
complicated.
    * This is likely to break caching. The full URL is (part of) the
cache key of HTTP content. If the hostname changes, the client will
have to reload the full file. Webapps these days are often powered by
hundreds of kilobytes of JavaScript, being able to cache those is
essential.
    * This also causes problems with SSL certificates. Wildcards and
other ways are available to work around this problem, but both are
still not supported by all user agents.
    * And in a related note, in a world where phishing is a real
threat to many, it must be easy for users to recognise reliable URLs.
Expecting that someone unfamiliar with how URLs and DNS work that
"https://lhr00.mail.foo.com/" is good while
"https://lhr000.com/mail.foo.com/" is bad, is unrealistic.
    * Having said that, this technique is already used for things like
video streaming, where the URL is not ever going to be visible to the
user and throughput is more important than latency.

* Most of all, both solutions only solve the problem for HTTP.
Although HTTP is definitely our main focus here, other protocols can
benefit from this:
  - SIP came up as an example, although it's not a great one, as the
resulting RTP stream is much more latency-sensitive than SIP itself,
so the right RTP server can be selected by the SIP proxy already.
  - VPNs
  - Various e-mail protocols

Using BGP/anycast instead doesn't have any of these issues. However,
it has its own imperfections. Although anycast is fairly good at
directing packets to the nearest endpoint, the nearest endpoint is not
always the right endpoint.

CDNs are about more than just latency. They're also about capacity.
When a node is full, the excess traffic should go elsewhere. Pulling
the IP announcement wouldn't get redirect the excess but all traffic.
There may be ways to refine this, but they're fragile and hard to
control.

Another problem with using anycast is the fact that redirecting
traffic (i.e. when taking down a destination for maintenance) moves
new *and* existing TCP connections, resetting all the existing
connections, often with ugly user-visible consequences. These
redirections can't be avoided since they're not just within the power
of the content provider.

Using DNS has its own imperfections, but the CDNs of today have mostly
dealt with them already. Think of problems like not knowing how many
application queries will be generated by once returning a certain IP
address in a DNS response.

The last problem unsolved so far is the wrong assumption that users
are always close to their resolvers. edns-client-subnet is the key to
making them work completely. It comes with its own problems and
challenges, but they affecte only the CDNs and the operators of
recursive resolvers who opt in to using the option, as explained
below.

3) COST OF IMPLEMENTATION

In our mind, edns-client-subnet would be turned on and used only by
those third party resolvers and authoritative name servers that need a
solution for problem (2). We don't see this option being enabled by
anyone else, or worse, by default.

In any case, if we had to break down the cost of this option, we'd have:

a - implementation cost, authoritative nameserver, and recursive resolver.

and, on servers that enable the option:

b - additional load caused by more queries being sent
c - additional resources to handle larger caches, as replies would now
be tied to networks
d - configuration and management overhead

So, on the CDN/authoritative resolver side: authoritative servers that
don't use DNS tricks, can keep living happily without implementing
this option. Authoritative servers used by CDNs are likely to care
about the latency of their users, and are likely to implement support
for this option, paying their share on the cost. (They can finally
provide good latency to users of third party resolvers.)

On the recursive resolver side: if you're running a recursive resolver
for a small network or for a set of users that are close to the
resolver, you can keep living happily without enabling this option.

If you are running recursive resolvers for a large ISP or distributed
network, you're probably *already* paying the price: you probably have
lot of recursive resolvers, close to your users, to avoid paying the
price of the extra latency. In this case, you can keep doing your
business as usual, or ... you can *decide* if you *want* to, to enable
edns-client-subnet. Knowing explicitly which servers are returning
responses that are tied to a particular IP address also helps with
debugging, implementing smarter algorithms for shared caches, and
planning for capacity (how many servers we need, where, ...). So, if
you're willing to pay the cost, it's probably because you get a
benefit.

Once you have a software that supports edns-client-subnet, what you'll
have to do is configure how many bits of the network of the user to
forward, and for which networks your server will forward
edns-client-subnet.

If you are running a third party resolver, well: you can keep doing
business as usual, so, ... live with the fact that some of your users
will have bad latency when connecting to some CDNs. Or enable
edns-client-subnet, with more or less the same
advantages/disadvantages as in the previous case.

Again, this should not be enabled by default, we don't see this being
generally useful.

As for other authoritative nameservers, that don't need or want
support for edns-client-subnet, they might see additional EDNS0
options from time to time, but as long as they ignore them, they're
fine. Broken firewalls and filters will affect edns-client-subnet as
any other EDNS0 option, so nothing special to say here.

We hope this answers the questions on the table. We'll keep working on
-02 and hope to have it ready soon after IETF78.

Regards,

Wilmer v/d Gaast, Carlo Contavalli,
Sean Leach, Darryl Rodden.

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 26 Jul 2010 09:22:43 +0900

> considered a layer violation.

Address selection based on latency is an issue at or above
the transport layer that trying to address it at the IP layer
is the layer violation.

At the connectionless IP layer, there is no notion of round
trip nor round trip latency.

> internals to the user (yesterday my request went to London, and today

I'm afraid I have already shown, with real world examples, that
the real world content providers such as yahoo and google actively
use the URLs you called "ugly".

						Masataka Ohta

---

From: Paul Vixie <[email-masked]>
Date: Mon, 26 Jul 2010 04:22:31 +0000

> Date: Mon, 26 Jul 2010 01:24:46 +0200
> From: Wilmer van der Gaast <[email-masked]>
> ...

no, we don't need to fix that.  it's not our bug.

open resolvers can either work well within the internet as it exists, or
they can improve themselves, or they can go home.  shifting the burden of
this bad idea (open resolvers) onto the architecture itself, for the
unilateral benefit of the advertisers, CDN's, and recursive dns share-
holders, would be not just bad engineering, but irresponsible stewardship.

next thing you know we'll be adding nxdomain support to application layer
protocols because, well, it just wasn't good for the open resolver
businessmen.  and we'll advise against any subsequent application or
protocol design that wouldn't work well for high-RTT recursive servers
(stub dnssec validation comes to mind.)

this way lies madness.  just say no.

(i'll say more about how to do content filtering in cooperating recursive
dns servers in my remarks at blackhat and defcon, next week in las vegas.
suffice to say, it doesn't require a new open recursive industry, at all.)

> Sean Leach, Darryl Rodden.

nothing personal, guys.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 26 Jul 2010 08:24:22 -0400

I am trying very hard to work out which is the 'bad idea' and which is
the 'architecture' here.

>From a security point of view, the idea that I delegate DNS resolution
to any random party that my laptop may be connecting to is stupid and
wrong. I do not and should not trust the local DNS at panera, IETF or
any other random location I happen to be at.

Secondly, from a security point of view it seems very sensible to
apply a 'default deny' approach and only enable resolution for the
limited subset of DNS domains that a party I trust deems to be 'safe'.
I have no interest in being able to resolve names registered by
Internet criminals attempting to defraud me.

-- 
Website: http://hallambaker.com/