Subject: Re: [Resolverless-dns] Paper on Resolver-less DNS

From: Erik Sy <[email-masked]>
Date: Sun, 1 Sep 2019 22:40:26 +0200

Hi Joe,

On 8/30/19 00:15, Joe Abley wrote:
> However, I do think it's also worth pointing out that geo load-balancing using the DNS today is already a terrible mess and it will be a joyous day when it can be replaced by something less bad.
>
> (soapbox noises)
>
> The general process by which a DNS server responds to a query for a (name, class, type) tuple that is configured with geo-specific answers is to process a set of rules at query time, consulting general-purpose databases maintained at the DNS edge that map an address to something like a (provider, city) pair, and then doing additional database lookups to determine the response that should be sent to the client for the particular service whose name (one of whose names) the client is looking up.
>
> The complexity of the code at the DNS edge is significant, and significant failures have been observed. In an enterprise DNS provider, the code that does this stuff is often not available for public review, since it constitutes secret sauce that tastes of competitive advantage. This stuff is really hard to do without failure and introducing customer-enraging latency; it's not unreasonable that people try to protect their investment.
>
> Centralised resolvers need to make use of the EDNS0 client-subnet option in order to allow the user's real location to be exposed to the server that is geo-answering. The EDNS0 client-subnet option has unintended consequences (see David Dagon's recent descriptions).
>
> The database that maps addresses to (provider, city) pairs is either frequently inaccurate (watch *NOG lists for pleas for help from operators whose customers are being served from nodes far away) or is accurate because it is maintained by mining other data sources. If you have a really broad anycast network you might be able to imagine doing this based on routing and answering node location; for everybody else those data sources are real user monitoring scripts running in browsers on not-obviously-related web sites, all performing performance measurements and centralising the results without the users of those stooge web sites necessarily being aware, or at least being given the option to opt-in.
>
> Complicated web services, perhaps especially those with complex UX, use all of these techniques together with a litany of short-lived, non-repeated DNS names intended to customise and optimise the performance for the end-user. The end result in some networks has been the observation that these short-lived names consitute such a high proportion of cached data (e.g. in campus resolvers) that "we don't have the RAM to handle another Facebook".
>
> Some big access providers configure resolvers with hundreds of exit addresses so that they will source queries for particular names towards enterprise DNS servers from dedicated sources, as a workaround for when the geo-ip on the server is known to fail. This might seem like a ridiculous optimisation until you appreciate that your entire network's capacity planning might depend on decisions being made by CDNs and enterprise DNS providers, and anything you can do to avoid mistakes in those third-party systems is another week you get to keep your customers.
>
> In short, the status quo is extremely slippery, architecturally messy, frequently non-deterministic and high-overhead, one that has been observed to put the burden of performance measurement and reporting on unrelated third-party Internet users, chasing an end-result that is often still not doing a very good job.
>
> As far as DNS geo-mapping for the purposes of content steering today is concerned, the only way is up.
>
> Rearchitecting the whole enterprise DNS and CDN industries is a task on a scale larger than moving from IPv4 to IPv6. However, what we *can* do is take specific components of the problem space and find more-elegant solutions. It seems to me that resolverless DNS has the potential to be one of those. The status quo here is almost not worth defending; some kind of change is needed.
>
Thank you for this detailed explanation!

I assume that web server supporting resolver-less DNS provide only DNS
records for a small number of domain names. This can be a benefit
compared to some traditional resolvers if the workload per cached domain
name increases. In such a situation, the described campus resolver has
difficulties to cache short-lived names or to conduct the additional
overhead of EDNS client-subnet to provide its clients with better/faster
DNS records. Here, web server supporting resolver-less DNS that care
only about a small set of domain names can still attempt to provide its
clients with the best available DNS records. In summary, traditional DNS
resolver may economize their effort per domain name to provide the
clients the best available DNS records, while a web server serving DNS
records for a small number of names is hardly impacted by a larger
workload per domain name.

Are you aware of more aspects of how resolver-less DNS can contribute to
a better geo load-balancing?

Erik

---

From: Joe Abley <[email-masked]>
Date: Sun, 1 Sep 2019 19:28:42 -0400

Hi Erik,

On 1 Sep 2019, at 16:40, Erik Sy <[email-masked]> wrote:

> records for a small number of domain names. [...]
> 

It's difficult to answer that question in any meaningful way, in a world as we are where resolverless DNS is neither clearly specified nor implemented.

I do think that improving traffic engineering opportunities for CDNs and their end-users and taking some of the heavy lifting involved in that out of the DNS would be a worthwhile goal and perhaps even a reason to work on it, though.

Clearly there are a lot of corner cases here that affect operations, security, performance, the whole supporting cast for user experience really, so I don't expect this to be easy. Finding a way to constrain the parts of the namespace that are in play within the TLS bundle to avoid cross-site threats does seem like part of the problem space though, to me.

Joe

---

From: Erik Sy <[email-masked]>
Date: Mon, 2 Sep 2019 22:52:30 +0200

On 8/30/19 07:30, Viktor Dukhovni wrote:
> On Thu, Aug 29, 2019 at 11:28:13PM +0200, Erik Sy wrote:
>
>> On 8/29/19 17:34, Viktor Dukhovni wrote:
>>> Resolver-less DNS as proposed disregards long-standing defenses
>>> against address forgery by third parties,
>> Can you please explain this point?
> IIRC the protocol allows any authenticated server (with Let's
> Encrypt, that's everybody) to return "out-of-bailiwick" answers
> (for any unrelated domain).
>
> Yes, a client's designated resolver can also do that, but some
> clients perform local DNSSEC validation,
Can you name a popular client doing DNSSEC validation in a web browsing
scenario?
>  and in any case there's a
> big difference between trusting name resolution to a set of dedicated
> resolvers, vs. each and every server one happens to visit (by
> clicking on a malicious link for example).
Note, that in resolver-less DNS the client does not trust received DNS
records like it is common for the traditional DNS. Instead, the client
validates these DNS records.
>
>>>  breaks geo load-balancing,
>> No, there are no new issues with geo load-balancing compared to using a
>> public DNS resolver.
> Well, the public DNS resolvers (which I do not use on my network)
> typically pass an EDNS0 client-subnet option to the upstream
> authoritative server.  I don't recall anything similar in the paper.
It is in the paper (see Section 4.3).
>
> But if we're doing away with geo load-balancing, and moving to only
> using anycast IPs to reduce latency, that's fine by me, there are
> other more important problems.
>
>>> It introduce a new cache-poisoning channel, and surprising differences
>>> between the IP addresses a browser might use to reach a site from cold
>>> start vs. after visiting some unrelated site.
>> Also traditional DNS resolvers are vulnerable to cache-poisoning attacks.
> Not with DNSSEC validation,
Do you mean the client is doing DNSSEC validation?
>  and modestly difficult off-path.  But
> with this proposal, you get cache poisoing by design from any
> off-path server the client happens to visit.

Do you mind describing an concrete attack?

>
>>> In the case of IPv6 it can be used to fingerprint and track clients by
>>> giving them ephemeral client-specific addresses (in the server's /64
>>> or broader prefix) for third-party servers, and then proxying their
>>> connections (at layer 4) to the real server, while tracking the
>>> client's access to each site.
>> This tracking mechanism is also feasible using traditional DNS.
> But not available to every off-path server I visit.
It is an easy task for an web server to track user visits to other sites
by using for example redirects as Google Search is doing.
>   My resolver
> does not do that. 
First, it is quite uncommon that people run their own DNS resolver.
Second, running an own DNS resolver generates new privacy and
performance problems.
>  Your proposal would delegate this ability
> to every random server I visit.  That's a security fail.

I do not understand why this should be a security fail. In my opinion it
is not even an additional privacy failure because the privacy leak
existed before using common techniques such as redirects. 

Erik

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 2 Sep 2019 17:32:41 -0400

> Can you name a popular client doing DNSSEC validation in a web browsing

Any client behind a nearby (local or private network) validating
resolver.

> validates these DNS records.

No, it validates that the server on the other end of the connection
has the expected certificate, which is rather different from the
actual addresses being valid.

> Do you mean the client is doing DNSSEC validation?

In most cases the resolver, not the client.  But the server may be
sufficiently local, and it is much easier for attackers to entice
users to visit a hostile site than to perform a off-path cache
poisoning attack.

> Do you mind describing an concrete attack?

The hostile website vends fake IPv6 addresses for domains it does
not control (in an address range it does control) to the browser.
When the user later visits one of the cache poisoned sites, the
connection is tracked.  Further surprising to the user is that
unlike visits to links via the site tracked via redirects, the
target domain need not even be one vended in any link from the
hostile site.  Use of a TTL captures furthe interaction with the
remote site, not just the initial connection.

> by using for example redirects as Google Search is doing.

Resolverless-DNS as proposed makes this far more troubling.  I don't
have to click on Google's tracked links.  But as proposed, a website
can arbitrarily populate my cache for destinations it wants to
monitor (for some time after I've long left the original site, given
that the records have a TTL).

> First, it is quite uncommon that people run their own DNS resolver.

Times change.  Some security-conscious users are starting to use
more secure home routers:

	https://www.turris.cz/en/omnia/

which include a sufficiently local validating resolver, that
communicates with a trusted upstream via DoT.

> Second, running an own DNS resolver generates new privacy and

FWIW, my validating resolver forwards to (load shares its queries
across) multiple public validating resolvers (Cloudflare, Google,
Verisign and Quad9).  So there are no new privacy issues, just
a reasonable expectation that validation happens sufficiently
close to my browser to make cache poisoning rather difficult.

> existed before using common techniques such as redirects.

The scope is much worse, as explained above.

-- 
	Viktor.

---

From: Erik Sy <[email-masked]>
Date: Tue, 3 Sep 2019 23:32:12 +0200

> Any client behind a nearby (local or private network) validating
It seems like in your described scenario the resolver is doing DNSSEC
validation and not the client.
> actual addresses being valid.

Most prominently the DNS resolves domain names into IP addresses. In my
view, a DNS record is certainly valid if I can reach the correct host at
the provided IP address. Note, that DNSSEC allows only to validate the
integrity of the DNS record but not that the IP address is valid for a
given domain name. Resolver-less DNS does not need to trust the
authoritative nameserver because it really validates that a given host
can be reached at the presented IP address.

> users to visit a hostile site than to perform a off-path cache
Note, that the attack is even easier for a hostile DNS resolver because
the user does not need to be enticed to visit a hostile site.
> hostile site.

The paper recommends in Section 3.2.2., that browsers should restrict
the context in which they use retrieved DNS records to the same website.
This prevents your described attack. I understand that your described
tracking mechanism is feasible but it is also feasible in a scenario
without resolver-less DNS. Furthermore, learning that the user follows a
provided link has never been very difficult.

> that the records have a TTL).
Not feasible as explained above.
> close to my browser to make cache poisoning rather difficult.

In my view, your described setup is not a traditional self-hosted DNS
resolver. From a privacy perspective, I think that sharing the DNS
lookups across a small number of entities can lead to a situation where
each of these entities may be able to reconstruct a large share of your
online activities.

Erik

---

From: Viktor Dukhovni <[email-masked]>
Date: Tue, 3 Sep 2019 18:32:50 -0400

> On Sep 3, 2019, at 5:32 PM, Erik Sy <[email-masked]> wrote:
> 

That's why I avoid hostile resolvers, which is much easier than avoiding
ever browsing a hostile website.

-- 
	Viktor.

---

From: Bob Harold <[email-masked]>
Date: Wed, 4 Sep 2019 08:58:18 -0400

> the context in which they use retrieved DNS records to the same website.

I find in Section 3.2.2:  "As a result, we recommend
user agents to restrict the contexts in which they trust the DNS records
retrieved via resolverless DNS. For example, web browsers can restrict the
usage of such DNS records within the context of the same browser tab or the
same website."

But I don't understand what is meant by "the same website".  If it means
the same hostname, then the DNS entry has already been found, to get to the
website, and there is no need for resolverless DNS or any DNS lookup.  What
else could it mean?

-- 
Bob Harold

> Resolverless-dns mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/resolverless-dns
>

---

From: Erik Sy <[email-masked]>
Date: Wed, 4 Sep 2019 15:51:55 +0200

On 9/4/19 14:58, Bob Harold wrote:
> user agents to restrict the contexts in which they trust the DNS
> records retrieved via resolverless DNS. For example, web browsers can
> restrict the usage of such DNS records within the context of the same
> browser tab or the same website."
>
> But I don't understand what is meant by "the same website".  If it
> means the same hostname, then the DNS entry has already been found, to
> get to the website, and there is no need for resolverless DNS or any
> DNS lookup. 
This is correct if we speak about a simple website that does not require
the user agent to load resources from other hostnames.
> What else could it mean?

Note, that the retrieval of a popular website requires on average
connections to about 20 different hostnames. Thus, if the first
connection to an average website provides you the DNS records for the
remaining 19 hostnames then using resolver-less DNS can save up to 19
traditional DNS lookups.

Erik

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 4 Sep 2019 10:50:18 -0400

> remaining 19 hostnames then using resolver-less DNS can save up to 19

A more conservative "scope" for any such address hints would be to
limit them to just to the display of the current "location" (web
page).  When the browser navigates to a new page, any such cache
should be flushed.  This should include navigation via links on the
page.  Only the retrieval of resources needed to display the current
page would then make use any returned addresses.

Many users re-use existing tabs to navigate to new sites independent
of the one previously displayed in the tab.

Some of the concerns I have raised are ameliorated if the scope of
the returned addresses is limited to just the page being displayed.
That way, one perhaps gets faster loading of images, Javascript
(and advertisements, if not blocked).

That said, browsers do DNS lookups in parallel, and the user's
resolver will often already have the answers in its cache.  Is the
resulting reduction in latency really worth the added complexity
and potential risks?

-- 
	Viktor.

---

From: Joe Abley <[email-masked]>
Date: Wed, 4 Sep 2019 11:10:29 -0400

> resulting reduction in latency really worth the added complexity

I think the context here should usefully be those sophisticated web pages that operate using large numbers of unique names whose only purpose relates to a particular (user, application) pair, and in some cases only have a single-time use (for which caching is just a waste of RAM).

Many single-use names have low or zero TTLs in order to signal that they should not be cached, but many caches impose local policy on minimum TTLs and hence they are cached anyway. The impact on caches can be significant [Dagon]. The queries have been observed to continue to circulate through the DNS globally, regardless of their intended single-use [Huston].

I think it's straightforward to imagine that many, perhaps most of the world's web content would not load more quickly/reliably/pleasingly just because of the introduction of resolverless DNS. However, scale matters and in a world of elephants and mice, the effect on the small number of elephants might matter more than the lack of effect on the rodents.

Joe

---

From: Erik Sy <[email-masked]>
Date: Wed, 4 Sep 2019 23:23:41 +0200

> resulting reduction in latency really worth the added complexity
Research work indicates that about 5% of DNS transactions take longer
than 100ms [1]. Furthermore, it is known that an additional delay of
100ms of the website load time reduces the conversion rate by about 7%
[2]. These facts highlight that achieving a fast website load time
really matters and that additional DNS lookups can significantly delay
the website load time. In my view, the performance gains of
resolver-less DNS are really worth it.

Erik

1: http://www.icir.org/mallman/papers/passive-dns-ccr13.pdf

2:
https://www.akamai.com/uk/en/about/news/press/2017-press/akamai-releases-spring-2017-state-of-online-retail-performance-report.jsp

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 4 Sep 2019 17:56:47 -0400

> On Sep 4, 2019, at 5:23 PM, Erik Sy <[email-masked]> wrote:
> 

You lose me at "conversion rate", and other exploitative metrics.

If most of the benefit (to users, not advertisers) can be obtained
while limiting the scope to the current page only, consider specifying
such a narrow scope.

If the bandwidth savings only result in users seeing more ads per
page, then I'd rather have the latency.

-- 
	Viktor.

---

From: Erik Sy <[email-masked]>
Date: Thu, 5 Sep 2019 00:16:19 +0200

> You lose me at "conversion rate", and other exploitative metrics.

Note, that the linked study refers to online shopping. You may
understand it as the revenue of online shops increases by 7% if the
website load time decreases by 100ms.

Erik

---

From: Viktor Dukhovni <[email-masked]>
Date: Wed, 4 Sep 2019 18:38:38 -0400

> On Sep 4, 2019, at 6:16 PM, Erik Sy <[email-masked]> wrote:
> 

Perhaps my original point was not clear:

  * I don't choose browsers by looking to maximize the revenue of online stores.
  * Nor should be a priority for the IETF.

But this is perhaps a distraction, let's move on...

-- 
	Viktor.

---

From: Paul Vixie <[email-masked]>
Date: Wed, 04 Sep 2019 18:53:33 -0400

Viktor, i think "time to next eyeball" is the metric that's driving this. it's bad engineering, insecure, and a layering violation. only the money matters. Paul

⁣Get BlueMail for Android ​