Subject: Re: [Add] [EXTERNAL] Re: My longer list of questions [from partial distribution at the MIC]

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 1 Aug 2019 12:15:01 +0200 (CEST)

On Wed, 31 Jul 2019, Alec Muffett wrote:

> On Wed, 31 Jul 2019, 14:02 Mikael Abrahamsson, <[email-masked]> wrote:
>
>> I feel we're back in the bad old days where a browser would just change
>> what default browser I want to use, just because I happened to install
>> that browser. Those kinds of policy choices are for me explicitly to make,
>> not for the browser vendor to do for me.
>
> So, how do you feel about a coffee shop owner, ISP, or national government
> doing the same thing to you?

I decide what I feel about it when I connect. If I don't trust the network 
I am connecting to, I'm going to use a VPN to encrypt everything or I 
might not use that network at all.

If I was at a hotel network I'd probably trust some quad-X provider more 
than the hotel network. If I am at home, I'd trust my ISP more. These are 
policy decisions I want to be able to make on a case by case basis.

-- 
Mikael Abrahamsson    email: [email-masked]

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 1 Aug 2019 12:25:30 +0200 (CEST)

On Wed, 31 Jul 2019, Stephen Farrell wrote:

> In much of the discussion on this list, I see what I perceive as people 
> (on all "sides") frequently describing DNS setups in ways that seem 
> tailored to suit their argument and not reflect reality with any 
> fidelity. I don't find any conclusions based on such arguments at all 
> convincing TBH, no matter how frequently those may be asserted.
>
> And I have a hard time seeing any WG working on this topic being able to 
> make progress in the face of such dodgy arguments.

You shouldn't see these as "dodgy arguments". It's just a reflection of 
very different realities that people live in. I travel between realities 
and there are cases where I want the system resolver to be used, and there 
are cases where I would want DoH to some quad-X provider to be used.

There is no single right answer to this. In windows there is a popup after 
I connect to a new network, asking me whether this is "public" or 
"private". I guess this is an proxy for "do you trust this network and the 
devices that are on it?" and the answer if I choose public is "no". 
Security settings (firewall and perhaps more) are under the hood set 
accordingly.

We're going to need similar settings here based on user input. I'd even 
think it was ok for the browser vendor to have sensible defaults per 
network type.

Only the user knows who they trust in each case. We have to ask them.

---

From: "Ralf Weber" <[email-masked]>
Date: Thu, 01 Aug 2019 13:11:49 +0200

Moin!

On 1 Aug 2019, at 0:02, Stephen Farrell wrote:

> On 31/07/2019 22:29, Ralf Weber wrote:
>> Long story short there is no reason for me
>
> It's fine that you conclude that.
>
>> or any other european
>
> That is incorrect. I do use CF's DoT and other DoT
> services and occasionally CF's DoH. Some of that is
> just playing about/testing, but my home ISP did have
> an incident last year where their DNS went awol for
> a weekend (seemingly related to the root key roll;-)
> so I set my home router to use CF's DoT. I've not
> bothered to reset it since as my view of the choice
> is between two for-profit foreign-owned companies
> and in fact I know more about CF than the latest
> in a long line of investors who happen to own the
> ISP today. I make no claim that that is common but
> it is an existence proof that your claim above is
> false.
Well I was not specific enough here. The argument
being made here is that ISP do bad things to my
data and therefore have to be avoided. While this
may be true in other part of the world it certainly
is not the case in Europe. That’s what I meant by
there is at least not that reason. Now there might
be other reasons (RFC5011 is hard ;-) and I don’t
think anyone here has a problem if an informed user
switched it’s DNS via a setting or VPN or whatever,
I just don’t want to have the browser do it for me.

> In much of the discussion on this list, I see what
> I perceive as people (on all "sides") frequently
> describing DNS setups in ways that seem tailored
> to suit their argument and not reflect reality with
> any fidelity. I don't find any conclusions based
> on such arguments at all convincing TBH, no matter
> how frequently those may be asserted.
Well there are multiple realities here and the hard
part will to find something that matches all or at
least most of them. There also might not be a single
solution for that.

So long
-Ralf
—--
Ralf Weber

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 1 Aug 2019 12:40:36 +0100

Hiya,

On 01/08/2019 11:25, Mikael Abrahamsson wrote:
>> In much of the discussion on this list, I see what I perceive as
>> people (on all "sides") frequently describing DNS setups in ways that
>> seem tailored to suit their argument and not reflect reality with any
>> And I have a hard time seeing any WG working on this topic being able
>> to make progress in the face of such dodgy arguments.
> 
> You shouldn't see these as "dodgy arguments". 

I guess we disagree. (BTW, I mean "dodgy" in the sense of
"bad logic" rather than "being deliberately disingenuous")

> It's just a reflection of
> and there are cases where I want the system resolver to be used, and
> there are cases where I would want DoH to some quad-X provider to be used.
> 
> There is no single right answer to this. In windows there is a popup
> after I connect to a new network, asking me whether this is "public" or
> "private". I guess this is an proxy for "do you trust this network and
> the devices that are on it?" and the answer if I choose public is "no".

Mostly agree. ISTM though that better and fairly widespread
OS support for that kind of subtlety would be needed before
applications that need to run on different OSes could really
benefit without having to do their own DoH stuff, and that's
not something that the IETF can really "fix."

> Only the user knows who they trust in each case. We have to ask them.

I'm less sure of that. Given most people don't know what
DNS is, nor the trade-offs involved, it's unclear how to
effectively ask what question. Someone could do research
of course that helps there, but I'm not convinced we could
write BCP-like text at the moment.

Cheers,
S.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 1 Aug 2019 14:12:53 +0200 (CEST)

> "bad logic" rather than "being deliberately disingenuous")

Thanks for the clarification, I did interpret your statement that you 
thought the arguments were disingenous which I thought was strange.

> not something that the IETF can really "fix."

Sure.

> write BCP-like text at the moment.

It seems most people agree that this is hard. The disagreements are about 
what to do when it's hard. Some say "it's hard, so let's just change it to 
DoH server that the browser vendor decides unless the user makes an active 
decision" and others seem to say "it's hard, so let's keep the status quo 
(system resolver) unless the user makes an active decision".

Doing browser/operating system integration is hard for most if you're not 
a vendor that happen to do both (there are a few). That doesn't mean it's 
not the right thing to do long term.

One thing that would be good to get agreement on is that there are valid 
use-cases for both using the system resolver by default and using a 
(remote, non-ISP) DoH resolver by default, and my estimation is that these 
are close to equally large/important. If we can agree on that and the 
discussion stops being on whose use-case is most important and instead 
being on how to solve this problem, then the discussion might be more 
fruitful and productive.

The IETF role in this I would say is that it's one of the few places where 
application/browser vendors and operators are in the same "place". So 
coming up with a BCP that is balanced, and works for a lot of different 
use-cases and make applications behave in a similar fashion across the 
board would be of great benefit.

---

From: Stephen Farrell <[email-masked]>
Date: Thu, 1 Aug 2019 13:54:33 +0100

Hiya,

On 01/08/2019 13:12, Mikael Abrahamsson wrote:
> (remote, non-ISP) DoH resolver by default, and my estimation is that
> these are close to equally large/important. If we can agree on that and
> the discussion stops being on whose use-case is most important and
> instead being on how to solve this problem, then the discussion might be
> more fruitful and productive.

Ah, interesting. FWIW I personally agree there are valid
use-cases for both, and maybe more too, e.g. cases where
an application may try DoH but fall back to the OS because
of <foo>. But I don't think documenting that'd remove the
disagreements. I wonder would it help to try to separately
describe the main abuse-cases as well? If we had objective
descriptions of both use- and abuse- cases maybe it'd be
easier to figure out what kinds of defaults and/or inputs
(e.g. user or admin or app selection) would be needed to
indicate that a particular setup was reasonable?

(Could end up a bit theoretical I guess, but I'm interested
in the general question of whether considering abuse-cases
might help in situations like this.)

Cheers,
S.

---

From: Tommy Jensen <[email-masked]>
Date: Thu, 1 Aug 2019 16:41:41 +0000

+1 for different valid use cases regarding default choice. Such a document, even attempted, would give a more concrete framework for this discussion anyway. I'd happily collaborate.

Thanks,
Tommy
________________________________
From: Add <[email-masked]> on behalf of Stephen Farrell <[email-masked]>
Sent: Thursday, August 1, 2019 5:54 AM
To: Mikael Abrahamsson <[email-masked]>
Cc: Paul Wouters <[email-masked]>; ADD Mailing list <[email-masked]>; Ralf Weber <[email-masked]>; Alec Muffett <[email-masked]>
Subject: Re: [Add] [EXTERNAL] Re: My longer list of questions [from partial distribution at the MIC]

Hiya,

Cheers,
S.

---

From: "Diego R. Lopez" <[email-masked]>
Date: Thu, 1 Aug 2019 16:51:19 +0000

Hi,

On 01/08/2019, 14:55, "Add on behalf of Stephen Farrell" <[email-masked] on behalf of [email-masked]> wrote:

I must say that this analysis of abuse-cases sounds like an interesting way to go, and even probably become a future practice for security analysis, much like current threat models.

--
"Esta vez no fallaremos, Doctor Infierno"

Dr Diego R. Lopez
Telefonica I+D
https://www.linkedin.com/in/dr2lopez/

e-mail: [email-masked]
Tel:         +34 913 129 041
Mobile:  +34 682 051 091
----------------------------------

Este mensaje y sus adjuntos se dirigen exclusivamente a su destinatario, puede contener información privilegiada o confidencial y es para uso exclusivo de la persona o entidad de destino. Si no es usted. el destinatario indicado, queda notificado de que la lectura, utilización, divulgación y/o copia sin autorización puede estar prohibida en virtud de la legislación vigente. Si ha recibido este mensaje por error, le rogamos que nos lo comunique inmediatamente por esta misma vía y proceda a su destrucción.

The information contained in this transmission is privileged and confidential information intended only for the use of the individual or entity named above. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this transmission in error, do not read it. Please immediately reply to the sender that you have received this communication in error and then delete it.

Esta mensagem e seus anexos se dirigem exclusivamente ao seu destinatário, pode conter informação privilegiada ou confidencial e é para uso exclusivo da pessoa ou entidade de destino. Se não é vossa senhoria o destinatário indicado, fica notificado de que a leitura, utilização, divulgação e/ou cópia sem autorização pode estar proibida em virtude da legislação vigente. Se recebeu esta mensagem por erro, rogamos-lhe que nos o comunique imediatamente por esta mesma via e proceda a sua destruição

---

From: Brian Dickson <[email-masked]>
Date: Thu, 1 Aug 2019 11:31:00 -0700

On Wed, Jul 31, 2019 at 5:12 AM Livingood, Jason <
[email-masked]> wrote:

> On 7/29/19, 7:30 AM, "Add on behalf of Paul Wouters" <[email-masked]
> on behalf of [email-masked]> wrote:
> >    I’m not a fan of the centralization. I am a fan of removing blind
> trust on unknown parties.
>
> The potential for an unknown party seems to arise when an app changes from
> the user's typical resolver to some unknown-to-the-user party, so I agree
> this is a risk.
>

For me, one of the major concerns is the unintended breakage (consequences)
of changing defaults or overriding existing behavior/configuration.

The analogy I'd like to reference, is from an earlier situation in DNS:
wildcard records in TLDs.

(This might be before the time of many current folks' involvement in DNS,
so I'll give a very brief, probably incomplete and/or inaccurate, summary
of the issue then.)

Verisign deployed a wildcard A record at the apex of the ".com" and ".net"
TLD zones in 2003.
This was a huge controversy, largely because of things that broke as a
result.
These wildcards were removed a relatively short time later, and everyone
agreed that it was not a good idea to do things like this. (Google "Site
Finder" for all the gory details, history, etc.)

I see this as a close analogy, and a strong precedent that should be
observed/followed.

Some examples of how/why SiteFinder was bad, have to do with behavior that
is expected and relied upon, as part of things like DNS search lists which
rely on NXDOMAIN responses. Wildcards at the same level as a missing name
always match, so this broke a lot of things. Also of note was the impact on
SMTP traffic, since mail transport relies on using both MX and A records,
while the wildcard A record wasn't answering on the SMTP port.

How is SiteFinder relevant? The particular use case of enterprise DNS is
impacted in a very similar way. The only difference is the underlying
mechanism: changing the DNS resolver, rather than wildcards on authority
servers. The effect is roughly equivalent: answers that are incorrect or
unexpected, from the perspective of the user, as a result of a change not
explicitly done by the user or the user's administrator (host, network,
DNS, or other kind of admin).

Whatever kinds of things we do with/to/in DNS, we need to be cognizant of
all consequences, intended or otherwise. In particular, major changes in
behavior MUST adequately protect against unintended breakage in a fail-safe
manner. This may include mandating that the changes be opt-in only, rather
than any other model (opt-out, require active policy management on hosts,
force MITM inspection of all HTTPS traffic, etc.)

Brian

---

From: Michael Richardson <[email-masked]>
Date: Thu, 01 Aug 2019 16:26:27 -0400

Mikael Abrahamsson <[email-masked]> wrote:
    > One thing that would be good to get agreement on is that there are
    > valid use-cases for both using the system resolver by default and using
    > a (remote, non-ISP) DoH resolver by default, and my estimation is that
    > these are close to equally large/important.

As a user, I'd sometimes like it on a tab-by-tab basis!
Certainly "Incognito" browsing might have a different policy.
{I'll also say if you are a dissident trying to hide/blend-in in a hostile
environment, the last thing you want is all your Do53 traffic to do is disappear.}

    > If we can agree on that and
    > instead being on how to solve this problem, then the discussion might
    > be more fruitful and productive.

Agreed.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Michael Richardson <[email-masked]>
Date: Thu, 01 Aug 2019 20:04:05 -0400

Brian Dickson <[email-masked]> wrote:
    > Verisign deployed a wildcard A record at the apex of the ".com" and
    > ".net" TLD zones in 2003.  This was a huge controversy, largely because
    > of things that broke as a result.  These wildcards were removed a
    > relatively short time later, and everyone agreed that it was not a good
    > idea to do things like this. (Google "Site Finder" for all the gory
    > details, history, etc.)

    > Some examples of how/why SiteFinder was bad, have to do with behavior
    > that is expected and relied upon, as part of things like DNS search
    > lists which rely on NXDOMAIN responses. Wildcards at the same level as
    > a missing name always match, so this broke a lot of things. Also of
    > note was the impact on SMTP traffic, since mail transport relies on
    > using both MX and A records, while the wildcard A record wasn't
    > answering on the SMTP port.

    > How is SiteFinder relevant? The particular use case of enterprise DNS
    > is impacted in a very similar way. The only difference is the

A good explanation, thank you.

And, having not learned at all from the SiteFinder disaster, a bunch of ISPs
decided to implement this in their recursive resolvers in order to drive
traffic to their hosted ad sites.

This was probably one of the things that created the need for the quadX
resolvers.

---

From: Matt Larson <[email-masked]>
Date: Tue, 6 Aug 2019 12:50:19 -0400

> On Aug 1, 2019, at 2:31 PM, Brian Dickson <[email-masked] <mailto:[email-masked]>> wrote:
> 
> Also of note was the impact on SMTP traffic, since mail transport relies on using both MX and A records, while the wildcard A record wasn't answering on the SMTP port.

Speaking as someone who was there at the time and has the scars and hate mail to prove it, there was indeed an SMTP agent answering on tcp/25 at the IP address returned by the .com and .net wildcard A records. This agent returned a 550 response code to RCPT commands. (Well, it was a bit more complicated than that. The high-level details are in the public record; see http://www.circleid.com/posts/site_finder_the_technical_legal_privacy_concerns/ <http://www.circleid.com/posts/site_finder_the_technical_legal_privacy_concerns/>.)

I am not defending Site Finder nor commenting on its merits or shortcomings; I'm just making this neutral comment in the interest of technical accuracy.

Matt

---

From: Bret Jordan <[email-masked]>
Date: Tue, 6 Aug 2019 22:48:41 -0600

I for one am terrified of the possibility of mass centralization of DNS traffic.  Especially given this is will be over http now and an app or client could include a plethora of other http headers that could give guaranteed per user tracking.  Further, once all DNS is in one place, that makes a great target.  

Bret 

Sent from my Commodore 64

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

> On Aug 1, 2019, at 10:41 AM, Tommy Jensen <Jensen.Thomas=[email-masked]> wrote:
> 
> Add mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/add

---

From: Rob Sayre <[email-masked]>
Date: Tue, 6 Aug 2019 23:34:13 -0700

On Tue, Aug 6, 2019 at 9:48 PM Bret Jordan <[email-masked]> wrote:

> I for one am terrified of the possibility of mass centralization of DNS
> traffic.
>

https://tools.ietf.org/html/rfc1958#section-6
"Confidentiality and authentication are the responsibility of end users and
must be implemented in the protocols used by the end users."

Obviously, the various DNS working groups haven't done very well on this
point. How would you differentiate confidentiality and encryption from
centralization? I mean, I guess the ability to intercept and change a DNS
response is decentralization in some sense, but it doesn't seem clear that
it's a good thing.

> Especially given this is will be over http now and an app or client could
> include a plethora of other http headers that could give guaranteed per
> user tracking.
>

I am not sure this is a valid point. Most HTTP header fingerprinting uses
features that needn't be included in a DoH query.

Further, once all DNS is in one place, that makes a great target.

It's certainly a new target, but DNS is in the clear for the most part. It
seems strange to imply this is a new weakness.

thanks,
Rob

---

From: Mikael Abrahamsson <[email-masked]>
Date: Wed, 7 Aug 2019 08:52:38 +0200 (CEST)

On Tue, 6 Aug 2019, Rob Sayre wrote:

> centralization? I mean, I guess the ability to intercept and change a 
> DNS response is decentralization in some sense, but it doesn't seem 
> clear that it's a good thing.

Of the people I talk to, nobody/few seem to be opposed to transport 
encryption of DNS queries. I hear from multiple other operators "sure, we 
can DoT enable our existing resolvers". DoH-enablement is typically also 
viewed as ok.

What people are opposed to is to steer away the resolver duty from the 
ISPs (typically handed out via DHCP) to something else without asking the 
user. Having the entire world of browser users all of a sudden use a 
quad-X service run by a handful of cloud providers makes a lot of people 
very concerned.

This is the decentralisation point we're talking about when it comes to 
DNS. Today there are probably tens/hundreds of thousands of entities 
running recursive DNS resolvers. In the proposed future of "all browsers 
will use one of a few DoH providers" this is no longer the case. If the 
lookups are encrypted or not, is less of a concern point.

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 00:02:05 -0700

On Tue, Aug 6, 2019 at 11:52 PM Mikael Abrahamsson <[email-masked]> wrote:

> ISPs

That does seem like the main issue.

But, the IETF defines protocols. It doesn't standardize which endpoints
must be consulted.

thanks,
Rob

---

From: Christian Huitema <[email-masked]>
Date: Wed, 7 Aug 2019 02:25:08 -0500

On 8/7/2019 1:34 AM, Rob Sayre wrote:

> It's certainly a new target, but DNS is in the clear for the most
> part. It seems strange to imply this is a new weakness.

That's true. But then, centralization would have other consequences,
such as a go-to place for censorship.

We have to assume that DoH and application level DNS is happening, just
like application level transport is happening. But we should also look
at ways to defeat centralization even when DoH is happening. That
probably mean relying more on the individual web sites, which have a
vested business interest in not providing their metadata streams to the
central services (or at least not for free). It probably also means some
form of caching and recursive resolving in the applications themselves.

To give an example, suppose that applications cache the ESNI record for
a particular "hidden" server. Suppose also that the record is not short
lived. For the duration of the record's TTL, the application does not
need to emit a DNS request for the hidden server. It just need to do
real time queries for the fronting server. And when it does that, it
does not expose anything more than what can be revealed with the TLS
connection through the fronting server.

Then go one step further. Suppose that the application knows the
preferred DoH server for the fronting server. Then it can send the real
time requests for that fronting server there, instead of relying on any
big centralized service. Do that for enough destination, and the central
service will only see a rather unteresting fraction of the DNS traffic.

There are probably many holes in my back-of-the-envelope design. But it
shows at least one thing: if we set up to find mitigations for the
centralization, we can most probably find some. And we may be able to
roll out these mitigation before the dreaded centralization happens.

-- Christian Huitema

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 00:44:14 -0700

On Wed, Aug 7, 2019 at 12:25 AM Christian Huitema <[email-masked]>
wrote:

> That's true. But then, centralization would have other consequences, such
> as a go-to place for censorship.
>

I don't think that's the case. As Mozilla's presentation stated: DNS is not
an effective control surface.

Even in an environment with ubiquitous, application-specific DoH, the PKI
system would still be fairly centralized. That would be the go-to place for
censorship, not that I think it's a particularly relevant issue for a DNS
mailing list.

thanks,
Rob

---

From: Andrew Campling <[email-masked]>
Date: Wed, 7 Aug 2019 10:11:22 +0000

On Wed, Aug 7, 2019 at 08:44 AM Rob Sayre <[email-masked]> wrote:

> As Mozilla's presentation stated: DNS is not an effective control surface.

It may have stated that, however that doesn’t make it true.

Andrew

---

From: "Ralf Weber" <[email-masked]>
Date: Wed, 07 Aug 2019 14:11:53 +0200

Moin!

On 7 Aug 2019, at 9:44, Rob Sayre wrote:

> I don't think that's the case.
So there are currently three big providers of globally available DoH 
servers (Google, Cloudflare, Quad9). All of them are big US companies. 
Compare that to thousands of ISPs worldwide. Given that behaviour of US 
organisations caused the increased encryption efforts in the IETF 
centralising central naming to US companies to me seems the wrong way as 
it is now much easier for those organisations to get this data.

The problem is that there are not a lot of commercial incentives to run 
a scalable global DNS/DoX service other then mining the data or getting 
direct interaction with the end user, and none of these are privacy 
goals. ISPs have a natural interest of supplying a good DNS service as 
it is a part of the overall Internet service they supply and the faster 
these are the more attractive they are for users.

> an effective control surface.
Well the fact that Mozilla stated it doesn’t make it a true statement. 
As with all things security there is no 100% effective security solution 
or control surface. There may be different levels needed to attack the 
control surface and DNS as a control surface is the sweet spot as:
- It is needed for most transactions on the Internet
- It is relatively easy to enforce it (redirect/block port 53)
- It is possible to circumvent, but with a moderate amount of effort 
(VPN)
Which is why it is used a lot for security (It’s an onion), parental 
controls, enterprises and governmental use.

Having worked with lawmakers in Europe on content control my experiences 
are that they are usually ok with a solution put into law that isn’t a 
100%, but if this solution goes down say 50% they will demand a better 
solution and given that we already had talks here about government 
should be able to decrypt transactions I fear that we could be worse off 
if we are abandoning DNS as a somewhat effective control point.

So long
-Ralf

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 7 Aug 2019 14:28:33 +0100

Hiya,

> servers (Google, Cloudflare, Quad9). All of them are big US companies.

I don't think those 3 organisations are commensurately sized,
nor are they all for-profit if I'm reading [1] correctly. They
are all US-homed I guess though IIRC some quad9 funding was
from the UK. (All 5eyes nonetheless;-().

   [1] https://www.quad9.net/about/

That said, IMO their home location is not at all what matters.

Centralisation is a concern that matters, if it results in
a situation where turning off one of these might break a
lot of things, or where one of them could go rogue but
be too big to turn off.

The possibility that a large actor could correlate DNS traffic
with other traffic is a concern that matters. I think the
TRR policy stuff Mozilla are attempting is interesting as a
way to handle that. It'd be great if there were ways to audit
that, but hard to think of one. (I bet researchers will figure
out ways over time though:-)

The possibility that a large actor might offer such a service
just to ensure that competitors don't get to see the DNS traffic
is interesting, but I'm not sure if it's a concern of mine.

The possibility that such a service provider might favour
some CDNs over others is maybe a concern for me as that may
result in inefficient services. I'll not lose sleep over
that though if it mostly means slower cat videos.

My tentative conclusion is that it'd be good if there were
maybe a dozen or so of those, rather than 3, not all of which
would need a full-global presence (i.e. they could be somewhat
local in the sense of their fine-grained knowledge of CDNs),
as applications doing DNS could decide which to use based on
the application's own location in the network.

I think it'd also be good if applications doing DNS didn't
always use the same service, so that failure of one wouldn't
need a flag day to get fixed, though figuring out a way to
switch sensibly is fairly hard.

Cheers,
S.

PS: As an aside, I worry less about the centralisation in
this case than in the web PKI - I'd really love to see another
letsencrypt equivalent (no cost, not for profit) CA handling
maybe 10% of what LE does, just in case stuff hits the fan
for LE sometime.

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 Aug 2019 09:47:28 -0400

On 7 Aug 2019, at 00:48, Bret Jordan <[email-masked]> wrote:

> I for one am terrified of the possibility of mass centralization of DNS traffic.

Two dozen resolver systems are responsible for some 80% of all traffic seen at ORG servers, today. Somewhere between 15 and 20% of all end-user systems use Google Public DNS for resolution. This already feels like mass centralisation to me.

Apologies if I've triggered any terror.

Joe

---

From: "Livingood, Jason" <[email-masked]>
Date: Wed, 7 Aug 2019 14:37:58 +0000

From: Add <[email-masked]> on behalf of Rob Sayre <[email-masked]>

> I don't think that's the case. As Mozilla's presentation stated: DNS is not an effective control surface.

You can say that, as can Mozilla, but the market & deployment data suggests the complete opposite. I know you have a strongly held contrary view but I think you are in the rough on this one and hope we can move on.

Jason

---

From: "Livingood, Jason" <[email-masked]>
Date: Wed, 7 Aug 2019 14:44:14 +0000

> But, the IETF defines protocols. It doesn't standardize which endpoints must be consulted.

In actuality, the IETF never really ignores how protocols are implemented. There is an Ops area and we have groups like v6ops and dnsop that focus on implementation of protocols. Importantly, during early scale deployments (e.g. IPv6) the ops/implementation WGs provide protocol feedback to the protocol-design WGs, which is a great & virtuous feedback loop. After all the IETF is about rough consensus *and* running code. As with my last email, I hope we can note your disagreement and move on – there is enough of a critical mass of folks with rough consensus on this subject and interest in doing productive work. :-)

Jason

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 7 Aug 2019 15:56:33 +0100

Hiya,

On 07/08/2019 15:44, Livingood, Jason wrote:
> there is enough of a critical mass of folks with rough consensus on this subject 

I don't believe we have a consensus-caller to hand for this so
don't see how you can be confident in that. As it happens, and
fwiw, I'm pretty sure I see a fair bit of a lack of consensus
as well. (Though like us all on this list, I'm just another
subscriber, this not being a WG list or anything.)

Cheers,
S.

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 15:22:59 +0000

Joe

While the top 80% being from 2 dozen resolvers there are others statistics that would be interesting and perhaps a good measurement of current DNS Resolver diversity. 

We know there are a great many small resolvers in the world ranging from those run by large and small corporate and organization IT, medium and small ISPs, all the  way down those run by individuals.   

Individually they don’t generate a lot of traffic, but they that’s ok as DNS was never designed to be centralized. Broad diversity has long been a fundamental strength of the system.  

The actual exact number of these long tail resolvers isn’t yet known but some estimates have put it in the 1 million to maybe 5-10 million range.  It could even be higher.   This large number of first level resolvers represents an element of great diversity which strengthens the DNS for users. 

So if you’re looking at data is it possible to say how big the tail is?  After the first 80%, how many different resolvers show up in the next 5%, 10%, 15%, 20% buckets?  What’s the total number of diverse resolvers?

For while measuring big users is a good measure of centralization, measuring small is perhaps a good measure of today’s infrastructural diversity and resiliency. 

-glenn

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 09:42:16 -0600

Thanks,
Bret
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050
"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg."

>  
> Especially given this is will be over http now and an app or client could include a plethora of other http headers that could give guaranteed per user tracking.
> 
> I am not sure this is a valid point. Most HTTP header fingerprinting uses features that needn't be included in a DoH query.
> 

An app or client could add additional http headers.  It could add a unique ID for each app or client, it could add in geo-location data, it could all kinds of things.  

> It's certainly a new target, but DNS is in the clear for the most part. It seems strange to imply this is a new weakness.

Having DNS data be in the clear, and having it all be centralized are TWO totally different things.  One could easily argue that having all client talk to an internal DNS servers and having that DNS server talk to the world provides per user confidentiality.

Bret

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 09:46:41 -0600

Thanks,
Bret

> 
> That's true. But then, centralization would have other consequences, such as a go-to place for censorship.
> 
> 

I heard that presentation as well.  But I would like to ask where is the data and research that backs up that assertion?  It seems more like it is the desire of Mozilla or certain people at Mozilla. 

Bret

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 09:49:04 -0600

> On Aug 7, 2019, at 4:11 AM, Andrew Campling <[email-masked]> wrote:
> 
>  
> On Wed, Aug 7, 2019 at 08:44 AM Rob Sayre <[email-masked] <mailto:[email-masked]>> wrote:
>  

Agreed.  At some level I think the browsers are trying to force their will upon the rest of the world.  It kind of feels like the 1990s when a certain OS tried to force everyone into a certain paradigm. 

Bret

---

From: "Dixon, Hugh" <[email-masked]>
Date: Wed, 7 Aug 2019 15:49:20 +0000

Joe - Is your .org data published via DNSOARC ?
How do you deduce the % of end user systems using 8.8.8.8 ?

Hugh

On 07/08/2019, 16:23, "Deen, Glenn (NBCUniversal)" <[email-masked]> wrote:

    Joe

    -glenn
    --
    Add mailing list
    [email-masked]
    https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ietf.org%2Fmailman%2Flistinfo%2Fadd&amp;data=02%7C01%7Chugh.dixon%40sky.uk%7C10835042c1654d25b89708d71b4b2ed1%7C68b865d5cf184b2b82a4a4eddb9c5237%7C0%7C0%7C637007882043370741&amp;sdata=12doPWqesUqFI5%2FeSBvea0Hj1LrsJe00bJkh08Bxrt0%3D&amp;reserved=0

Information in this email including any attachments may be privileged, confidential and is intended exclusively for the addressee. The views expressed may not be official policy, but the personal views of the originator. If you have received it in error, please notify the sender by return e-mail and delete it from your system. You should not reproduce, distribute, store, retransmit, use or disclose its contents to anyone. Please note we reserve the right to monitor all e-mail communication through our internal and external networks. SKY and the SKY marks are trademarks of Sky Limited and Sky International AG and are used under licence.

Sky UK Limited (Registration No. 2906991), Sky-In-Home Service Limited (Registration No. 2067075), Sky Subscribers Services Limited (Registration No. 2340150) and Sky CP Limited (Registration No. 9513259) are direct or indirect subsidiaries of Sky Limited (Registration No. 2247735). All of the companies mentioned in this paragraph are incorporated in England and Wales and share the same registered office at Grant Way, Isleworth, Middlesex TW7 5QD

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 Aug 2019 11:50:27 -0400

Hi Glen!

> So if you’re looking at data is it possible to say how big the tail is?  After the first 80%, how many different resolvers show up in the next 5%, 10%, 15%, 20% buckets?  What’s the total number of…

I agree with you though that the data does show a very long tail. The number of resolvers observed depends on the time base of the observation, since there are a great number that only send queries very occasionally. While the top talkers are easy enough to identify as resolvers, that becomes a little more speculative when you consider the nature of initiators who send one query every two weeks, for example: are those really resolvers with a population of stub resolvers behind them? Is the population just really small? Are the resolvers hopping between addresses? Are they doing something else?

The rapid descent of the histogram into the noise floor still provides good support to the idea that *traffic* is centralised to a small number of clients, though, as viewed from the authoritative server.

More generally, you're not the only one who is interested in more specific details about the traffic we see at ORG, and I hope to be able to publish something less anecdotal before too long. We have a DNS-OARC meeting coming up in November -- perhaps we can slice and dice our traffic stats and have something useful to show the class, there.

Joe

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 7 Aug 2019 16:51:03 +0100

On 07/08/2019 16:42, Bret Jordan wrote:
> An app or client could add additional http headers.  It could add a
> unique ID for each app or client, it could add in geo-location data,
> it could all kinds of things.
An application could as easily leverage qnames and answers
to get as good tracking or covert channel via DoT.

So, again, this is down to trusting the application code
as a DNS client that's allowed to talk to it's selected
recursive... or not. It's not really a DoT/DoH distinction.

That said, I do agree it'd be easier to accidentally enable
tracking via DoH, if one used some library or re-used some
code that did send cookies etc. And as the fine presentation
in pearg demonstrated, http session or other ancialliary
information may make it easier to do traffic analysis on
DoH vs. DoT. I'm not convinced either factor is a long term
thing though.

Cheers,
S.

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 Aug 2019 11:55:55 -0400

Hi Hugh,

I don't want to hijack the thread with a tangent, but identification of traffic that originates at Google and lands at Afilias (the back-end registry provider for PIR) is straightforward, as is confirmation that of that DNS traffic the queries that don't relate to Google Public is a rounding error.

Afilias is not the only provider of authoritative DNS services for PIR, and it's certainly possible that there is some systematic bias in the distribution of query originators observed in just Afilias anycast nodes. However, the elephants are so big and the mice are so small that I remain fairly confident that the observation made is representative of the system as a whole.

As I mentioned before these observations have all been made internally within PIR and Afilias and we don't currently publish them in any way. I am certainly in favour of publishing things, though, so long as we can do so with some degree of rigour and not promote ambiguity.

Joe

> On 7 Aug 2019, at 11:49, Dixon, Hugh <Hugh.Dixon=[email-masked]> wrote:
>

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 09:58:35 -0600

Really great points Glenn.

Thanks,
Bret

> On Aug 7, 2019, at 9:22 AM, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> 
> 
>

---

From: Jim Reid <[email-masked]>
Date: Wed, 7 Aug 2019 17:07:12 +0100

> On 7 Aug 2019, at 16:42, Bret Jordan <[email-masked]> wrote:
> 

Indeed. IMO this sort of behaviour is something the IETF should work on: some do’s and don’ts on privacy issues for DoH clients. Which WG is most appropriate?

Apologies yet again for another meaningful and relevant Subject: header. I’m hoping that practice will catch on. :-)

---

From: "STARK, BARBARA H" <[email-masked]>
Date: Wed, 7 Aug 2019 16:10:18 +0000

> I don't believe we have a consensus-caller to hand for this so don't see how
> you can be confident in that. As it happens, and fwiw, I'm pretty sure I see a
> fair bit of a lack of consensus as well. (Though like us all on this list, I'm just
> another subscriber, this not being a WG list or anything.)

And given IETF structure (where WG chairs get first crack at calling consensus within a WG), this is why I suggested someone who actually operates a network should be a chair (wherever work gets done). Given how little influence network operators have in IETF, how frequently operational concerns are ignored or dismissed as not being worthy of consideration, and how insulting and inflammatory language towards all operators (and, by association, all people employed by all operators) is often tolerated and not called out or discouraged by list moderators, I agree that we need to wait and see what the IESG decides before suggesting there has been a decision regarding "consensus". 

Barbara

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 10:21:18 -0600

I agree 100% Barbara. 

Thanks,
Bret

> On Aug 7, 2019, at 10:10 AM, STARK, BARBARA H <[email-masked]> wrote:
>

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 16:22:17 +0000

Hi Joe

> On Aug 7, 2019, at 8:50 AM, Joe Abley <[email-masked]> wrote:
> 
> While the top talkers are easy enough to identify as resolvers, that becomes a little more speculative when you consider the nature of initiators who send one query every two weeks, for example: are those really resolvers with a population of stub resolvers behind them? Is the population just really small? Are the resolvers hopping between addresses? Are they doing something else?

If we are asking the question of what’s centralized and what’s diversified then it doesn’t matter what they are - they aren’t coming in via a centralized pathway. 

If we were to do this scientifically we might find the mean TTL of the resolved in each bucket and use a bracketed range of that (1,2 and 3 standard deviations) that to find a set of time intervals to look at. 

> The rapid descent of the histogram into the noise floor still provides good support to the idea that *traffic* is centralised to a small number of clients, though, as viewed from the authoritative…

It does show that there is a number of big players - I’m guessing it will have mobile operators and some of the large ISPs in it, but that noise also tells us that there is an existing ecosystem of many diverse
resolvers.  

However this would be very very different from what the patterns form DoH would look like. Diversity will drop off even lower and the long tail may completely vanish. 

Glenn

---

From: Eric Orth <[email-masked]>
Date: Wed, 7 Aug 2019 12:30:44 -0400

(Thanks for branching off to another subject.  I wanted to respond in the
other thread when this was brought up, but it was kinda off-topic for most
of what was being discussed there.)

Agreed.  There is certainly lots of room for a DoT or DoH client to
unintentionally make bad privacy decisions.  Publishing guidance and
information can help ensure they have thought things through before they
make privacy-affecting decisions.  This is a very actionable area for
IETF.  I think there would be enough consensus on many areas of guidance
(but obviously not all) to get a decent sized document.

Not everything would necessarily be a hard and fast do or don't, as
different tradeoffs may be relevant for different applications and use
cases.  That's okay since the important part is providing the information
to help applications make the correct decisions for themselves.  E.g.,
accepting cookies is one of the areas already briefly discussed in RFC8484,
and I think it took the right approach: "HTTP cookies SHOULD NOT be
accepted by DOH clients unless they are explicitly required by a use
case."  I think we need more guidance like that with more detail on the
tradeoffs and considerations.  It may seem obvious that applications should
never send or accept cookies with DoH, but some special-purpose application
may want to use DoH to send authentication information to a special-purpose
DNS server that will only return information to authorized users.  DoH and
DNS are not limited just to the use case of browsers requesting addresses
of all hostnames to be visited.

On Wed, Aug 7, 2019 at 12:07 PM Jim Reid <[email-masked]> wrote:

> unique ID for each app or client, it could add in geo-location data, it
> could all kinds of things.
>
> Indeed. IMO this sort of behaviour is something the IETF should work on:
> some do’s and don’ts on privacy issues for DoH clients. Which WG is most
> appropriate?
>
> Apologies yet again for another meaningful and relevant Subject: header.
> I’m hoping that practice will catch on. :-)
>
> --
> Add mailing list
> [email-masked]

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 Aug 2019 12:32:18 -0400

Hey Glen,

> If we are asking the question of what’s centralized and what’s diversified then it doesn’t matter what they are - they aren’t coming in via a centralized pathway.

For sure. The point I was trying to make (badly :-) was that the relationship between observed upstream traffic and size of downstream client base is not necessarily the same for clusters that we identify as systems acting as resolvers as they are for clusters that might be doing something else.

> If we were to do this scientifically we might find the mean TTL of the resolved in each bucket and use a bracketed range of that (1,2 and 3 standard deviations) that to find a set of time intervals…

TTLs and re-query rates are certainly interesting, as are diurnal/calendar effects, label counts in QNAMEs, QTYPE distribution, ... Sebastian Castro used a large number of predictors in their work doing cluster analysis of clients in NZ, trying to distinguish between resolvers and non-resolvers. I found over 30 predictors were relevant in trying to distinguish between actual resolvers with different end-user populations without reference to source address. The patterns exist over more dimensions than I was expecting, but they are there.

> It does show that there is a number of big players - I’m guessing it will have mobile operators and some of the large ISPs in it, but that noise also tells us that there is an existing ecosystem of…

Of many diverse clients, certainly.

> However this would be very very different from what the patterns form DoH would look like. Diversity will drop off even lower and the long tail may completely vanish.

There's certainly much less variety in the population of clients doing DoH today, and without a coherent discovery mechanism for DoH services, most clients' selection of DoH service I think will be naturally biased towards those that are well-known (those that are blessed by Mozilla, for example; those that exist as examples in the stubby configuration file).

Transport between the application and the resolver is going to be hard to determine using measurement at the authoritative server, however. The lack of measurement points outside the DoH service provider and application is kind of why everybody keeps getting their pitchforks out.

Joe

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 7 Aug 2019 17:33:19 +0100

Hi Glenn,

I'm confused sorry...

On 07/08/2019 17:22, Deen, Glenn (NBCUniversal) wrote:
> However this would be very very different from what the patterns form
> DoH would look like. Diversity will drop off even lower and the long
> tail may completely vanish.
Joe's data was for traffic seen at some authoritative
servers for .org IIUC. DoH is stub to recursive only.
So I don't see how the use of DoH would or would not
be reliably visible from those vantage points. (It may
be visible now due to some specifics in how different
DoH servers recurse I guess but presumably that'd be
fairly brittle.)

That said, 80% of queries to .org being from two dozen
systems is fairly sad/scary/surprising so I'd love to
see more info on that whenever that's ready.

Cheers,
S.

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 16:41:34 +0000

> On Aug 7, 2019, at 8:51 AM, Stephen Farrell <[email-masked]> wrote:
> 

One of the downsides of encryption is it will be much harder to actually audit the data being sent by applications to resolvers.   

I’m not against encryption, but I’m bothered when things are encrypted for my protection but are also invisible to me.  

  I’m relatively sure there will be researchers who will make the effort to keep the big browser makers honest, but the ability for others apps to add DoH and undisclosed http headers for tracking is likely something we should expect to see in the future.   This is a specific worry to DoH since it’s over http. 

My point is that we shouldn’t be complacent here and be too quick to dismiss worries. Instead it would be nice if researchers were engaging now up front to poke at things and explore where the abuse opportunities are. 

Glenn

---

From: Stephen Farrell <[email-masked]>
Date: Wed, 7 Aug 2019 17:51:15 +0100

Hiya,

On 07/08/2019 17:41, Deen, Glenn (NBCUniversal) wrote:
>> So, again, this is down to trusting the application code as a DNS
>> client that's allowed to talk to it's selected recursive... or not.
>> It's not really a DoT/DoH distinction.
>> 
>> tracking via DoH, if one used some library or re-used some code
>> that did send cookies etc. And as the fine presentation in pearg
>> demonstrated, http session or other ancialliary information may
>> make it easier to do traffic analysis on DoH vs. DoT. I'm not
>> convinced either factor is a long term thing though.
> 
> One of the downsides of encryption is it will be much harder to
> actually audit the data being sent by applications to resolvers.

TBH I think you're badly conflating two independent things
there.

- concentration of sources for traffic to a TLD authoritative
  as reported by Joe

- use of encrypted transports for DNS between stub and
  recursive

Given the history of people claiming the sky is about to fall
"because encryption," such conflation is IMO fairly unhelpful
given we want a rational analysis of all this ADD stuff.

Cheers,
S.

> 
> I’m not against encryption, but I’m bothered when things are
> encrypted for my protection but are also invisible to me.
> 
> I’m relatively sure there will be researchers who will make the
> effort to keep the big browser makers honest, but the ability for
> others apps to add DoH and undisclosed http headers for tracking is
> likely something we should expect to see in the future.   This is a
> specific worry to DoH since it’s over http.
> 
> My point is that we shouldn’t be complacent here and be too quick to
> dismiss worries. Instead it would be nice if researchers were
> engaging now up front to poke at things and explore where the abuse
> opportunities are.
> 
> Glenn
>

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 16:55:04 +0000

> On Aug 7, 2019, at 9:33 AM, Stephen Farrell <[email-masked]> wrote:
> 
> I'm confused sorry...
> 

What I meant was that if the browsers are directing all their queries to a small set of “approved” DoH services then there traffic Joe observes will reduce that 2 dozen top senders down to 3, 4 ? approved DoH services  with quad 8 being the lions share?

Glenn

---

From: Joe Abley <[email-masked]>
Date: Wed, 7 Aug 2019 12:49:08 -0400

> My point is that we shouldn’t be complacent here and be too quick to dismiss worries. Instead it would be nice if researchers were engaging now up front to poke at things and explore where the abus…

Nicely put.

Centralisation of resolution is already upon us (I claim, without presenting right now to back it up, film in Austin).

DoH may or may not make things more centralised, depending on how it is implemented and used; however, I don't find a shift from almost entirely centralised (now) to a bit more centralised (possible future) that much of a concern. Relatively speaking, it's kind of the same thing.

However, the nature of the data that is available for collection is (a) potentially different and (b) hard to quantify if you're not a generator or a collector. That's a change from today. I think that's interesting.

Joe

---

From: Christian Huitema <[email-masked]>
Date: Wed, 7 Aug 2019 09:52:29 -0700

> see more info on that whenever that's ready.

The ICANN/ITHI project has been collecting a variety of metrics since 2018. One of those is an analysis of resolver behavior, in cooperation with APNIC: https://ithi.research.icann.org/graph-m5.html. The figures that we have confirm the long tailed nature of the resolver distribution. The 10000 largest resolvers account for 92.5% of eyeballs.

We are working to provide better characterization of the resolver population, such as grouping resolvers based on autonomous system number.

Note that the %of eyeballs numbers that we collect are different from the % of queries mentioned here. The fraction of queries seen by servers depends heavily on resolver behaviors such as caching and proactive fetching. It is not an unbiased estimator of resolver market share.

-- Christian Huitema

---

From: Adam Roach <[email-masked]>
Date: Wed, 7 Aug 2019 11:05:52 -0600

> On Aug 7, 2019, at 10:41, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> 

Thanks, Glenn. I think this is the first time I’ve seen this explained clearly (outside of the malware case). Can you say more about these applications and what the worst-case outcomes might be? My (possibly naïve) view is that such apps are typically vertically integrated and therefore have much more damaging ways to track users, but it’s quite possible that we’re thinking of two very different kinds of applications. 

/a

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 17:11:02 +0000

> On Aug 7, 2019, at 9:51 AM, Stephen Farrell <[email-masked]> wrote:
> 

And between recursive resolvers too. 
> given we want a rational analysis of all this ADD stuff.

I don’t think I am conflating - certainly that’s not my intent. 

Encryption of dns traffic can be done without accompanying centralization.  It’s some deployment plans that have conflated the two by linking doing the later as part of the former.  

As I said I’m not trying to fire up anti-encryption arguments.  

For the record: I am not against encryption. 

However I do worry that it removes the ability for security researchers to poke and prod and examine what’s being sent and that takes away valuable assurances we get from researchers doing their thing. 

I also worry that it removes my ability to see what’s being sent on my behalf, thought this isn’t specific to DoH - I have long had this issue with IoT devices when they talk back to their home using encrypted flows.  

But I don’t want to digress this thread any further than I have already. 

Glenn

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 11:18:02 -0600

You mean both “unintentionally and intentionally”. 

The privacy consideration and security consideration sections need to be written for end-users / organizations / enterprises as well as for developers.  They need to say, hey, if you let this protocol through your network, here are the new risks you need to worry about.  Here is the increased attack surface.  Here are the new risks to privacy.  Or maybe what we need is a “Privacy and Risk” section in every RFC.  Something that calls out how this new fangled thing is going to impact operational security.  

Thanks,
Bret

> On Aug 7, 2019, at 10:30 AM, Eric Orth <[email-masked]> wrote:
> 
> (Thanks for branching off to another subject.  I wanted to respond in the other thread when this was brought up, but it was kinda off-topic for most of what was being discussed there.)
> 
> Agreed.  There is certainly lots of room for a DoT or DoH client to unintentionally make bad privacy decisions.  Publishing guidance and information can help ensure they have thought things through before they make privacy-affecting decisions.  This is a very actionable area for IETF.  I think there would be enough consensus on many areas of guidance (but obviously not all) to get a decent sized document.
> 
> Not everything would necessarily be a hard and fast do or don't, as different tradeoffs may be relevant for different applications and use cases.  That's okay since the important part is providing the information to help applications make the correct decisions for themselves.  E.g., accepting cookies is one of the areas already briefly discussed in RFC8484, and I think it took the right approach: "HTTP cookies SHOULD NOT be accepted by DOH clients unless they are explicitly required by a use case."  I think we need more guidance like that with more detail on the tradeoffs and considerations.  It may seem obvious that applications should never send or accept cookies with DoH, but some special-purpose application may want to use DoH to send authentication information to a special-purpose DNS server that will only return information to authorized users.  DoH and DNS are not limited just to the use case of browsers requesting addresses of all hostnames to be visited.
> 
> On Wed, Aug 7, 2019 at 12:07 PM Jim Reid <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> > On 7 Aug 2019, at 16:42, Bret Jordan <[email-masked] <mailto:[email-masked]>> wrote:
> > 
> [email-masked] <mailto:[email-masked]>
> https://www.ietf.org/mailman/listinfo/add <https://www.ietf.org/mailman/listinfo/add>

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 17:18:35 +0000

> Thanks, Glenn. I think this is the first time I’ve seen this explained clearly (outside of the malware case). Can you say more about these applications and what the worst-case outcomes might be? My…

Adam,

I was thinking about applications outside of browsers, either desktop apps or more likely mobile apps. It seems that mobile apps are the ones that seek to push the boundaries and collect data on users and track them. 

Glenn

---

From: Warren Kumari <[email-masked]>
Date: Wed, 7 Aug 2019 13:10:17 -0400

[ TOP-POST]

Please trim the 'To:' list -- this has now exceeded mailman's "Too
many recipients" trigger and I'm having to go approve each email,
which causes delays (and makes me kvetch too!).

W
P.S: I'm sending this with the original To: harder so everyone gets it :-P

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 17:13:40 +0000

Adam,

Glenn

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 11:21:29 -0600

Joe,

I would push back a bit.  Right now it is super easy for an organization, enterprise, managed network, or secure network to block TCP/UDP 53 outbound accept for your internal DNS server. So it makes it really easy to prevent clients and apps from using a centralized source. 

Just like we wrote an RFC about how pervasive monitoring is bad.  We should write an RFC for how centralization of data is bad and how tracking of users inside of sessions is bad. 

Thanks,
Bret

> On Aug 7, 2019, at 10:49 AM, Joe Abley <[email-masked]> wrote:
>

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 11:26:31 -0600

Glenn,

This is my thinking as well.  It would be so easy to have unique IDs per client install, grab all of the geo-location information, you would have time information, you could then correlate this data in really scary ways. 

Thanks,
Bret

> On Aug 7, 2019, at 11:13 AM, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> 
> 
>

---

From: Brian Dickson <[email-masked]>
Date: Wed, 7 Aug 2019 10:27:22 -0700

On Wed, Aug 7, 2019 at 10:11 AM Deen, Glenn (NBCUniversal) <
[email-masked]> wrote:

> Encryption of dns traffic can be done without accompanying
> centralization.  It’s some deployment plans that have conflated the two by
> linking doing the later as part of the former.
>
> However I do worry that it removes the ability for security researchers to
> poke and prod and examine what’s being sent and that takes away valuable
> assurances we get from researchers doing their thing.
>
> I also worry that it removes my ability to see what’s being sent on my
> behalf, thought this isn’t specific to DoH - I have long had this issue
> with IoT devices when they talk back to their home using encrypted flows.
>
>
I think this is a very valid point, and one thing that is probably valid to
consider at the full "ADD" scope, i.e. applicable to all encrypted DNS
traffic from any/all applications.

And, I think, is architectural in nature.

The initial DoH spec may have been overly prescriptive in how to implement,
or at least is being implemented without thought to the above issues.

It basically boils down to: if there is no on-host place to inspect the DNS
traffic before it goes out an encrypted transport, it becomes problematic
to try to see what is being sent.

If it were feasible to separate the TLS portion, architecturally, from the
DNS portion, in a way that facilitated this visibility, those concerns
would be alleviated.

For example, if the app wrote to a unix-domain socket, where the other end
of the socket was a "service" that did all the TLS set-up and
encrypt/decrypt stuff for DoH/DoT, possibly for multiple client apps, that
might provide enough of a monitoring "surface".

It's not perfect, and I'm still opposed to DoH and in favor of DoT for the
general use scenarios, but it does give security folks (researchers and
operators) enough to work with at an app vs host level.

Brian

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Wed, 7 Aug 2019 17:40:05 +0000

> On Aug 7, 2019, at 10:27 AM, Brian Dickson <[email-masked]> wrote:
> 
> It basically boils down to: if there is no on-host place to inspect the DNS traffic before it goes out an encrypted transport, it becomes problematic to try to see what is being sent.
> 
> If it were feasible to separate the TLS portion, architecturally, from the DNS portion, in a way that facilitated this visibility, those concerns would be alleviated.
> 
> For example, if the app wrote to a unix-domain socket, where the other end of the socket was a "service" that did all the TLS set-up and encrypt/decrypt stuff for DoH/DoT, possibly for multiple client apps, that might provide enough of a monitoring "surface".

Thanks for writing that up. It helped me recognize that part of my ongoing uneasiness with the DoH functionality being in the browser and not the OS
Is that it pulls the dns logs out of the system logs into only the browser tab.  

It also pulls it out of any device wide policy settings that I manage today at the OS. 

While I see the ease of deployment putting it in the application brings, it would be so much better from a device administration standpoint if it were better integrated at with the OS as a distinct service outside browser.  If you can’t do that, perhaps the browsers could write dns errors to the device logs. 

On device log data is so useful in debugging.  Yes, I still use syslog when fixing things. 

Glenn

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 10:51:31 -0700

> It also pulls it out of any device wide policy settings that I manage

I don't think this is true (anyone in control of the OS can intercept TLS).
It's not a topic for the IETF, though.

thanks,
Rob

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 7 Aug 2019 10:54:16 -0700

On Wed, Aug 7, 2019 at 7:38 AM Livingood, Jason <[email-masked]>
wrote:

> *From: *Add <[email-masked]> on behalf of Rob Sayre <
> [email-masked]>
>
> You can say that, as can Mozilla, but the market & deployment data
> suggests the complete opposite.
>
I know you have a strongly held contrary view but I think you are in the
> rough on this one and hope we can move on.
>

I read Martin's presentation as saying that DNS is not a good control point
(for the reasons he indicates) and that it is going to become increasingly
less effective. That's not really something which the data you are
referring to speaks to, and I don't actually think there's consensus on the
contrary point.

-Ekr

---

From: Eric Orth <[email-masked]>
Date: Wed, 7 Aug 2019 13:57:16 -0400

Depends on the scope of the particular documents.  If we're talking a
document of privacy guidance/considerations specifically for protocol
implementers, only unintentional is in scope, since malware developers
would obviously ignore such guidance.  If a document for multiple audiences
(such as the protocol specs themselves), or separate documents for other
audiences, then yes, guidance for network operators on dealing with
intentional action is also important.

I would argue that guidance documents on DoH/DoT privacy considerations
should be separate documents for implementer guidance and guidance for
network operators dealing with the traffic in their networks.  Somebody
seeking such guidance is very likely going to be one or the other and the
guidance will be much better communicated if targeted to them.  We should
certainly have many different parties helping write all such documents to
provide valuable feedback, but I think we should separate and target the
intended audiences of such documents.

On Wed, Aug 7, 2019 at 1:18 PM Bret Jordan <[email-masked]> wrote:

> The privacy consideration and security consideration sections need to be
> written for end-users / organizations / enterprises as well as for
> developers.  They need to say, hey, if you let this protocol through your
> network, here are the new risks you need to worry about.  Here is the
> increased attack surface.  Here are the new risks to privacy.  Or maybe
> what we need is a “Privacy and Risk” section in every RFC.  Something that
> calls out how this new fangled thing is going to impact operational
> security.
>
> Thanks,
> Bret

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 7 Aug 2019 10:59:43 -0700

> some do’s and don’ts on privacy issues for DoH clients. Which WG is most

How would this be different from
https://tools.ietf.org/rfcmarkup?doc=8484#section-8?

-Ekr

---

From: Eric Orth <[email-masked]>
Date: Wed, 7 Aug 2019 14:16:09 -0400

> https://tools.ietf.org/rfcmarkup?doc=8484#section-8?
I believe the assumption is that informational RFCs could discuss the
privacy considerations in more depth and breadth than is appropriate in the
protocol RFC (or than is agreeable to the higher bar for a standards RFC).
I don't know if that assumption is completely true or not.  Depends on what
sort of content people can come up with.  I think it's certainly worth
exploring further.  I'd like to see specific ideas/drafts for such a
document.  Then we can criticize it as redundant with RFC8484 section 8 if
it is so.

---

From: Jim Reid <[email-masked]>
Date: Wed, 7 Aug 2019 19:43:14 +0100

> On 7 Aug 2019, at 18:59, Eric Rescorla <[email-masked]> wrote:
> 
> How would this be different from https://tools.ietf.org/rfcmarkup?doc=8484#section-8?

Well for starters, section 8 of RFC8484 says nothing directly about client-side privacy. Though it does do that for privacy considerations in the server. RFC8484 mentions there are new considerations because HTTP offers extra opportunities for correlation. It doesn't clearly say what those considerations are or how to address them.

Saying "DoH is not known to introduce new concerns beyond those associated with HTTPS." is all very well. [Though perhaps that's the easy way out.] Even so I think it would be good to have a discrete document which identified those privacy concerns in a DoH setting and their trade-offs/consequences. That would be very helpful. IMO it would also save DoH developers from having to make sense of almost everything in the HTTP document set before deciding what protocol features would be good or bad from a privacy perspective in their client-side software.

And who knows, maybe those running DoH servers might configure them to ignore 'naughty' headers or even reject queries that use the privacy-unfriendly stuff mentioned earlier. It's nice to dream, eh?

---

From: Adam Roach <[email-masked]>
Date: Wed, 7 Aug 2019 13:43:57 -0500

> I was thinking about applications outside of browsers, either desktop apps or more likely mobile apps. It seems that mobile apps are the ones that seek to push the boundaries and collect data on us…

Oh, I absolutely get that point. If I read things correctly, you're 
talking about native apps for things like Facebook, Slack, Twitter, and 
the like, right? The point I was trying to make above is that apps in 
*that* class can collect extremely detailed information about user 
behavior using less Rube-Goldbergesque mechanisms than laundering such 
data through DNS servers. I'm trying to tease out how you think the use 
of centralized DoH servers makes that particular landscape any 
different: it's not obvious to me, and so I want to make sure I'm not 
missing something.

/a

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 13:04:55 -0600

The problem I have with all of this, is when protocol developers go off and produce designs without fully understanding and documenting how this is going to impact people and networks. If we do this as a left-hand and right-hand effort, where the two hands do not know each other, that is a problem. In some cases, if we had done the research, and involved enough groups, then the decision might have been, we should not do this at all.  Just because we _can_ do something, does not mean we _should_.

I think this discussion, the meeting in Montreal, and the meeting in Prague is showing that the industry is starting to wake up to the things people in the IETF are trying to do.  I think there is a lot of push back now.  And personally, I think that makes some people uncomfortable.  Maybe RFCs need to back to what they were originally intended to be, and that is a “request for comments” on an idea.  

The industry has moved beyond the 1980s and 90s when people were just trying to make things work and when you could get a representative from every group that was doing something with the internet in the same room. 

Thanks,
Bret

> On Aug 7, 2019, at 11:57 AM, Eric Orth <[email-masked]> wrote:
> 
> Depends on the scope of the particular documents.  If we're talking a document of privacy guidance/considerations specifically for protocol implementers, only unintentional is in scope, since malware developers would obviously ignore such guidance.  If a document for multiple audiences (such as the protocol specs themselves), or separate documents for other audiences, then yes, guidance for network operators on dealing with intentional action is also important.
> 
> I would argue that guidance documents on DoH/DoT privacy considerations should be separate documents for implementer guidance and guidance for network operators dealing with the traffic in their networks.  Somebody seeking such guidance is very likely going to be one or the other and the guidance will be much better communicated if targeted to them.  We should certainly have many different parties helping write all such documents to provide valuable feedback, but I think we should separate and target the intended audiences of such documents.
> 
> On Wed, Aug 7, 2019 at 1:18 PM Bret Jordan <[email-masked] <mailto:[email-masked]>> wrote:
>> On Aug 7, 2019, at 10:30 AM, Eric Orth <[email-masked] <mailto:[email-masked]>> wrote:
>>

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 7 Aug 2019 12:06:19 -0700

On Wed, Aug 7, 2019 at 11:43 AM Jim Reid <[email-masked]> wrote:

> Well for starters, section 8 of RFC8484 says nothing directly about
> client-side privacy.

I don't know what you mean by "client side privacy". Much of S 8.2 seems to
be about
what information the client leaks to the server.

Though it does do that for privacy considerations in the server. RFC8484
> mentions there are new considerations because HTTP offers extra
> opportunities for correlation. It doesn't clearly say what those
> considerations are or how to address them.
>

You must have something different in mind than I do. For instance, the
following seems on point:

  HTTP's feature set can also be used for identification and tracking
  in a number of different ways. For example, Authentication request
  header fields explicitly identify profiles in use, and HTTP cookies
  are designed as an explicit state-tracking mechanism between
  the client and serving site and often are used as an authentication
  mechanism.

-Ekr

---

From: Jim Reid <[email-masked]>
Date: Wed, 7 Aug 2019 20:30:26 +0100

On 7 Aug 2019, at 20:06, Eric Rescorla <[email-masked]> wrote:
> 
> 
> You must have something different in mind than I do.

Yup. 

> For instance, the following seems on point:
> 

I said earlier RFC8484 "doesn't clearly say what those considerations are or how to address them". The text you quoted above doesn't seem to do that IMO.

For instance, what sorts of HTTP header fields should DoH clients use or not use to enhance or ensure privacy? Why? The RFC says DoH clients SHOULD only use cookies when "they are explicitly required by a use case". How is that explicit requirement/signalling done and what use case(s) does that apply to? What should/shouldn't go in those cookies?

I think these sorts of considerations need to be better documented. Or has that work already been done elsewhere?

BTW, section 8.2 is titled "In the Server". A client developer might well mistakenly conclude they can ignore it because server-side privacy considerations don't apply to them. YMMV.

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 7 Aug 2019 12:45:12 -0700

On Wed, Aug 7, 2019 at 12:30 PM Jim Reid <[email-masked]> wrote:

> I said earlier RFC8484 "doesn't clearly say what those considerations are
> or how to address them". The text you quoted above doesn't seem to do that
> IMO.
>

Well, indeed YMMV. I think it does in fact say what those considerations
are, and leaves it up to the reader to decide how to address them. I would
encourage you to try writing some text you think would be more satisfactory
and see if it's actually practical to get more specific. With that said,
the privacy considerations sections in this RFC are really far better than
those in the vast majority of RFCs that I reviewed when I was on the IESG,
and we've tolerated the truly terrible DNS privacy situation for quite some
time, so the particular focus on the residual privacy impact of the
interaction between the DoH client and the DoH server seems a bit
incongruous.

>
> For instance, what sorts of HTTP header fields should DoH clients use or
> not use to enhance or ensure privacy? Why?

"Implementations are advised to expose the minimal set of data needed to
achieve the desired feature set."

Honestly, a complete enumeration of every header seems out of place in this
document.

The RFC says DoH clients SHOULD only use cookies when "they are explicitly
> required by a use case". How is that explicit requirement/signalling done
> and what use case(s) does that apply to?

I'm not really sure what you're looking for here. It's not possible to
enumerate every possible use case, so what this text says is "don't do it
unless you have to".

What should/shouldn't go in those cookies?

Huh? Cookies are opaque at the client. Clients either send or don't send
them, so the question of what goes in them doesn't apply.

> BTW, section 8.2 is titled "In the Server". A client developer might well
> mistakenly conclude they can ignore it because server-side privacy
> considerations don't apply to them. YMMV.
>

Yeah, I would generally categorize this complaint as in the "nit" category.

-Ekr

---

From: Jim Reid <[email-masked]>
Date: Wed, 7 Aug 2019 22:13:23 +0100

> On 7 Aug 2019, at 20:45, Eric Rescorla <[email-masked]> wrote:
> 
> Well, indeed YMMV.. I think it does in fact say what those considerations are, and leaves it up to the reader to decide how to address them.

This goes to the heart of our disagreement. IMO leaving these considerations to the reader to make (and with little context to understand the consequences or trade-offs) creates a gaping hole with lots of scope for all sorts of unpleasantness.

> I would encourage you to try writing some text you think would be more satisfactory and see if it's actually practical to get more specific.

Thanks, I will. Though this'll be a new I-D - not an effort to update or change 8484.

> we've tolerated the truly terrible DNS privacy situation for quite some time, so the particular focus on the residual privacy impact of the interaction between the DoH client and the DoH server seems a bit incongruous.

Maybe. Though if the IETF is going to do something about the truly terrible DNS privacy situation -- and I agree it’s been truly terrible since forever -- we have to start somewhere. DoH and DoT are steps in the right direction of course. The next steps IMO should be how these protocols get deployed and operated. And further steps will be needed too, particularly for vanilla DNS.

> For instance, what sorts of HTTP header fields should DoH clients use or not use to enhance or ensure privacy? Why? 
> 
> "Implementations are advised to expose the minimal set of data needed to achieve the desired feature set."
> 
> Honestly, a complete enumeration of every header seems out of place in this document.

I wasn’t suggesting putting that into an updated RFC8484, just that it gets documented somewhere.

> I'm not really sure what you're looking for here. It's not possible to enumerate every possible use case, so what this text says is "don't do it unless you have to”. 

Indeed. But what would “unless you have to” mean in practice? For those that have noble privacy ambitions it’ll mean one thing. For those who don’t, it'll mean something very different. If my intention is outrageous privacy violation of everyone on the Internet, the semantics of “unless you have to” gives me enough scope to achieve my desired feature set.

---

From: Adam Roach <[email-masked]>
Date: Wed, 7 Aug 2019 16:38:46 -0500

On 8/7/19 16:13, Jim Reid wrote:
>> ...

I encourage you to understand the scope of what you're proposing; see 
https://www.iana.org/assignments/message-headers/message-headers.xhtml

> Indeed. But what would “unless you have to” mean in practice? For those that have noble privacy ambitions it’ll mean one thing. For those who don’t, it'll mean something very different. If my inten…

Sure, and I don't think intentional bad actors are going to change their 
behavior even with more detailed guidance. What we need is enough 
explanation of the basic principle that people writing code in good 
faith end up doing the right thing. "Don't send it unless omitting it 
breaks your use case" seems like reasonable guidance for those folks.

I mean, if you want to put together a DoH-oriented document that 
analyzes on the order of 200 HTTP header fields to determine which ones 
might reasonably be sent when, including some description of the privacy 
implications of doing so, more power to you -- but I suspect it will be 
a lot of work with little to no practical effect beyond that of the text 
currently in RFC 8484.

/a

---

From: nusenu <[email-masked]>
Date: Wed, 07 Aug 2019 21:49:00 +0000

Also consider reading the attempts to get clearer wording into RFC8484
from about a year ago - which unfortunately hasn't been accepted.

relevant threads:
https://mailarchive.ietf.org/arch/msg/doh/vHjITrOMhWSdrozGFe4-eGNMEJc
https://mailarchive.ietf.org/arch/msg/doh/5bdW_B1XW1xoVKC5AfAuH40WKfo
https://mailarchive.ietf.org/arch/msg/doh/Y1imMm9vJi2D5emVr3SfKkqMzqA

Sara started this but didn't have time to continue:
https://datatracker.ietf.org/doc/draft-dickinson-doh-dohpe/

relevant to this on the implementation level (Firefox):
https://bugzilla.mozilla.org/show_bug.cgi?id=1543201
https://bugzilla.mozilla.org/show_bug.cgi?id=1544724

-- 
https://twitter.com/nusenu_
https://mastodon.social/@nusenu

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Thu, 8 Aug 2019 00:06:16 +0000

Hi Adam

> On Aug 7, 2019, at 11:44 AM, Adam Roach <[email-masked]> wrote:
> 
> Oh, I absolutely get that point. If I read things correctly, you're talking about native apps for things like Facebook, Slack, Twitter, and the like, right? The point I was trying to make above is that apps in *that* class can collect extremely detailed information about user behavior using less Rube-Goldbergesque mechanisms than laundering such data through DNS servers. I'm trying to tease out how you think the use of centralized DoH servers makes that particular landscape any different: it's not obvious to me, and so I want to make sure I'm not missing something.

I’ve been involved in a set of parallel conversations today on this list which do not overlap. 

This particular concern does not involve the concern over centralization. 
In this particular case, I’m talking about native OS apps that run on the desktop or the tablet or the phone.  Once the DoH code and infrastructure is in the wild inside browsers it will start to pop up in non-browser applications.   

Your point about apps inside browsers does make
me think this issue lies there too. 

The issue I’m raising in this thread is the question of how researchers will be able to examine  what applications are up to with DoH. 

One suggestion was some form log record which could help in the in browser app situation and if the DoH access was via a service in the device or the OS.   That maybe behavior the IETF could write up. 

Glenn

---

From: Ted Lemon <[email-masked]>
Date: Wed, 7 Aug 2019 17:16:59 -0700

On Aug 7, 2019, at 5:06 PM, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> Once the DoH code and infrastructure is in the wild inside browsers it will start to pop up in non-browser applications.   

Glenn, can you explain why this is being presented as a hypothetical?   The status quo is that this is available to apps.   It’s trivial to do.   I did my first tunnel over port 443 over a decade ago, and I was not innovating.  Any little bit of malware or privacyware that wants to do this can do it today.

I apologize for being blunt here, but this discussion has gotten really long, and a lot of it is discussion about stuff that is in no way actionable.   I think that anything based on the above assertion falls into that category; if you think otherwise, it would help if you could explain.

---

From: "Martin Thomson" <[email-masked]>
Date: Thu, 08 Aug 2019 10:27:49 +1000

On Thu, Aug 8, 2019, at 00:38, Livingood, Jason wrote:
> > DNS is not an effective control surface.
> 
> suggests the complete opposite. I know you have a strongly held 
> contrary view but I think you are in the rough on this one and hope we 
> can move on. 

I want to unpack this a little, because I think that declaring that view as in the rough is a little presumptive.  At least from my perspective.

(I don't want to sealion here, so feel free to tell me to go away if these questions seem disingenuous or if I have a gap in my understanding that would be better addressed by myself.)

There are three questions bound up in this, each that bears examination:

1. Is DNS an effective control surface?

2. Should DNS be an effective tool for control?  ...and for what?

3. Why would someone want to use DNS to exert control?

The arguments I presented at the meeting concentrated on the first and partly the second, on the assumption that going to the root of the issue might allow us to avoid the latter questions.  That was because these get increasingly difficult to grapple with, especially when it comes to use cases.  But I'm happy to talk about them if the first is indeed false, or even merely not absolute.

On the first, the argument I offered was fairly simple: Name resolution doesn't exclusively use DNS, nor does it exclusively use network-provided resolvers.  That suggests that any solely DNS-based mechanism cannot capture and control all potential paths toward an endpoint.  

On the second, I looked at captive portals.  I know that some networks perform DNS-based interception, but the vast majority of portals use HTTP interception.  The former can be destroyed by alternative resolvers and resolution methods (to the first point) and DNSSEC; the latter by HTTPS and HSTS.  The entire reason we have a working group is to provide an alternative method (or control point) for getting a captive portal in front of a person in those adverse conditions.

I realize now that the first argument might not be considered dispositive, so maybe we can talk more about that some more.  In what ways is this control surface still effective, and in what ways does that make the Internet better?

Cheers,
Martin

---

From: Paul Ebersman <[email-masked]>
Date: Wed, 07 Aug 2019 18:43:29 -0600

mt> There are three questions bound up in this, each that bears
mt> examination:

mt> 1. Is DNS an effective control surface?
mt> 2. Should DNS be an effective tool for control?  ...and for what?
mt> 3. Why would someone want to use DNS to exert control?

[...]

mt> On the first, the argument I offered was fairly simple: Name
mt> resolution doesn't exclusively use DNS, nor does it exclusively use
mt> network-provided resolvers.  That suggests that any solely DNS-based
mt> mechanism cannot capture and control all potential paths toward an
mt> endpoint.  

This sounds like the argument that anything less than perfect isn't
worth doing. Firewalls miss things. Signature based detection misses
things. Anti-virus misses things. Anti-spam misses things. But all have
their place. Sometimes whack-a-mole is a win if you whack a high enough
percentage of the moles that moles run scared.

mt> On the second, I looked at captive portals. I know that some
mt> networks perform DNS-based interception, but the vast majority of
mt> portals use HTTP interception.

All sorts of networks other than captive portals do DNS-based
interception. My understanding was that DoH was there primarily to get
around DNS-based interception where the end user didn't give their
consent freely.

In cases where the user did consent (parental controls, the user wants
to stay employed, etc), DNS filtering is an effective tool. Not perfect,
not sufficient in and of itself but hardly useless and widely deployed.

mt> I realize now that the first argument might not be considered
mt> dispositive, so maybe we can talk more about that some more.  In
mt> what ways is this control surface still effective, and in what ways
mt> does that make the Internet better?

See above. For things like parental control & malware/botnet blocking or
as part of a multi-layer employer network security plan, it's in very
wide spread use. Even if in some cases (parental blocking) it's more
effective at making parents feel better than giving a 100% guarantee
that their kids won't get to bad stuff, that still has value and makes
the internet "better" for the parents. Blocking malware and botnets
makes things better for everyone.

Note that in all the above use cases, I mentioned freely given user
consent. The reason I am tolerating the incredible volumes of email
related to DoT/DoH is because there are cases (like hotel captive
portals) where DNS is being filtered/altered/intercepted *without* that
consent and a more standardized way of giving users informed choice
would be a "good thing" for the internet too.

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Thu, 8 Aug 2019 00:44:14 +0000

Hi Ted

On Aug 7, 2019, at 5:17 PM, Ted Lemon <[email-masked]<mailto:[email-masked]>> wrote:

On Aug 7, 2019, at 5:06 PM, Deen, Glenn (NBCUniversal) <[email-masked]<mailto:[email-masked]>> wrote:
Once the DoH code and infrastructure is in the wild inside browsers it will start to pop up in non-browser applications.

Glenn, can you explain why this is being presented as a hypothetical?   The status quo is that this is available to apps.   It's trivial to do.   I did my first tunnel over port 443 over a decade ago, and I was not innovating.  Any little bit of malware or privacyware that wants to do this can do it today.

I did not use the word hypothetical.  This is very real and something I'm saying we should be talking about in depth and seeking approaches for addressing.  Logging has been suggestion as one idea, there are no doubt more.

Since DoH and DoT are responses to privacy concerns, we should make sure we are not simply trading one exposure for another.

I do believe there are actionable tasks here.

1. Get security researchers engage now at analyzing attack methods against DoH as well as attack methods against users via DoH.

1b document those in an I-D so we can work on them

2. Logging standards and best practices for clients in information that should be logged.

3. Review and expansion of the current RFC ( sorry don't know from memory) that Ekr cited earlier.

There are likely more that will pop up when the very clever ietf and security researcher communities put
their minds to it.

Glenn

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Thu, 8 Aug 2019 01:10:07 +0000

Hi Martin 

> On Aug 7, 2019, at 5:28 PM, Martin Thomson <[email-masked]> wrote:
> 
> Is DNS an effective control surface?

Many organizations and legal frameworks have looked at DNS and have decided, Yes, it is sufficiently effective as a control surface for a variety of issues such as malware and restricting school and library access to things like pornography. 

Is it perfect? no it is not.  

Is it effective, yes it is as part of a layered defense. 

> 2. Should DNS be an effective tool for control?  ...and for what?

That’s trending into a policy question.   From a technical perspective it is.  When I needed to manage what my own kids could access on the internet DNS was and is one of my tools

> 3. Why would someone want to use DNS to exert control?

This is very much into the policy territory
 
In exploring the topic I’ve seen a large gamut of reasons:    malware, restricting child access to adult and hate material, enforcing corporate policies such as no p2p (sorry folks who want to watch movies at work), gambling sites etc. not to orgs that block access to social media by workers. 

And let’s not forget botnet mitigation where known c&c domains are blocked. 

These are all technical controls due to policy choices by the device owner and in some cases network owner. 

—

And DNS is part of defense in depth.   It’s a particularly good component which reduces the problem surface significantly.   It’s not perfect and so additional controls are often deployed. 

I still lock my house and car even though a thief can easily break a window and steal my valuables. 

-glenn

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 19:35:26 -0600

Well said Glenn.  I completely agree.  

No solution is perfect, but this represents a layered defense that organizations rely on. This coupled with other solutions can make a sound solution.

Thanks,
Bret

> On Aug 7, 2019, at 7:10 PM, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> 
> Hi Martin 
>

---

From: Ted Lemon <[email-masked]>
Date: Wed, 7 Aug 2019 18:36:52 -0700

On Aug 7, 2019, at 5:44 PM, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> On Aug 7, 2019, at 5:17 PM, Ted Lemon <[email-masked] <mailto:[email-masked]>> wrote:
> 
>> On Aug 7, 2019, at 5:06 PM, Deen, Glenn (NBCUniversal) <[email-masked] <mailto:[email-masked]>> wrote:
> I did not use the word hypothetical. 

Glenn, I’m quoting what you said above.   You said “once DoH code and infrastructure is in the wild.”   This suggests that it is not already in the wild, which is not true.   That’s what I mean by “presented as a hypothetical.”

> Since DoH and DoT are responses to privacy concerns, we should make sure we are not simply trading one exposure for another. 

This is what I mean by hypothetical.   You are stating this as if it’s a future event that may or may not happen: “make sure we are not trading X for Y.”   There is not tradeoff to be made, because the situation is already that we have Y, and that was never under our control to begin with.   We were never in a position to trade.

This is what I mean when I ask you to unpack the hypothetical.   Do you think there is something to be prevented or controlled here?   How do you propose to prevent or control it?

> 1. Get security researchers engage now at analyzing attack methods against DoH as well as attack methods against users via DoH. 

What question would you be asking?  Can you state a specific question that you want answered?   I can think of lots of obvious attacks against users of DoH, and I’m sure you can too.   I can’t think of a lot that wouldn’t also be attacks against Do53.  Do you have reason to believe that there are attacks against users that are possible with DoH but not with Do53?   Are you just talking about leaking information in HTTP headers?   If so, that’s a known issue and as far as I know is already being worked on.

> 2. Logging standards and best practices for clients in information that should be logged. 

So these would be cooperating clients?   The idea being to be able to debug problems that are caused by resolvers?   What’s the interface going to be for this?   What’s the problem you are hoping to solve here?   How bad a problem is it?   Do you know?

I ask because the information you’d need to reveal to debug a problem in a cooperating app would probably violate the user’s privacy, so there’s a whole bunch of work that would have to be done to figure out a framework for enabling that.   And then you’d have to have a process for safeguarding the information once collected. There’s a cost/benefit tradeoff here.  If we are going to go down this path, it would help to know how bad the problem is that we are trying to solve.

Generally speaking, I would expect that if an app fails to contact a DoH resolver that’s been burned into it, which I wouldn’t recommend, two things will be true.   First, it will have done a DNS query to find the server, so you can track that.   Second, when the resolver doesn’t answer, it will display something in the UI.   And if it doesn’t, that’s a problem with the app, not a problem for the ISP to solve.

This is why I’m puzzled by what you are asking for.   I am not seeing the problem you are trying to solve that we don’t already know how to solve.

To be clear, I am not saying you are wrong.   But if you think there is a problem, you have a reason for thinking it’s a problem.   It’s either a good reason, or not.   If it’s a good reason, you should be able to explain it in such a way that I understand it and see that it is a good reason too.   If you are asking someone else to do that, and don’t actually know what the problem is yourself, we are at a bit of an impasse.

> 3. Review and expansion of the current RFC ( sorry don’t know from memory) that Ekr cited earlier. 

What would you add to it?

---

From: Brian Dickson <[email-masked]>
Date: Wed, 7 Aug 2019 18:57:43 -0700

On Wed, Aug 7, 2019 at 5:17 PM Ted Lemon <[email-masked]> wrote:

> Once the DoH code and infrastructure is in the wild inside browsers it
> will start to pop up in non-browser applications.
>
>
> Glenn, can you explain why this is being presented as a hypothetical?
> The status quo is that this is available to apps.   It’s trivial to do.   I
> did my first tunnel over port 443 over a decade ago, and I was *not*
>  innovating.  Any little bit of malware or privacyware that wants to do
> this can do it today.
>
> I apologize for being blunt here, but this discussion has gotten *really* long,
> and a lot of it is discussion about stuff that is in no way actionable.   I
> think that anything based on the above assertion falls into that category;
> if you think otherwise, it would help if you could explain.
>
>
I'm not (explicitly) speaking on anyone's behalf (i.e. Glenn), but I'll
bite:

Specifically, one important subset of environments is enterprise, i.e. not
coffee-shop, not ISP.
Managed; possibly trusted or untrusted.

The distinction is largely on how {something} knows how to pick a
destination IP address for its tunnel to {destination}.

Prior to DoH, the options were: hardcode an IP; hardcode an FQDN and use
DNS; hardcode a DGA (domain generation algorithm) seed and use DNS. DNS
could mean Do53, or later, DoT.

The control-plane activity and data-plane activity were discernible.
Control-plane == DNS, data-plane == anything else (or possibly even
tunneled over DNS).

A hardcoded IP would be conspicuous by its absence from DNS logs prior to
the tunnel.
The resulting IP could be black-listed if necessary.
A hardcoded FQDN could be black-listed if necessary.
DGA functionality is a harder thing to respond to, especially if 99% of the
names don't exist, and non-existence doesn't break the client functionality
(typically malware).
In most cases, the DNS logs are very useful in detection, and DNS is useful
for mitigation.

In all of the previous cases, the managed environment enabled the network
to do something to enforce the DNS monitoring. It could block off-net port
53 destinations; it could intercept port 53; it could block off-net port
853. DoT would be possible only to a managed end-point.

In an enterprise managed network, it would be fair to say that TOS
implicitly or explicitly requiring acceptance of monitoring DNS in order to
use the network can be assumed.

Here's two points about your "trivial" observation:

   - How did you configure your 443 tunnel end-points?
      - Were they hard-coded IPs (no DNS), or did you look up names in DNS?
      - (Either way, the pre-DoH environment was somewhat manageable in
      detecting and/or responding to these kinds of tunnels.)
      - I.e. malware or privacy-ware doing tunnels like this that don't
      also do their own DNS, would get detected and/or blocked, and those that
      also do their own DNS would get detected and/or blocked (eventually, IMHO
      at least).
   - This all changes with DoH... (see next bit.)

With DoH in use by a browser to an off-site resolver, there is no longer a
discernible "control-plane". There is no DNS traffic prior to new
data-plane traffic, no clue about why new HTTPS destinations would show up.

There are two distinct aspects to this change:

   - There is no ability to observe DNS queries to a white-listed resolver
   (or list of resolvers).
   - There is no way to distinguish vanilla HTTPS traffic hitting some
   arbitrary web site, and DNS over HTTPS traffic hitting a malicious resolver.
   - There is no ability to interfere with DNS resolution to block traffic
   to destinations on the basis of FQDN alone. (The RPZ use case.)
   - When ESNI is added, there will no longer be any way in the data plane
   to identify the FQDN end-point (no HTTP headers or SNI names)
   - Without DNS inspection, there would be no specific indication of which
   source-dest pair among a bunch of sources going to the same destination IP,
   is specifically going to a particular FQDN hosted on the destination IP.
   - Thus, even after discovering an FQDN is hosting malware, there is no
   easy way to determine which client(s) might have been infected (versus any
   previous info from DNS logs and such).
   - Security in the enterprise requires scalability. Scalability in the
   enterprise means network-level enforcement, DNS monitoring, and DNS
   blocking.
   - The alternatives to the above are: give up entirely; Deploy HTTPS
   MitM; require strictly managed hosts exclusively, and deny access to any
   others (BTW, necessary but not sufficient); black-list any software with
   DoH support.

The problem with DoH isn't just that it bypasses the local resolver for
some users; it is that it bypasses the local resolver for all users, and
breaks all the DNS tools currently in use for both detection and prevention.
There is a fundamental issue here; you can't have all the benefits of DoH
without the costs of DoH.

Using DoT, on the other hand, enables a slightly more limited version of
privacy, modulo whatever the available (reachable, non-blocked) DoT
services provide, if any DoT services are available. DoT is at least
nominally compatible with DNS monitoring and blocking at enterprise scale,
so long as the enterprise stands up DoT on their DNS resolvers. The
enterprise would then be able to block off-site DNS (again assuming the
non-existence of DoH within the enterprise, or globally.)

Not everyone agrees that enterprises should be able to block off-site DNS;
my opinion is that that is at best a scoff-law attitude, and not really a
strong position to build global standards upon.
Here's my question: If enterprises were offering guest networks that didn't
block off-site DNS, accessible via DoT, would DoH be necessary in those
enterprises at all?

I think we all agree that there remains a problem that we may not have
consensus on a solution for -- the dissident use case -- but given the
above re-statement of my analysis, I'm not sure there's strong consensus on
the benefits of DoH outweighing the problems caused by DoH.

Note that my position is only a current position; I remain open to
arguments that there are ways to avoid the problems I see in having
DoH-enabled software within enterprise networks.

Brian

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Thu, 8 Aug 2019 02:06:35 +0000

Hi Ted,

There is a lot you wrote here, I’ll try to address it all but I’ll likely miss some.

On Aug 7, 2019, at 6:37 PM, Ted Lemon <[email-masked]<mailto:[email-masked]>> wrote:

On Aug 7, 2019, at 5:44 PM, Deen, Glenn (NBCUniversal) <[email-masked]<mailto:[email-masked]>> wrote:
On Aug 7, 2019, at 5:17 PM, Ted Lemon <[email-masked]<mailto:[email-masked]>> wrote:

While DoH is in browser and there are a few resolvers out there, and while DoH is a point of big discussion inside the IETF it is not a thing that is there yet to the rest of the world.    Hence my qualifying of when it’s available.

DoH is not yet turned on as the default in Firefox or Chrome.  There is still opportunity to examine, discuss, and make new choices.

I learned long ago that asking a narrow question is the wrong approach with security researchers.  They are an inventive and creative lot and often see things from perspectives that I could never imagine.  So I would not ask them to verify or deny a simple assertion; instead I would say, here is this things and this is how it works,  what can you do with it to compromise security or compromise the user?

In the case of apps running in browsers they are subject to the browsers rules and controls, for those using OS services the same applies.   This is not only debugging, this is exposing to the device admin what the device is doing.

How is logging on the users device violating their privacy?    Part of what I’m trying to solve is to give the user visibility on what their app is doing on their behalf. If anyone’s privacy is violated it would be the app’s privacy to do whatever they want in the name of the user.

As others have discussed today on list today, there are issues that people want to consider.

Regards
Glenn

---

From: Ted Lemon <[email-masked]>
Date: Wed, 7 Aug 2019 19:13:08 -0700

I am fairly sure that we all understand that DoH quite effectively prevents us from using DNS as a control plane.  What this means is that you can no longer use DNS as a control plane.   That is a fact of your life now.

What I’m asking is not “is there a problem that’s going to cause you some heartburn?”   There definitely is, and you have my sympathy.   What I’m asking is, “what can the IETF do to help you with this problem.”   I think the answer is “nothing.”

If you think the answer is something other than nothing, can you please state explicitly what you think the answer is?   Do you think the IETF can tell Google and Cloudflare to stop offering DoH, for example?   Or do you want us to write a document explaining how to block port 443 on public DoH resolvers?

Combating malware is a moving target.   DNS was a great tool for doing that, and it probably still is.   But the way it’s going to be used to do that has shifted in the past few years.   Most of the work on mitigating attacks using DNS has been done outside of the IETF, and I suspect that will continue to be the case.   The relationships between entities that do this work will have to change, and will have to account for the privacy promises made by DoH operators.   There is definitely work to do here.   But it’s not work that can be done in the IETF.   The IETF can’t turn back the clock, any more than we could have prevented it ticking forward.

---

From: Ted Lemon <[email-masked]>
Date: Wed, 7 Aug 2019 19:14:24 -0700

> How is logging on the users device violating their privacy?    Part of what I’m trying to solve is to give the user visibility on what their app is doing on their behalf. If anyone’s privacy is vio…

If the app is going to violate the user’s privacy, why would it log information that would show that it had done so?

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 19:14:47 -0700

On Wed, Aug 7, 2019 at 6:35 PM Bret Jordan <[email-masked]> wrote:

> No solution is perfect, but this represents a layered defense that
> organizations rely on.
>

I think the point of Martin's presentation was that this is a poor
decision, and will become increasingly poor. The presentation did
acknowledge that people made this decision--it's right there in the slides.
So, saying that "organizations rely on" this ineffective technique is not
exactly a strong rebuttal.

This has come up once in this thread already, but the problem with the
"defense in depth" argument is that it can be used to rationalize
ineffective measures.

thanks,
Rob

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Thu, 8 Aug 2019 02:18:03 +0000

> On Aug 7, 2019, at 7:14 PM, Ted Lemon <[email-masked]> wrote:
> 

It may not have a choice if it’s running inside a controlled environment such as a browser or and OS.  Again, is it perfect: no; is it one tool in the tool box: yes.

Glenn

---

From: Ted Lemon <[email-masked]>
Date: Wed, 7 Aug 2019 19:23:07 -0700

> It may not have a choice if it’s running inside a controlled environment such as a browser or and OS.  Again, is it perfect: no; is it one tool in the tool box: yes.

Okay.   So you want the IETF to tell browser vendors and O.S. vendors how to do logging?

---

From: "Deen, Glenn (NBCUniversal)" <[email-masked]>
Date: Thu, 8 Aug 2019 02:23:28 +0000

Hi Rob,

On Aug 7, 2019, at 7:15 PM, Rob Sayre <[email-masked]<mailto:[email-masked]>> wrote:

On Wed, Aug 7, 2019 at 6:35 PM Bret Jordan <[email-masked]<mailto:[email-masked]>> wrote:
Well said Glenn.  I completely agree.

No solution is perfect, but this represents a layered defense that organizations rely on.

I think the point of Martin's presentation was that this is a poor decision, and will become increasingly poor. The presentation did acknowledge that people made this decision--it's right there in the slides. So, saying that "organizations rely on" this ineffective technique is not exactly a strong rebuttal.

I'm still very much missing the why and the supporting facts on this assertion.

This has come up once in this thread already, but the problem with the "defense in depth" argument is that it can be used to rationalize ineffective measures.

Again, where is the data to show they are ineffective?

 I get that there is a view that doesn't like this approach, but I also get that there are a heck of lot of people and organizations globally that are quite satisfied with the results they currently get from these controls.

.
.
.

I'm going to follow Stephen F's  example and self silence for a bit on this list. I've made my points heard and anything else  will be me repeating myself.

-Glenn

---

From: Paul Ebersman <[email-masked]>
Date: Wed, 07 Aug 2019 20:27:56 -0600

bjordon> No solution is perfect, but this represents a layered defense
bjordon> that organizations rely on.

sayrer> I think the point of Martin's presentation was that this is a
sayrer> poor decision, and will become increasingly poor.

And you base this on what beyond your own personal opinion? Have you had
extensive experience supporting large enterprise security customers?
Have you worked at a large ISP with customers using these services? Do
you have data showing that the efficacy has decreased, where and how
much?

There are vendors and ISP operators on this list who have all said that
this is effective and useful. Not 100% effective, but effective.
Effective for millions of users today. Sure, all defenses get less
effective with time. Not news. But you haven't shown that DNS filtering
isn't an effective tool right now.

Can we move on?

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 19:37:13 -0700

On Wed, Aug 7, 2019 at 7:27 PM Paul Ebersman <[email-masked]> wrote:

> And you base this on what beyond your own personal opinion?

That's what the slides say. One slide says "is an effective control
surface", the next one crosses it out and says "was an effective control
surface".

Or, do you need more data on this?

> Have you had

I have worked at them, and they do not rely on DNS.

> Can we move on?

Firstly, this not a particularly polite way of telling someone to shut up.
Secondly, move on to what? Some document that says "you MUST use the ISP
DNS"? No thanks.

thanks,
Rob

---

From: Bret Jordan <[email-masked]>
Date: Wed, 7 Aug 2019 21:18:19 -0600

I agree fully with what Glenn has stated.   For those that day DNS is not a good control plan, where is your research and data to back that up? 

Bret 

Sent from my Commodore 128D

> On Aug 7, 2019, at 8:23 PM, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
> 
> 
> Hi Rob,
> 
> On Aug 7, 2019, at 7:15 PM, Rob Sayre <[email-masked]> wrote:
> 
> I’m still very much missing the why and the supporting facts on this assertion.
>> 
>  I get that there is a view that doesn’t like this approach, but I also get that there are a heck of lot of people and organizations globally that are quite satisfied with the results they currently get from these controls.
> 
> .
> .
> .
> 
> I’m going to follow Stephen F’s  example and self silence for a bit on this list. I’ve made my points heard and anything else  will be me repeating myself.
> 
> -Glenn 
> 
>

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 21:39:40 -0700

> Again, where is the data to show they are ineffective?

I'm not sure the burden of proof should be on me.

The arguments I hear basically amount to "well, people made a decision to
use DNS for these purposes". But every time I look at a claim, I find other
solutions that do the job much better. Parental controls: see Apple or
Microsoft. Enterprise: see all of the various management tools that root
the OS and all browsers. Malware: OS and browser security. Access control:
people just use VPNs--they are high-grossing apps. So, there's your data.

Perhaps we should shift the burden of proof to the DNS policy fans. What's
so great about using DNS for any of these purposes?

thanks,
Rob

---

From: Joe Abley <[email-masked]>
Date: Thu, 8 Aug 2019 00:56:49 -0400

Hi Martin,

> 3. Why would someone want to use DNS to exert control?

Why are these useful questions, given the (surely indisputable) knowledge that DNS is widely used as a control plane today, and that many organisations and end-users spend non-trivial amounts of time and money to do so?

Is there a benefit I'm not seeing to returning to architectural first principles and ignoring actual deployed infrastructure? If so, we should also throw out BGP which is frankly quite the dog's breakfast.

Joe

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 8 Aug 2019 07:02:30 +0200 (CEST)

On Wed, 7 Aug 2019, Rob Sayre wrote:

> Perhaps we should shift the burden of proof to the DNS policy fans. 
> What's so great about using DNS for any of these purposes?

It keeps the regulator happy. Do not make the regulator angry. Bad things 
happen then. A lot worse than we have today.

See Ralf Webers email from yesterday. DNS based blocking is deemed "good 
enough" by regulator today. They want blocking or policy implemented. If 
DNS stops being "good enough", they will demand more intrusive measures to 
make things work.

https://www.engadget.com/2019/07/31/how-ag-barr-is-going-to-get-encryption-backdoors/
https://www.techradar.com/news/uk-porn-block-everything-you-need-to-know
https://www.telenor.com/media/press-release/telenor-introduces-child-pornography-filter-in-sweden

Better give them a blunt tool that kind of works and is not super 
intrusive, than the alternative.

If browser vendors change the default to DoH via some quad-X without the 
agreed policies between industry and regulator, the regulator (and other 
bodies) will wake up in a lot of countries. Some already have.

I'd prefer not to get regulators involved and solve this as an industry. 
Things typically get very complicated when regulators and politicians get 
involved.

---

From: Eric Rescorla <[email-masked]>
Date: Wed, 7 Aug 2019 22:25:32 -0700

> Again, where is the data to show they are ineffective?

Isn't the entire premise of this conversation that people who use this
technique are unhappy that DoH allows people to evade it?

-Ekr

>  I get that there is a view that doesn’t like this approach, but I also
> get that there are a heck of lot of people and organizations globally that
> are quite satisfied with the results they currently get from these controls.
>
> .
> .
> .
>
> I’m going to follow Stephen F’s  example and self silence for a bit on
> this list. I’ve made my points heard and anything else  will be me
> repeating myself.
>
> -Glenn
>
>
> --
> Add mailing list
> [email-masked]

---

From: Brian Dickson <[email-masked]>
Date: Wed, 7 Aug 2019 22:58:46 -0700

Sent from my iPhone

> On Aug 7, 2019, at 7:13 PM, Ted Lemon <[email-masked]> wrote:
> 

What I think the IETF can do is document this, so it is quite explicit and crystal clear.

> DoH quite effectively prevents us from using DNS as a control plane.

This sums it up.

> If you think the answer is something other than nothing, can you please state explicitly what you think the answer is?   Do you think the IETF can tell Google and Cloudflare to stop offering DoH, f…

The problem isn’t entirely the public resolvers, although DoH using them does cause some breakage in some enterprise environments. The bigger problem is that any server can stand up DoH, including malicious DoH servers responding with forged responses.

The IETF could mandate (as in, mandatory to implement) client DNSSEC validation. That would go a long way to mitigation of the malicious server problem.

> 
> Combating malware is a moving target.   DNS was a great tool for doing that, and it probably still is.   But the way it’s going to be used to do that has shifted in the past few years.   Most of the work on mitigating attacks using DNS has been done outside of the IETF, and I suspect that will continue to be the case.   The relationships between entities that do this work will have to change, and will have to account for the privacy promises made by DoH operators.  

I’m not sure I’d classify those entities as DoH operators. Yes, they make DNS available via DoH, but also do so via DoT, as well as Do53.

I’d prefer to call them public resolvers over encrypted transports.

> There is definitely work to do here.   But it’s not work that can be done in the IETF.   The IETF can’t turn back the clock, any more than we could have prevented it ticking forward.

I think there are lots of things that can be done in various venues, with different impact and perceptions. 

Currently the DoH bus is being driven by the browser folks. The perception from the outside is that DoH was atypical in how it went through the IETF process, and not enough attention was given to these issues before 8484 was published, IMNSHO.

Any re-bottling would have to be driven by the browser folks. If, and it is an enormous if, they decide to abandon DoH, there would be no residual need for the quad X folks to serve DoH, since DoT meets all the privacy needs for DNS. The IETF can’t turn back the clock, but the browsers can.

The IETF should provide guidance to enterprise networks on the new issues that DoH has created, so the they can develop responses on an expedited basis.

IMHO, the IETF has to own DoH, which means it needs to provide this advice before the browsers turn on DoH by default.

Brian

---

From: Adam Roach <[email-masked]>
Date: Thu, 8 Aug 2019 01:02:06 -0500

> The issue I’m raising in this thread is the question of how researchers will be able to examine  what applications are up to with DoH.

I see that. You seem to be assuming that non-browser native apps would 
be using DoH to collect information about user behavior. My line of 
questioning is trying to figure out why you think they would do that, 
since they currently collect information about user behavior and send it 
back using their own, non-DoH protocols today. Those bespoke protocols 
are at least as opaque as DoH, and infinitely more flexible in terms of 
data that can be sent back.

/a

---

From: Brian Dickson <[email-masked]>
Date: Wed, 7 Aug 2019 23:25:25 -0700

Sent from my iPhone

> On Aug 7, 2019, at 10:25 PM, Eric Rescorla <[email-masked]> wrote:
> 
> 
> 
>> On Wed, Aug 7, 2019 at 7:23 PM Deen, Glenn (NBCUniversal) <[email-masked]> wrote:
>> 
>> Hi Rob,
>> 
> Isn't the entire premise of this conversation that people who use this technique are unhappy that DoH allows people to evade it?
> 
> -Ekr

No.

People are unhappy that browsers switching to DoH-on-by-default without informed user consent forces all users to bypass any and all DNS-as-control-and-detection-plane without the impacted users necessarily having any understanding of the implications or even being aware. In theory, some of those implications can be personal, especially in restrictive environments. It is an opt-out vs opt-in situation.

The exposure of the control enabling opt-in or opt-out to non-admin users weakens the overall host security model.

The refusal to do client dnssec validation further weakens the resulting host+app combo.

Putting this inside enterprise networks with no ability to detect the general situation, or to detect aberrant behavior, or identify what resolver is being used makes some people particularly cranky.

It’s not just one aspect of this, it is the sum total.

Hope this clarifies things.

Brian

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 23:32:05 -0700

On Wed, Aug 7, 2019 at 11:25 PM Brian Dickson <[email-masked]>
wrote:

> Putting this inside enterprise networks with no ability to detect the
> general situation, or to detect aberrant behavior, or identify what
> resolver is being used makes some people particularly cranky.
>

The issue with this argument is that any secure network doesn't allow
random devices.

can we do better?,
Rob

---

From: Rob Sayre <[email-masked]>
Date: Wed, 7 Aug 2019 23:51:08 -0700

> technique are unhappy that DoH allows people to evade it?

Let's be fair. Encrypted DNS will disrupt some lines of business. Shouldn't
we allow anyone with an accidental ad network to say "policy" or
"operational concerns"?

No one is obligated to listen, but they should still be able to have their
voice heard.

thanks,
Rob

---

From: "Robert Mortimer" <[email-masked]>
Date: Thu, 08 Aug 2019 08:22:32 +0100

On 08/08/2019 01:07:29, Deen, Glenn (NBCUniversal) <[email-masked]> wrote:

DoH code is already in use in non-browser mobile applications (at least), not all of which are doing certificate verification and not all of which are using DoH purely for "normal" DNS lookups. These are applications available in the main google and apple app stores, so not overtly dodgy downloads.

I've seen about half a dozen or so identifiable mobile applications that fall into the category of not-browser apps, not doing certificate verification. I'd assume that there are more that actually do certificate verification and so I don't have visibility of them.
-- 
Robm
873
  "Ask not what I can do for the stupid, 
         but what the stupid can do for me" - Graeme Garden

---

From: "Ralf Weber" <[email-masked]>
Date: Thu, 08 Aug 2019 13:46:33 +0200

On 8 Aug 2019, at 4:13, Ted Lemon wrote:

> I am fairly sure that we all understand that DoH quite effectively 
> prevents us from using DNS as a control plane.  What this means is 
> that you can no longer use DNS as a control plane.   That is a fact of 
> your life now.
As stated by me and others that depends on what you see as effective and 
of course how DoH servers are deployed. It is no longer a 100% effective 
control plane, but to be honest it never was.

> What I’m asking is not “is there a problem that’s going to cause 
> you some heartburn?”   There definitely is, and you have my 
> sympathy.   What I’m asking is, “what can the IETF do to help you 
> with this problem.”   I think the answer is “nothing.”
Well we could describe deployment considerations/guidelines for DoH 
clients/servers. We could work on mechanisms to discover local DoH 
servers. There is a lot of stuff we could work on and most of it already 
has been brough up in the add bof or mailing lists.

> If you think the answer is something other than nothing, can you 
> please state explicitly what you think the answer is?   Do you think 
> the IETF can tell Google and Cloudflare to stop offering DoH, for 
> example?   Or do you want us to write a document explaining how to 
> block port 443 on public DoH resolvers?
Well if Google (who I believe have stated that) and Cloudflare deploy 
there DoH servers in a way that e.g enterprises could block them via IP 
filtering and we could maybe write that up as a guideline for DoH server 
operators it would surely help enterprises. Similar to if we had a 
protocol to discover local DoH server or the presence of split 
namespaces or policies which DoH clients would follow this would also 
help a lot of use cases where DNS is used as a control plane.

So long
-Ralf

---

From: Bret Jordan <[email-masked]>
Date: Thu, 8 Aug 2019 09:03:25 -0600

You are correct Mikael. 

Thanks,
Bret

> On Aug 7, 2019, at 11:02 PM, Mikael Abrahamsson <[email-masked]> wrote:
> 
>> Perhaps we should shift the burden of proof to the DNS policy fans. What's so great about using DNS for any of these purposes?
> 
> It keeps the regulator happy. Do not make the regulator angry. Bad things happen then. A lot worse than we have today.
> 
> See Ralf Webers email from yesterday. DNS based blocking is deemed "good enough" by regulator today. They want blocking or policy implemented. If DNS stops being "good enough", they will demand more intrusive measures to make things work.
> 
> Better give them a blunt tool that kind of works and is not super intrusive, than the alternative.
> 
> If browser vendors change the default to DoH via some quad-X without the agreed policies between industry and regulator, the regulator (and other bodies) will wake up in a lot of countries. Some already have.
> 
> I'd prefer not to get regulators involved and solve this as an industry. Things typically get very complicated when regulators and politicians get involved.
> 
> --

---

From: Ted Lemon <[email-masked]>
Date: Thu, 8 Aug 2019 08:43:25 -0700

> People are unhappy that browsers switching to DoH-on-by-default without informed user consent forces all users to bypass any and all DNS-as-control-and-detection-plane without the impacted users ne…

This may be true, but two questions come up:
- Which people?  Users, or operators?
- How do the privacy and security properties of DoH compare with the privacy and security properties of SP recursive DNS?

Personally, I don’t actually know on a practical level whether I’m better off using DoH or SPrDNS, but given some of the disclosures we’ve seen historically about ISP’s behavior with respect to passive snooping, it’s certainly not a given that SPrDNS is better.

My point here is not to criticize what you are saying, but just to encourage you to go a little deeper on your assertions.  “People” is way too general, for example.

I don’t know that it would be a bad thing to write a document talking about the tradeoffs that we face here.   It might be very difficult, though, if we can’t be crystal clear about what we’re discussing.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 8 Aug 2019 08:49:18 -0700

On Aug 8, 2019, at 4:46 AM, Ralf Weber <[email-masked]> wrote:
> Well if Google (who I believe have stated that) and Cloudflare deploy there DoH servers in a way that e.g enterprises could block them via IP filtering and we could maybe write that up as a guideline for DoH server operators it would surely help enterprises. Similar to if we had a protocol to discover local DoH server or the presence of split namespaces or policies which DoH clients would follow this would also help a lot of use cases where DNS is used as a control plane.

These aren’t unreasonable proposals for new work.   I believe we are already working on the latter project, and I don’t oppose that in principle as long as we don’t create security vulnerabilities in the process.

As far as a “best practices” document, we could certainly write one, and it would make sense for it to say what you are suggesting.  The question I would have is, who is the target audience?   If Google and Cloudflare are already doing it, who are you speaking to?

---

From: Brian Dickson <[email-masked]>
Date: Thu, 8 Aug 2019 09:08:17 -0700

On Thu, Aug 8, 2019 at 8:43 AM Ted Lemon <[email-masked]> wrote:

> People are unhappy that browsers switching to DoH-on-by-default without
> informed user consent forces *all* users to bypass any and all
> DNS-as-control-and-detection-plane without the impacted users necessarily
> having any understanding of the implications or even being aware. In
> theory, some of those implications can be personal, especially in
> restrictive environments. It is an opt-out vs opt-in situation.
>
>

So far, everyone to whom I've explained what we (in this list) understand
about DoH and its various implications. That list includes users, network
operators, security ops folks, authority operators, recursive operators in
enterprises.

> - How do the privacy and security properties of DoH compare with the
> privacy and security properties of SP recursive DNS?
>

I'm not familiar with the acronym "SP", in this context.

But, I'd also suggest DoT as an alternative to DoH, since those are very
closely matched, even if DoT is not (yet) natively supported by browsers.

The security and privacy of DoT can be achieved by using a DoT resolver in
"forwarder" mode on your local host, and pointing to that in whatever the
host's DNS resolver choice configuration controls are.

The benefit of doing that are, the benefits apply to all apps equally.

If the DoT resolver supports DoH clients, then you can configure your
browser to point at your local host's resolver/forwarder in the DoH config,
and get all the fringe benefits on DoH within the browser (if/when those
show up).

Plus, you can configure the local resolver to do DNSSEC validation.

I'd encourage using forward-only mode, so as to gain all the privacy and
scaling benefits of continuing to use a non-local resolver, regardless of
choice of resolver.

Having enterprise resolvers enabling DoT would be a very good thing, and
mostly these days that's just a code upgrade and a config change, for most
major resolver packages.

>
> Personally, I don’t actually know on a practical level whether I’m better
> off using DoH or SPrDNS, but given some of the disclosures we’ve seen
> historically about ISP’s behavior with respect to passive snooping, it’s
> certainly not a *given* that SPrDNS is better.
>
> My point here is not to criticize what you are saying, but just to
> encourage you to go a little deeper on your assertions.  “People” is way
> too general, for example.
>

I understand. I was using "people" generally as an answer to an earlier
question.

>
> I don’t know that it would be a bad thing to write a document talking
> about the tradeoffs that we face here.   It might be very difficult,
> though, if we can’t be crystal clear about what we’re discussing.
>

For sure.

Brian

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 8 Aug 2019 18:46:46 +0200 (CEST)

On Thu, 8 Aug 2019, Ted Lemon wrote:

> Personally, I don’t actually know on a practical level whether I’m 
> better off using DoH or SPrDNS, but given some of the disclosures we’ve 
> seen historically about ISP’s behavior with respect to passive snooping, 
> it’s certainly not a given that SPrDNS is better.

That is exactly the point. There is no one-size-fits-all. Speaking from 
personal view:

If I am at home I want to use my ISP provided resolver.
If I am at some random open wifi I want to use a cloud DoH resolver.

This is not possible today, and I'd like this to be configurable by me.

Otoh, if my browser vendor decides to without asking me in a unilateral 
action during a regular version upgrade change that I should use some (to 
me) random US based DoH provider with no GDPR protection I'm going to take 
that as huge betrayal of my trust in that product.

---

From: Ted Lemon <[email-masked]>
Date: Thu, 8 Aug 2019 09:49:50 -0700

> This is not possible today, and I'd like this to be configurable by me.

Okay, but people who are going to configure this are a tiny minority.   Is it worth spending time documenting best practices to support this use case?

---

From: Paul Ebersman <[email-masked]>
Date: Thu, 08 Aug 2019 10:53:54 -0600

swmike> This is not possible today, and I'd like this to be configurable
swmike> by me.

mellon> Okay, but people who are going to configure this are a tiny
mellon> minority. Is it worth spending time documenting best practices
mellon> to support this use case?

Now we get back into some kind of semi-useful autodiscovery and more
thoughtful default values.

I think Mikael's use case would be useful to many folks if they didn't
have to do a lot of poking around or it was done for them.

---

From: Mikael Abrahamsson <[email-masked]>
Date: Thu, 8 Aug 2019 19:44:20 +0200 (CEST)

On Thu, 8 Aug 2019, Ted Lemon wrote:

> Okay, but people who are going to configure this are a tiny minority. 
> Is it worth spending time documenting best practices to support this use 
> case?

I don't know what "this use case" you're refering to out of what I wrote.

I want to be asked each network attachment if I trust the operator of this 
network. In my mind that's the entire thing we're discussing here. Windows 
is doing this question when I connect somewhere new, and they have a 
decently large installed base that they probably have some experience with 
how users handle this.

There is no "DoH via cloud-based quad-X provider is the right thing" vs 
"system resolver is the right thing" heuristic that'll get it right even 
most of the time.

Some people are claiming that ISPs and governments are crooks and not to 
be trusted. Other people are claiming cloud-based quad-X providers are 
commercial greedy data-gathering data-suckers. Both and none of this can 
be true depending on your views, where you are in the world, and a lot of 
other factors.

There is no single answer to this that is correct for all situations and 
all users.

This is not a one-size-fits-all. I can't believe some people are making 
this extremely complicated issue into "let's use cloud-based quad-X DoH 
for the entire user-base because that's the right answer". It's just 
mindboggling.

---

From: Christian Huitema <[email-masked]>
Date: Thu, 8 Aug 2019 11:43:42 -0700

On 8/8/2019 9:49 AM, Ted Lemon wrote:
> On Aug 8, 2019, at 9:46 AM, Mikael Abrahamsson <[email-masked]
> <mailto:[email-masked]>> wrote:
>   Is it worth spending time documenting best practices to support this
> use case?

Ted,

There is actually a similar example in the Windows world. Users who
connect to a network are asked whether they consider the network
"public" or "private". If the network is private, the system will enable
file and print sharing over the network, which is convenient but also
significantly increases the attack surface. If the network is public,
the system will close that surface by default. There is a system API
that exposes the status of the network
(https://docs.microsoft.com/en-us/windows/win32/api/netlistmgr/nf-netlistmgr-inetwork-getcategory),
and applications can use that to change their behavior if they wish.

It would not be absurd to configure the browser and other applications
to use DoH on a public network and the local DNS service on a  private
network, or in an enterprise network.

-- Christian Huitema

---

From: Paul Wouters <[email-masked]>
Date: Thu, 8 Aug 2019 15:14:20 -0400

> On Aug 8, 2019, at 14:43, Christian Huitema <[email-masked]> wrote:
> 
> 
> It would not be absurd to configure the browser and other applications to use DoH on a public network and the local DNS service on a  private network, or in an enterprise network.
> 

I have said this many times too, for example in relation to updating the search domains or not. On Linux, you still get starbucks.com in /etc/resolv.conf

The designation of public or private can only be made by a human though as often a coffeshop wifi running WPA cannot be detected as a public network, as it appears as an enterprise network.

Ideally all this would be in the OS hotspot handling code and the browsers would be able to just query the status as known by the OS.

Paul

---

From: "Martin Thomson" <[email-masked]>
Date: Fri, 09 Aug 2019 11:58:50 +1000

On Thu, Aug 8, 2019, at 14:57, Joe Abley wrote:
> Why are these useful questions, given the (surely indisputable) 
> knowledge that DNS is widely used as a control plane today, and that 
> many organisations and end-users spend non-trivial amounts of time and 
> money to do so?

Thanks Joe, that's a salient question.

I think that there is a reasonable case for saying that conditions have changed.  As you have noted, "OTT" DNS is increasingly prevalent, as is encryption and even DNSSEC.

As we've seen, there is a good case to be made for various uses of DNS in protection against different classes of attack.  For instance, we've had lengthy discussions with DNS providers about the forensic capabilities that logs provide.  As an aside, you can expect to see adjustments to Mozilla policies along those lines.

But it's not clear how this example reads on the question, except to the extent that it forms another point in the perpetual  debate about where intelligence and responsibility (necessarily) resides.  For that specific case however, the resolver that an application chooses might reasonably provide the same capabilities.  Indeed, those services is a core value proposition of several "OTT" DNS providers.

If the point of this discussion is to dissuade applications from performing their own resolution, or choosing who they partner with to do that, then I suggest it's not going to be productive.  We can all, with perfectly defensible reasons, claim to be "for the users" [1] in such a discussion.

The approach we've taken with the capport working group, which is looking at a very narrowly targeted piece of this larger puzzle is less naturally adversarial, even if it isn't completely free of contention.  The idea there was to acknowledge a specific reason for networks to communicate with endpoints and provide a clear, alternative path for that communication.  As an aside, framing that as a channel for information - as opposed to a mechanism for exerting control - had a huge positive impact on those discussions.

Part of why I asked these questions was not to be dismissive of the investment that has been made in systems that use DNS to achieve their goals, but to try to better identify what needs to be preserved as opposed to those things that will eventually need to find alternative paths to the same end goal.  There's a fair spread here, clearly.  For instance, if your use case is pervasive surveillance, then I hope you will find that to be less well-supported in the future.

I realize now that with all that is going on in this space, asking very general questions is unproductive.  The very general answers I received show that asking in an abstract form was an error.

What I really want is to understand specific ways in which networks use DNS to affect specific ends.  Understanding those cases in some depth - in the same way that I believe that I appreciate the captive portals case - might reveal how they depend on properties of the system we used to have (that is, open to all forms of surveillance, interception, and modification).

You might rightly point out that such an endeavour is foolishly broad in scope.  But I don't think that it is necessary to understand the system in totality.  Working through a few of the most critical cases might be enough.  My hope for today, which may be vain indeed, is that working through the those important cases in some detail will lead to better information about how other cases might be handled.

Cheers,
Martin

p.s., I'm restricting myself to a single email response per day.  I apologize for its length, but I hope that spending more time on a single email is more respectful of others' time than a less thoughtful response.

[1] https://tools.ietf.org/html/draft-nottingham-for-the-users

---

From: Rob Sayre <[email-masked]>
Date: Thu, 8 Aug 2019 21:16:31 -0700

On Thu, Aug 8, 2019 at 6:59 PM Martin Thomson <[email-masked]> wrote:

>
> You might rightly point out that such an endeavour is foolishly broad in
> scope.  But I don't think that it is necessary to understand the system in
> totality.  Working through a few of the most critical cases might be
> enough.  My hope for today, which may be vain indeed, is that working
> through the those important cases in some detail will lead to better
> information about how other cases might be handled.
>

I agree that captive portals are an uncontentious problem to solve here.
Something similarly ubiquitous and uncontentious might be resolving
intranet resources (phonebooks, wikis, issue trackers, etc). Those often
don't resolve outside of the office (or its VPN). It seems like always
using Cloudflare et al. would break this use case, although I don't mean to
imply anyone is /trying/ to break it.

thanks,
Rob

---

From: Bret Jordan <[email-masked]>
Date: Fri, 9 Aug 2019 08:15:22 -0600

Split DNS is a critical use case.  It is not just development systems that are not addressable on the outside, but nearly all HR and financial systems as well.  It is very common for organizations to have several orders of magnitude more internal names than external names.

Thanks,
Bret

> On Aug 8, 2019, at 10:16 PM, Rob Sayre <[email-masked]> wrote:
> 
> On Thu, Aug 8, 2019 at 6:59 PM Martin Thomson <[email-masked] <mailto:[email-masked]>> wrote:
> 
> I agree that captive portals are an uncontentious problem to solve here. Something similarly ubiquitous and uncontentious might be resolving intranet resources (phonebooks, wikis, issue trackers, etc). Those often don't resolve outside of the office (or its VPN). It seems like always using Cloudflare et al. would break this use case, although I don't mean to imply anyone is /trying/ to break it.
> 
> thanks,
> Rob
> 
> -- 
> Add mailing list
> [email-masked]

---

From: Bob Harold <[email-masked]>
Date: Fri, 9 Aug 2019 10:42:06 -0400

As an example of split DNS:
Split DNS is needed when we have a VPN from our internal network to a cloud
provider (like AWS) that uses RFC1918 addresses and NAT to public IP
addresses.  Things that can be reached over the VPN must use the internal
RFC1918 address, but everything else needs to use the public address.

-- 
Bob Harold

On Fri, Aug 9, 2019 at 10:15 AM Bret Jordan <[email-masked]> wrote:

> Split DNS is a critical use case.  It is not just development systems that
> are not addressable on the outside, but nearly all HR and financial systems
> as well.  It is very common for organizations to have several orders of
> magnitude more internal names than external names.
>
>
> Thanks,
> Bret

---

From: Ted Lemon <[email-masked]>
Date: Fri, 9 Aug 2019 08:42:54 -0700

On Aug 9, 2019, at 7:15 AM, Bret Jordan <[email-masked]> wrote:
> 
> Split DNS is a critical use case.  It is not just development systems that are not addressable on the outside, but nearly all HR and financial systems as well

Yes, indeed, this has to work. Would be nice if it could be trusted based on authenticity rather than a leap of faith.