Subject: Re: [DNSOP] Concerns around deployment of DNS over HTTPS (DoH)

From: Kenji Baheux <[email-masked]>
Date: Wed, 13 Mar 2019 11:33:14 +0900

*(Sincere apologies about the multi-posting but the discussion seems to be
happening in different places...)*

Hi,

I'm involved with Chrome's DoH efforts.

I've noticed a few drafts listing concerns about certain types of
deployment for DoH. It appears that the key concerns are based on
assumptions about various browsers' plans. So, to address
misunderstandings, I'd like to share some background, and the high level
principles that guide our work on DoH in Chrome.

Our motivations in pursuing DoH in Chrome is to offer our users a better
user experience:

   -

   Stronger privacy and security.
   -

   Hopefully, some performance wins.

Principles and illustrative* examples:

   1.

   Provide our users with meaningful choice and control, e.g. allow
   end-users/admins to control and configure the feature, whether they want to
   use a custom DoH server, or just keep on using their regular DNS.
   2.

   Don't surprise our users, e.g. don't silently force a different provider.
   3.

   Continue to support admins for Education and Enterprise use cases, and
   parents for family use cases, e.g. prevent students/employees/kids from
   accessing unsafe/inappropriate websites.

*: not necessarily our actual concrete plans (still WIP for the most part,
subject to change due to unforeseen implementation hurdles, or informed by
feedback and discussion, etc.), only shared as a comprehension aid.

Tentative plans:

   -

   We are considering a first milestone where Chrome would do an automatic
   upgrade to DoH when a user’s existing resolver is capable of it.
   -

   There are some unanswered questions about how we will be doing that
   discovery, and would welcome input from the community. Perhaps, a good
   topic for IETF 104.
   -

   For Education and Enterprise use cases, we believe that a group policy
   to disable and/or configure the feature will be enough.
   -

   For family use cases, we believe that allowing users to keep using their
   existing solution or opting into a DoH compatible solution (if available)
   will be enough.
   -

   There are no plans to force any specific resolver without user consent /
   opt-in (see principle #2)

We believe that starting with an automatic upgrade approach should address
a lot of the concerns expressed in the various drafts, because it gives a
chance to existing players to play an active role in providing stronger
privacy and security to end-users.

PS: I won't be able to join IETF 104 to discuss this face to face, but I
will see if someone from our side can represent us.

-- 
Kenji BAHEUX
Product Manager - Chrome
Google Japan

-- 
Kenji BAHEUX
Product Manager - Chrome
Google Japan

---

From: Paul Vixie <[email-masked]>
Date: Wed, 13 Mar 2019 07:41:32 +0000

On Wednesday, 13 March 2019 02:33:14 UTC Kenji Baheux wrote:
> ...

thank you for this information. can i request that you offer DoT as a 
solution, not just DoH? they offer the same capabilities of secrecy and 
authenticity, but DoT can be cheaply disabled by the network operator, whereas 
a malicious user or app using DoH will be very expensive to detect or prevent 
at the network level.

vixie

---

From: Kenji Baheux <[email-masked]>
Date: Wed, 13 Mar 2019 19:20:58 +0900

> a malicious user or app using DoH will be very expensive to detect or

I'm not sure I understand in which scenario this would provide user
benefits over DoH.
I'm also not sure why / how a browser could prevent these type of issues
from happening by merely shipping DoT on top of DoH.

If there is a malicious user or app on a network that someone is trying to
protect, isn't the very existence of these players the actual issue that
needs to be addressed?

On the other hand, for the set of use cases that I do understand, the
ability for a network operator (not the admin) to cheaply disable DoT, and
as a result downgrading DNS to vanilla queries, appears to be a significant
downside for end-users. Taken to an extreme, it seems to imply that
shipping DoT is effectively the same as not shipping it.

What am I missing?

-- 
Kenji BAHEUX
Product Manager - Chrome
Google Japan

---

From: Erik Kline <[email-masked]>
Date: Wed, 13 Mar 2019 12:23:55 -0700

> protect, isn't the very existence of these players the actual issue that

I tend to think this is the real issue.  Any app can craft its own
non-cleartext-DNS name resolution service; DoH makes it a bit easier
perhaps, but not much (vis. JSON DNS, etc...).

My suspicion is that controlling a network's DNS is less and less likely to
be a decent control point for network security w.r.t. to the craftier
apps.  I'm sure the monitoring and interference with things looking up
"really-evil.evil" still has some value.  But much more sophistication is
probably required nowadays to deal with even moderately competent
adversaries...I suspect.

---

From: nusenu <[email-masked]>
Date: Wed, 13 Mar 2019 19:52:00 +0000

>    upgrade to DoH when a user’s existing resolver is capable of it.

Thanks for sharing these insides.

Are you also considering to implement support for this I-D currently in the DoH WG 
(once it is a RFC)?
https://datatracker.ietf.org/doc/draft-ietf-doh-resolver-associated-doh/

kind regards,
nusenu

-- 
https://twitter.com/nusenu_

---

From: Paul Vixie <[email-masked]>
Date: Wed, 13 Mar 2019 23:06:51 +0000

On Wednesday, 13 March 2019 10:20:58 UTC Kenji Baheux wrote:
> > ... can i request that you offer DoT as a

i plan (speaking as a network operator) to deploy and use DoT, but not DoH. i 
would like my users to have the benefits DoT offers, since it is an 
improvement over plain text in some parts of the topology here.

> from happening by merely shipping DoT on top of DoH.

i intend to cooperate with all network operators. when i travel, if DoT fails, 
i will know that the network operator does not want me to have privacy. i can 
then make an informed choice about whether to violate that policy, or not.

DoH deliberately obfuscates that condition, which is dangerous to me.

> protect, isn't the very existence of these players the actual issue that

i'll be happy to discuss the ways in which controlled RDNS helps detect such 
malicious activity, as a valuable contribution to addressing that activity. 
but i think i err'd in mentioning it here, since as you say, it's not the 
browser maker's problem.

> shipping DoT is effectively the same as not shipping it.

encryption and authenticity of DNS transactions has value even when the user 
and the network operator intend to cooperate. i'm not going to deploy DoH, 
however, nor use it when i travel. DoT, by working only when the user and the 
network operator are in cooperation, is a valuable technology for my needs as 
both an end user and a network operator.

thank you very much for your time.

vixie

---

From: Paul Vixie <[email-masked]>
Date: Wed, 13 Mar 2019 23:10:33 +0000

> perhaps, but not much (vis. JSON DNS, etc...).

if you guys would appreciate a half day seminar on network security economics, 
in which the value of anomalousness will figure prominently, let's meet up.

> apps.

your suspicion directly contradicts both my long-term and recent experience.

> I'm sure the monitoring and interference with things looking up

alas, meeting only the most competent adversaries is not a choice we can make.

vixie

---

From: Erik Kline <[email-masked]>
Date: Wed, 13 Mar 2019 16:12:50 -0700

> in which the value of anomalousness will figure prominently, let's meet up.

I'd be a fool to turn down such an offer.  Thank you.

---

From: Wes Hardaker <[email-masked]>
Date: Fri, 22 Mar 2019 21:03:56 -0700

Kenji Baheux <kenjibaheux=[email-masked]> writes:

>   * We are considering a first milestone where Chrome would do an automatic

Sorry for the delayed question, but with respect to this bullet:

1) Do you have evidence that DOH is faster than DOT, since speed was one
of your goals?

2) What other reasons are you considering when doing DOH instead of DOT
to protect privacy.  Specifically, you're preferring DOH but your stated
goals are "Stronger privacy and security." and "Hopefully, some
performance wins.", without providing rational for each of the potential
solutions.  DNS plain clearly doesn't meet the first, but likely does
the second.  But you fail to provide a goal that distinguishes why you'd
prefer DOT vs DOH to meet both these goals.

-- 
Wes Hardaker
USC/ISI

---

From: Paul Vixie <[email-masked]>
Date: Sat, 23 Mar 2019 13:37:40 -0700

Wes Hardaker wrote on 2019-03-22 21:03:
> 1) ...
> 2) ...

while i feel and echo wes's two questions, mine is different.

if all you have is an ip address (say, from dhcp or resolv.conf), how 
would you decide whether the https endpoint you found at that address, 
was using an x.509 key you had any reason to trust? https wants names.

i've run into this before. http://dot.tt.ed.quad/ is an easy grab, but i 
don't know how to negotiate for https://dot.tt.ed.quad/. if this is a 
solved problem, then i apologize to all present, for not doing my 
homework before opening up in public.

-- 
P Vixie

---

From: Olli Vanhoja <[email-masked]>
Date: Sat, 23 Mar 2019 22:09:49 +0100

If I'm not mistaken, currently the solution used by at least Cloudflare
bootstraps using traditional DNS as the certificate they are using for DoH
is just a standard X.509 certificate issued by DigiCert. I believe you
could just hardcode both the host and IP address on the client side if you
want to avoid this "legacy" step.

On Sat, Mar 23, 2019 at 9:38 PM Paul Vixie <[email-masked]> wrote:

> _______________________________________________
> DNSOP mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/dnsop
>

---

From: nusenu <[email-masked]>
Date: Sat, 23 Mar 2019 21:24:00 +0000

Paul Vixie:> if all you have is an ip address (say, from dhcp or resolv.conf), how
> would you decide whether the https endpoint you found at that
> address, was using an x.509 key you had any reason to trust? https
> wants names.

https works also without names it is just less common.
Example:
https://1.1.1.1/

but event that is not needed in this specific case since Kenji mentioned
that they plan to initially ship a hardcoded list that maps resolver IPs
to DoH URIs. From the thread on the dns-privacy list:

> Until there is a better discovery story, we could have a list of
> known-to-be-DoH-compatible DNS providers that we could check against and
> map accordingly.

-- 
https://twitter.com/nusenu_
https://mastodon.social/@nusenu

---

From: Kenji Baheux <[email-masked]>
Date: Sun, 24 Mar 2019 14:43:29 +0900

> 1) Do you have evidence that DOH is faster than DOT, since speed was one

Speed isn't a primary goal, hence the use of "hopefully". Based on
Mozilla's early results [1], there is hope that the performance will be
improved at the high percentiles and remain neutral/acceptable otherwise.

[1]
https://blog.nightly.mozilla.org/2018/08/28/firefox-nightly-secure-dns-experimental-results/

> to protect privacy.

We are not considering DOT, just DOH.

I believe that there has been many discussions in the past that offered
reasons why a browser / ... would naturally prefer DoH over DoT.

I don't think I have anything to add that would sway the debate in way or
another. In fact, I think it's fair to say that the debate is unlikely to
come to an end anytime soon.

Instead, I imagine it would be more effective to focus on specific
scenarios: who are the actors, what do they want or accept/understand, is
the scenario reasonable to all parties involved, what can they do, etc.

Specifically, you're preferring DOH but your stated

---

From: Brian Dickson <[email-masked]>
Date: Sun, 24 Mar 2019 04:48:27 -0700

> We are not considering DOT, just DOH.

I sincerely hope you reconsider this position.

See below on a variety of factors on why DoT >> DoH, for enterprise
use/deployment(s).

The concerns that have been raised about blocking DoT are, IMHO, somewhat
"red herrings".
The specific blocking scenarios are likely to be "block all DoT EXCEPT this
specific set of DoT servers", which is not the same as "blocking DoT". In
particular, it does not downgrade to Do53.

However, it is much more likely, at least in enterprises (especially in
data centers or in multiple security zones) that DoH will be blocked
entirely, while DoT will be restricted (but not blocked).

> the scenario reasonable to all parties involved, what can they do, etc.
Let me give some specific examples of actors, including DNS operators, DNS
software maintainers, malware and other bad actors (with very specific
examples), and CIO/CISO types.

DNS operators (i.e. those operating recursive dns resolvers) generally have
specific existing environments, consisting of selected DNS software
vendor(s), DNS configurations, plus additional DNS-related components.
Changing software vendors is not something done lightly, and generally
requires compelling reasons. Many aspects of DNS configurations do not work
the same, or at all, if vendors are switched.

DNS operations frequently entails a number of additional elements,
including policies for resolution (white lists, black lists, dynamic
white/black lists), security-zone specific configuration elements
(typically maintained with automation tools), logging, inspection,
alerting, and active mitigations (such as RPZ).

DNS operations may involve combinations of caching, recursive resolution,
forwarding, including on a "split-brain" server, and sometimes integrated
authoritative service for a configured set of zones (typically via AXFR
secondary).

The existing mostly-UDP nature of Do53 is problematic for traffic that
crosses security boundaries, specifically because of the stateless nature
of UDP, and the ability of spoofing source addresses. DoT improves this
aspect of DNS operations considerably.

The incremental development of DoT, compared to Do53, is relatively minor,
and from a software perspective, very constrained in the additional code,
pathways, and ability to inspect/audit the changes to the software. This is
especially true when the same software packages are used, with the DoT
being essentially an upgrade. Upgrade/downgrade of software used by DNS
operators means configuration files, zone files, etc., don't require
modification, limiting the impact of requirements to perform roll-backs if
needed.

DNS operators in enterprises often are required to support multiple
environments, including end-users with a large variety of COTS (commercial
off the shelf) equipment, including multiple form factors (desktops,
laptops, tablets, mobile devices), multiple vendors, multiple OS (even
within vendors), multiple applications (everything from OS stub resolvers,
to mail packages, to browsers, to VPN software, to third party
applications), across possibly large ranges of major and minor versions.

The support for multiple environments of greatly simplified, if the same
DNS software packages can be used in multiple environments, including using
a small number of shared caching resolvers (each with their own security
profiles, configurations, etc.).

The operations and support issues strongly lends itself to use of DoT for
inter-security zone resolution, and for shared use across apps/OS/etc which
have DoT enabled and configured. The use of DoT at an egress boundary for
an enterprise, provides the greatest level of all of the features required
and/or desired, for the variety of clients across the enterprise. Operating
the DoT server itself enables all of the functions mentioned above in the
Do53 scenarios (monitoring, alerting, policy, logging, and mitigations). To
the degree permitted/mandated, other aspects of the larger security/privacy
can be centrally administered and enforced (obviously depending on DNS
software implementations providing the necessary features). Examples would
include protection of PII and/or anonymization, log encryption/redaction,
etc.

Other security issues not discussed previously in this thread, which
provide context and motivation:
Case #1:
One major enterprise concern, is "information leakage". DNS queries which
get sent in the wrong direction, may leak sensitive information about
infrastructure. The sensitive material is often the domain within which a
query name exists, such as internal-only domains. As an example, PCI-DSS
considers infrastructure information, including names, to be sensitive
data. Depending on the security zone for the corresponding domain names,
leaking those queries may violate PCI-DSS (this is a bad thing). Thus, DNS
operations for these kinds of environments requires that the DNS resolution
be local, and available for inspection, logging, alerting, and blocking,
and thus the network operations must be able to block any DNS resolution
outside of the specific DNS servers and protocols explicitly in use. When
combined networks for users and non-user machines (servers) exist, the
servers' requirements trumps any desire to offer users third party
privacy-enabled DNS servers, regardless of protocol (DoH or DoT). Providing
users with equivalent services in-house, generally favors single-protocol
and same-vendor solutions, i.e. DoT on existing DNS software platforms.

Case #2:
There exists malware "in the wild", which thankfully has only been seen in
very limited environments, that makes use of DNS as part of its primary
functionality.
Rather than trying to be sneaky, and re-implementing whatever resolver
capabilities it needs, this malware use case is actually orthogonal to DNS
resolution per se.
What it is doing, is using DNS queries (and in some cases responses), as a
covert communication channel.
One know case involves data exfiltration using a flow:
data->encryption->chunking->encoding, where the encoding is a single DNS
label. The label has a specific domain appended, and the query is sent.
The result is ignored, and the malware only requires the public encryption
key to do its dirty work. It leverages the host's on DNS stub, and the
malware is extraordinarily simple and small as a result.
DNS operators are unable to see the unencrypted content or infer anything
about the query names, other than observing relatively high levels of
entropy.
Exfiltration done slowly is likely to not be detected (or even be
detectible). Visible entropy can be reduced by the malware by using simple
"book codes" (old cipher techniques), at the expense of lowering effective
transmission rate.

^^ This is why operators are likely to insist on having DoT -- there is no
guarantee the operators will be able to actually detect such malware, but
if they do, they will have the ability to mitigate (by blocking) the
outbound query traffic necessary for the data exfiltration. If the DNS
operator is not able to MITM the outbound DNS queries, this detection and
mitigation becomes categorically impossible.

Third party (global) operators of DNS resolution have, at least in some
cases, indicated that they will support both DoT and DoH.

The missing element is the support for both DoT and DoH on the part of ALL
browser vendors.

The second element is the requirement that DoH for sure, and to a lesser
degree DoT, needs to be STRICTLY OPT-IN. (This probably requires a new
thread to explain, but the short explanation is, "Did we collectively learn
nothing from SPAM and the opt-out vs opt-in issue?").

Thirdly, the user choice should generally be third-party operator, with
possibly preferences on DoT vs DoH (which has higher preference). A given
operator accessible over DoH or DoT is what the user cares about. Unless
there is any clear evidence/proof that either has unique vulnerabilities,
my assertion is as follows: If the destination operator is the same, the
security/privacy of DoH and DoT are identical. If a network operator
chooses to only permit one of the two protocols (DoH or DoT), that should
not have a negative impact on users, if their choice of provider is
accessible over the non-blocked protocol. (Limiting choice to specific
operators of DoH/DoT, including in-house enterprise servers, is an
orthogonal issue.)

I hope this provides sufficient "color" to explain to implementers (e.g.
browser folks) who don't currently "do" DNS operations, why it is that DNS
operators have requirements, and not just opinions, on the underlying
issues. In many cases the decisions involve approvals made by senior
managers (e.g. CIO or CISO), or are constrained by per-jurisdiction
regulations or laws (GDPR), or involve business decisions based on scalable
deployment and operations. There are often accompanying issues, such as
contracts for software support by vendors of DNS software, feature parity
and compatibility, and constraints which are not necessarily obvious.

>> prefer DOT vs DOH to meet both these goals.
I have a similar follow-up question:
I believe that both DoH and DoT construct ordinary DNS Messages (in wire
format), "under the hood", and those messages are then encapsulated over
whatever mechanism the respective protocols employ (naked DNS messages over
a TLS-encrypted TCP session for DoT, and HTTP GET or POST (per DoH
standard) over TLS-encrypted channel (HTTPS) for DoH.

Since both standards require that the query and answer be constructed and
parsed/handled, regardless of transport mechanism, this suggests that
browser support for DoT should be very close to trivial, if not actually
trivial.

Am I missing something, and does this potentially encourage adoption of DoT
by browsers as a first-class supported DNS resolution mechanism?

(Please do not suggest plug-in use, lest this discussion devolve into
impolite exchanges.)

One additional question:
If DoH is supported (with or without DoT support), is there any potential
(and dare I suggest, plan,) to augment the DNS resolution by browsers, to
add support for DANE? Especially for the ability to deploy DoH and DoT
servers, DANE would be a huge win. (Please say "yes". Pretty please, with
sugar on top.)

Brian

---

From: Tony Finch <[email-masked]>
Date: Mon, 25 Mar 2019 13:56:02 +0000

> https works also without names it is just less common.

It's difficult to get a certificate for an IP address.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
Forties, Cromarty, Forth, Tyne: Northwest 5 or 6, backing west 4 or 5,
occasionally 7 at first in Forties. Rough, occasionally very rough at first in
Forties, becoming moderate. Fair. Good.