Subject: [Doh] Locally served zones in DoH

From: Ted Lemon <[email-masked]>
Date: Thu, 19 Jul 2018 18:57:54 -0400

I think that the document needs to speak more explicitly about
locally-served zones.   Right now it mentions that there can be issues, but
doesn't make any clear or unequivocal recommendation.   I think that by
default you can't use DoH for locally served zones, but if you wanted to
offer that service, there would have to be a way to tell that you were on
the network where a particular DoH server served a particular set of
locally served zones.   I think that's not something for the base spec, but
I mention it because I think this issue needs to be seriously considered
and the base spec needs to call it out specifically.

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 19 Jul 2018 19:03:38 -0400

On Thu, Jul 19, 2018 at 06:57:54PM -0400, Ted Lemon wrote:
> clear or unequivocal recommendation.   I think that by default you can't use
> DoH for locally served zones

Could you say why?

Thanks,

A
-- 
Andrew Sullivan
[email-masked]

---

From: Ted Lemon <[email-masked]>
Date: Thu, 19 Jul 2018 19:07:35 -0400

Sure.   If you point your resolver at a DoH server that is not served by
the local provisioning domain, then any queries for zones like home.arpa,
services.arpa, 168.192.in-addr.arpa, ipv4.arpa, etc, will be answered with
NXDOMAIN, because that's what the delegation from the root says.

On Thu, Jul 19, 2018 at 7:03 PM, Andrew Sullivan <[email-masked]>
wrote:

> _______________________________________________
> Doh mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/doh
>

---

From: Ask Bjørn Hansen <[email-masked]>
Date: Thu, 19 Jul 2018 19:13:16 -0400

> On Jul 19, 2018, at 19:07, Ted Lemon <[email-masked]> wrote:
> 
> Sure.   If you point your resolver at a DoH server that is not served by the local provisioning domain, then any queries for zones like home.arpa, services.arpa, 168.192.in-addr.arpa, ipv4.arpa, etc, will be answered with NXDOMAIN, because that's what the delegation from the root says.

How is that different from other variations of “differences when using a differently configured resolver”?

Ask

-- 
http://askask.com/

---

From: Patrick McManus <[email-masked]>
Date: Thu, 19 Jul 2018 19:14:13 -0400

That's true - but its a property of an open cloud resolver, not DoH (which
could be local or cloud). open cloud resolvers are already ~10% of the
internet and ~0% of that is DoH.

(no cite on that number - its ok if you choose not to believe me :))

DoH does mention it but imo I think it does so with the appropriate amount
of emphasis. It might be helpful to have such a statement talking about
cloud based dns services independent of protocol.

-P

---

From: George Michaelson <[email-masked]>
Date: Thu, 19 Jul 2018 20:49:23 -0400

If (for instane) I'm bound to a VPN of some kind with split horizon
back to home net, and I have DOH served DNS, what would I want .home
and .local to do? I'd want them to be "inside" which in a split
horizon DNS, I can "see"

so to me, this is contextual: if you are inside the home serve
context. DOH should serve local not NXDOMAIN.

If you use a public resolver service over DOH then no. its NXDOMAIN

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 19 Jul 2018 20:50:59 -0400

Sure. But none of this is unique to DoH, is it?

A

--
Please excuse my clumbsy thums
----------

---

From: Ted Lemon <[email-masked]>
Date: Thu, 19 Jul 2018 21:27:59 -0400

To be clear, I am not saying doh can’t serve locally-served zones. I am
saying that we need to talk about the problem explicitly to make sure
resolver implementers are aware of the issue.

---

From: Star Brilliant <[email-masked]>
Date: Fri, 20 Jul 2018 02:49:45 +0000

Actually the resolver cannot control the behavior of local zone resolution. It is controlled by the operating system.

>From my experience from a DoH setup on a Mac, mDNSResponder issues a "local. IN SOA" query to DoH to detect local-zone availability, and switch to mDNS (multicast DNS) when it founds DoH server not responsible for local zone.
Also, AFAIK Linux uses a combination glibc, avahi, and sytemd-resolvd to automatically do this.

So in most situation, the OS uses unicast DNS for local zone if available, and falls back to multicast DNS when unavailable. You don't need to do anything to make it work.

But your situation is different: You use a split-tunnel VPN. Depending on your OS and network manager (NetworkManager, systemd-networkd, SystemConfiguration.framework, NetConnection), when you connected to a VPN, your previous DNS configuration may be lost.
If this happens, I suggest you to install a local DNS server to do split DNS resolving.

My recommendation is Unbound, although dnsmasq and knot-resolver are also popular.
You may refer to "forward-zone" in the Unbound documentation to build your resolver configuration.

In conclusion, this is neither a problem of DNS or a problem of DoH. It's just your OS and network manager should be aware that you need two or more resolvers for a mixed network environment.

On Fri, Jul 20, 2018 at 11:28 AM Ted Lemon <[email-masked]> wrote:
>
> To be clear, I am not saying doh can’t serve locally-served zones. I am saying that we need to talk about the problem explicitly to make sure resolver implementers are aware of the issue.  
>
> On Thu, Jul 19, 2018 at 8:51 PM Andrew Sullivan <[email-masked]> wrote:
>>
>> >>>> Doh@ietf..org

---

From: "Eliot Lear (elear)" <[email-masked]>
Date: Fri, 20 Jul 2018 05:25:09 +0000

That it’s not unique to doh doesn’t mean it isn’t a problem.  But I’m sure that absent additional information it will mean people having to turn doh off in split horizon environments. Having some notion of locality is important and not always easily available on first use.  As we discussed early on some of this can be remediated with additional queries but only when NXDOMAIN is returned. 

Eliot

> On Jul 20, 2018, at 04:49, Star Brilliant <[email-masked]> wrote:
> 
>>>>>> On Thu, Jul 19, 2018 at 7:03 PM, Andrew Sullivan <ajs@anvilwalrusden..com>
>>>>>> wrote:
>>>>>>>

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 20 Jul 2018 08:45:59 -0400

On Fri, Jul 20, 2018 at 05:25:09AM +0000, Eliot Lear (elear) wrote:
> That it’s not unique to doh doesn’t mean it isn’t a problem.

Well, yes, obviously, but I don't understand how RFC 6303 isn't a
consideration here.

DoH servers serve DNS data, and considerations about the data servers
are supposed to serve apply equally to them as to any other
DNS-serving server, I think.  Isn't that enough to say?

>But I’m sure that absent additional information it will mean people
>having to turn doh off in split horizon environments. Having some
>notion of locality is important and not always easily available on
>first use.

Why not?  The way this works in DNS is usually on the base of source
port.  A DoH server knows that, too, no?

Or is the worry that I have a DoH server configured and it doesn't
know about my network?  How is that different from today, if I
configure "my own" resolver?  If you configure 8.8.8.8 on your laptop
and take it into your example.com corporate network, then
"floor1printer.example.com" is also unlikely to work, right?

I think it is useful to note this, but unless the idea is that every
resolver has knowledge of every possible context (plainly, an
impossible goal), then this is going to be a constant problem I
think.

I think Olafur was making this point in the driu BoF: this was a big
part of a problem in MIF, and that WG's approach didn't fully solve
it.  I suspect that the non-solution had to do with the basic
impossiblity of solving this completely generally in an
internet-compatible way.

A

-- 
Andrew Sullivan
[email-masked]

---

From: dagon <[email-masked]>
Date: Fri, 20 Jul 2018 15:08:54 +0000

> Sure. But none of this is unique to DoH, is it?

Locally served zones rely on what might be described as a small
layering violation. That is, knowledge of the stub's IP differentiates
the zone used in reply.  L3 informs the L7 answer.

So-called "Eenterprise DOH" servers inherits this layering issue.

But it also hands the DOH resolver a second problem, inherent in the
DOH client.  There appears to be no way to differentiate organic DOH
queries (generated by a DOH client such as a browser) from induced
queries generated by a soup of js document.write() calls and similar
strategies.

For global DOH resolvers, this problem is easy, since they are
unlikely to offer local zones.  Global DOH operators can elect to
filter *.in-addr.arpa queries, as perhaps categorically unlikely to be
created by any known DOH client, Or they can answer, to ensure DOH
clouds are full DNS speakers, and provide a faithful duplicate of
traditional DNS resolution, all without revealing internal network
information.

Enterprise DOH, particularly those offering local zones, have to come
up with a strategy that does not allow induced crawls of internal
hosts.  One might think ECS could be helpful, but consider that in
most cases, the DOH client rendering a spiteful HTML/js page will be
inside the local network.  If anything, ECS merely increases the harm,
as the scope and mask information will inevitably be leaked back to
the site generating the DOH query by javascript.

Perhaps instead HTTP header properites can help, since they could
indicate what remote site prompted the DOH queries.  For example,
enterprise DOH servers and cloud DOH resolvers might use this, to
manage scrapes and crawls from botnets, ad farms, and the like.  (In
an older post to this list, I raised this issue, perhaps too
forcefully to be appreciated.)

Back to the HTTP header filtration solution: assume
malicious-ad.example.com would be tagged in the DOH header, as the
true origin of a query for *.168.192.in-addr.arpa.  The enterprise DOH
server could intervene with policy for such queries (RCODE=5, or an
analog in HTTP), and allow queries for *.168.192.in-addr.arpa if the
DOH http headers indicate the upstream query came from, say,
ceo.example.net.  A similar approach could rely on a user-agent
encoding used by the DOH client sub, but the Dickinson & Toorop
draft currently states:

   3.5.1.  User agent
   DoHPE clients SHOULD omit the user-agent header.  DoHPE clients MAY
      set this header to 'DoHPE client'

The more mature Hoffman and McManus draft says this:

   Additionally, the User-Agent and Accept-Language request header
   fields often convey specific information about the client version
   or locale.  This facilitates content negotiation and operational
   work- arounds for implementation bugs.  Request header fields that
   control caching can expose state information about a subset of the
   client's history.  Mixing DoH requests with other HTTP requests on
   the same connection also provides an opportunity for richer data
   correlation.

I don't know what the phrase "richer data corrleation" means, but
perhaps it has some security properties as well.  In any event, assume
some header information (an origin, or user agent) is used by the
enterprise DOH server to filter organic DOH queries versus induced
crawls.  This might be useful, but now the historically small
"layering violation" (perhaps that's too harsh a description) used to
handle local zones in DNS, now must become much larger, and include
information about the L7 application that prompted the DOH query.
(I.e., L3/L7 mixing, long tolerated in DNS for local zone management,
now becomes L3+L7/L7.)

One drawback to a header-fix is that it requires strict whitelist-only
discipline.  In an enterprise, the DOH deployment likely now involves
a separate web team, on top of the network and security groups
responsible for safe deployment.

A second drawback is that this assumes no further manipulations of the
HTML, beyond naive javascript document.write() calls.  There's a
healthy industry of extraordinarly talented penatration testers who
will ensure that all probes of the local zone appear to originate from
ceo.example.net, or some other whitelisted source, and that the user
agent matches, if required.

The problems I've identified so far assume an ideal enterprise DOH
deployment.  Yes, local zones are a problem, and perhaps there are
some solutions.  But let's also not ignore self-inflicted wounds.

The long tail of low-end HTTP proxy vendors will no doubt soon see a
spike in their traffic, investigate to discover the volume is
attributable to DOH, and motivated by a basic desire to sell a new
module or be feature-competitive, include a poorly architected DOH
handler.  "DJB DOH" will be offered, somehwere, by a vendor who
understands marketing, but neither DNS or HTTP well.  Others will see
DOH destined for Cloudflare, redirect the query to a local DOH
resolver created by the intern, and point to the reduced latency.

These and other bad choices seem not only probable, but inevitable.  I
am preparing a set of test cases, using a hypothetical NGO, to be
scanned by hypothetical hackers from the Freednian government, to
fully illustrate the malicious uses of DOH.  It seems like local zones
will become public via local DOH, unless someone has a clever fix.

It seems prudent to discuss this in some form in a protocol, if only
to caution against obviously bad DOH enterprise models.  Then again,
this list is public, and "someone told you so", might be an adequate
response to a proxy-turned-DOH vendor who hands over local zone
details to an anonymous adversary.

I caution the concerns I've raised have only been briefly tested.  I
may have even more concerns to raise, and also apologize if my
whiteboard-quality comments are not sufficiently clear.

-- 
David Dagon
[email-masked]
D970 6D9E E500 E877 B1E3  D3F8 5937 48DC 0FDC E717

---

From: George Michaelson <[email-masked]>
Date: Fri, 20 Jul 2018 11:10:04 -0400

putting "leak of secrets" to one side, if you have DoH you have client
header proffer in HTTP(S)

So, we can move beyond bits in DNS, to state we carry in HTTP to
signal intent and desire.

But, somebody is sure to cry "won't somebody think of the secrets"

---

From: dagon <[email-masked]>
Date: Fri, 20 Jul 2018 18:06:19 +0000

> But, somebody is sure to cry "won't somebody think of the secrets"

I'm not arguing against engineering a new lawn dart.  I'm merely
suggesting some safety instructions.

DOH was designed to be a substitute for traditional DNS.  But there
are circumstances, such as local zones, that present operator hazards.

Considerations:

  -- Should cloud DOH answer reverse queries?  Or only answer a
     handful of common forward types?

     The harm here seems annoyance of the cloud DNS operator, as well
     as the use of advertising tools to conduct global DNS scanning
     from unwitting third parties.

  -- Should or must DOH client implementations somehow identify their
     organic queries via header proprities, or the site inducing the
     DNS message over http?  The harm here is the enumeration of local
     networks, by malicious remote site owners.

  -- Should edge/enterprise DOH also serve local zones?  Same.

  -- Should ECS be turned off or dropped (or default zero scope) in
     edge/enterpise networks, or operate under the scope and mask
     properties set by the local network?  The harm here is the
     enrichment of the data set to remote site operators, detailing
     internal network allocations.

  -- Should http proxies redirect DOH queries, away from cloud DNS?
     The use of cloud DNS is an audit criticism in enterprise network
     evaluation.  Some might think redirecting the DOH volume to an
     internal server solves this problem. It instead would expose the
     local network information to remote sites, as noted.

We of course don't have to consider these. There is an argument, as
you seem to suggest, for a strictly utilitarian model of a protocol
description: coder beware.

Still others view lawn darts as profitable, as well as their injuries.

Prudent advice, in the form of considerations in the draft, might help
those implementing DOH to appreciate the design choices they'll
encounter.

---

From: "Eliot Lear (elear)" <[email-masked]>
Date: Fri, 20 Jul 2018 18:29:17 +0000

Forgive the cell phone top posting, but I think the suggestion is to note the issue.  I don’t think I read anyone suggesting delving into solutions. The reason it can be tricky is that locality can involve multiple groups of networks and multiple domains. This is quite typical in large organizations. 

Eliot

> On Jul 20, 2018, at 08:46, Andrew Sullivan <[email-masked]> wrote:
>

---

From: Ted Lemon <[email-masked]>
Date: Mon, 23 Jul 2018 08:38:48 -0400

Correct, Eliot.   What I would like to see is it clearly stated that if a
DoH resolver doesn't have some kind of bypass for locally-served zones, the
end user will not have access to those zones, and this will likely cause
operational problems which the user will be unprepared to diagnose.   As
someone pointed out, existing resolvers configured to use 8.8.8.8 have this
same problem, so it is not unique to DoH.   This is just an opportunity to
raise awareness of the issue, and I think it's worth doing.   I think
something along the lines of treating the local network as a separate PvD
(RFC 7556) might work, but is out of scope for this document.

BTW, separately, I think sending information to a generic DoH server about
which web page triggered a DNS query is a bad idea.

On Fri, Jul 20, 2018 at 2:29 PM, Eliot Lear (elear) <
elear=[email-masked]> wrote:

> Forgive the cell phone top posting, but I think the suggestion is to note
> the issue.  I don’t think I read anyone suggesting delving into solutions.
> The reason it can be tricky is that locality can involve multiple groups of
> networks and multiple domains. This is quite typical in large
> organizations.
>
> Eliot
>